Anything larger than a 44mag espeicallly in a Encore with a 10"bbl with out a brake will have alot of recoil. A scope will add weight to the handgun, and that will help.
If going with a rifle cartridge, maybe a .243 would be better, still a 14"bbl in my opinion will serve better.
You have to look at it two ways, a Large bore handgun will knock something down with a larger, heavier bullet, they are made for shorter bbls but at the distances involved you need to become proficent at shooting a straightwalled case (ie: .44 mag, .357 Max, .45LC, or .454 Casull, .475 Linebaugh even the .480 ruger & newer S&W .460 & .500's are excellent. But getting into those larger bbls a brake is almost manditory, if not for your arm and aim , but also for the scopes life.
The rifle cartridges shoot a lighter bullet much faster and could possibly have more "down range" power if not accuracy depending on your choice of calibers. Granted you are loosing a great deal of velocity in a 10"bbl vs a Rifle bbl. But a .308 class of case whether is be a .308, .243, .260, or 7-mm08, are certainly very good deer cartridges in a handgun. I just feel that they are handicaped more the shorter a bbl is. That is why I mention a 14"bbl, in a 10" bbl you will still be needing a brake whether you want it or not, but a 14" bull is much more managable.
I would suggest that you call SSK Industries or even EABCO and see what they have to say, They specialize in these things and would definately be a good source of advice.
Hope this is helpful,
Dan