You are going to get 100 different answers from .22 to 50cal. The best thing to do is go here and get familar with ballistics. This link will save you so much time and be able to give you drop charts for your loads as well. Fantastic and FREE!!!
JBM - Calculations
If you go to the Trajectory (or Trajectory/Drift if you want wind drift too), select a bullet make/weight from the list you plan to use. Enter the FPS your load or factory clocks at, the twist of your barrel and the distance you plan to shoot (default is 1000yds) and hit enter, it will give you the drop, velocity and energy of that load at a given distance in 100yd increments. All the data you need will be presented.
May say a good rule of thumb is 1800 fps (depending on bullet expansion) and 1500 ft/lbs of energy on target. To some, that will be over-kill, to others the min. Also, shot placement is key as well. A big bullet not in the kill zone is not as effective as a smaller one in the right spot. You will also here "there is never too dead".. which means biggest you can handle is best. All subjective to an extent. 100yds away with a .243 is different than 800yds away with a magnum.
Distance - How far do you plan to shoot? Max limit you place on yourself
Recoil Tolerance - Can you hit what you are aiming at with a Magnum?
Weight - Can you carry a heavier gun to absorb recoil of bigger caliber or do you need a uber light carry.
All things I've taken in on my learning process.