Wildrose makes a great point about toughest game and longest range, then working backwards from there.
Many shooters only have one or two centerfire rifles for various reasons from personal preference to finances. If you have several it is easy to fine tune the caliber to the game or activity, if you have only one it is all about trade offs.
I must admit I like big caliber rifles for long range hunting, 338 cal and up. Here is why.....
Lets pick a 6.5/284 and a 338 Lapua for example:
Will the 6.5 kill an antelope at 500 yards, yep. Will the 338 kill an antelope at 500 yards, yep.
Will the 6.5 kill an elk at 1000 yards, maybe, but very marginal. Will the 338 kill an elk at 1000 yards, yep, with authority.
Obviously two simple examples, but you get the point. The arguments against the larger calibers go like this. They are overkill for smaller game at short ranges. So what? Dead is dead. Unless you are talking about .50 BMGs the meat damage from the larger calibers is comparable to the smaller calibers.
Weight. Big caliber rifles weigh too much. You can build a 338 at the same weight or very close to the same as a 6.5, and you can build both so light that most people cannot shoot them accurately in the field.
Recoil. You got me there. The big boys kick more unless you use a good brake (I always do) then recoil is quite mild even with the big 300 grainers.
Reloading expense. Yep, the big boys cost more to reload but unless you are using some kind of premium bullet, not much more than the smaller calibers. Unless you are burning thousands of rounds a year, the extra loading cost is minimal compared to everything else you spend on hunting. You will probably spend more on gas driving out west for an elk than you will spend on reloading supplies for the year.
All that to say this: If you are going to go long, go big.