Hi Olson,
You'll probably get opinions from everyone on their favorite caliber. What I don't see mentioned is whether you want to use the rifle just to bang a gong at 1000 yards, break a rock at 1200, or take down a deer or elk. The caliber you pick will be based on your objective. Popular ones for hunting are the 7mm/300/338 families.
I agree with the "start small" idea, and then working up. In fact, I recently went so small that I picked up a(nother) .22 LR rifle built specifically for learning long range techniques. It costs pennies to shoot, and the local 100 yard indoor range can teach me a lot about long range before I move to a 200 yard outdoor range when the weather improves. To hit at 100 yards I have to adjust the elevation of the scope just like on a long range gun, and it's interesting to play with.
Sure, I've got larger caliber rifles too, but before I start spending $1-2 per round on an education I'm spending $0.01-0.02 per round.