I shoot a 25-06, my son shoots a 243. We've both taken 2 to 4 deer every year for quite a few years now. I shoot a 100 grn Barnes TSX, he shoots an 85 grn Barnes TSX. They both shoot around 3200-3300'/sec with said loads. I've also loaded the 25-06 with 85grn Ballistic Tips @ around 3600'/sec for varmints and coyotes.
I personally like the 25-06 better since I handload, it's more versatle.
If you're planning on shooting Factory Ammo, the 243 will give you alot better ammo selection......more brands and bullet weights available in .243. And, it will do the job you mentioned out to the distances you mentioned quite nicely.
More guns and varieties are available for the 243 also.
I cut my teeth on a 6mm rem. and low drag bullets are more available in 6mm, but it's harder to find than either of the above, and still falls short of the 25-06 on heavy bullet velocity. However, maybe a 6mm AI or (better yet) a 6-06 Improved with a fast twist would be something to look at if you want to go full custom and reach out to the 500 or even 600 mark on deer. Still low recoil/cost and fun to shoot.