I'll second the H4831sc as it works very well in the standard 6.5-06. You mentioned shooting 3.7" at 5oo yards. Not sure what your intended purposes are but that is not bad shooting. You are well under minute of angle and plenty good enough for long range hunting.
You should give the match bullets a try like the 142 Sierra, 140 A-max, 140 JLK, and 140 Berger. If you are under minute of angle already with just a hunting bullet, the gun sounds like it has promise.