Re: 6.5-06 with 140grain bullets
Check the ammo for concentricity/straightness. Biggest problem to accurate shooting. You want max .005" out of round, try for 0.002".
Are you BR prepping and neck sizing the brass? Consider a bushing or collet neck sizing die. Is the chamber/bore straight? How badly is it shooting? What is the scope, how much mag? how far are you shooting?
With the components you are using, if the rifle is well assembled, bedded well, trigger crisp and light, with a good scope solidly mounted, there is no reason why this rig will not shoot. How are your rests/bench? How experienced a shooter are you and your friend? Do you use wind flags?
some of the basics that could help eliminate your problems.