Straydog, assuming good quality bullets, I would steer you towards these areas: Velocity variations, shooting rests and technique, wind, and mirage.
It is not uncommon to see this type of situation. I can usually trace this back to poor powder burn/large vel. variations. Most commonly shown in ball powders especially older styles ie military surplus, and some extruded powders.
Used a ball powder in my hunting '06 that shot inside an inch at 200yds. Thought that would be great for LR. At 500yds, it wouldn't hit a 8X11 piece of paper.
Switching to a new type of consistently burning powder ie Hodgdon Extreme line or Reloader, problem goes away. I now use H4350 Extreme and get consistent sub MOA to as far as I want to shoot.
Are the groups stringing? If vertical stringing, then your load is probably at fault. If horizontally, it is probably the wind.
The further you shoot, the more important environmental conditions are. Mirage and wind can send you all over the map. Not easy to shoot MOA at LR even if your gun can.
If your load is stable and you are using quality bullets, once it gets to 300yds, it will continue to be accurate until it hits the ground. Bullets only show some dispersion when they go trans sonic (sometimes) which is a long ways from where you are.
How are your rests? Shooting technique? Try getting the most solid repeatable set up. A little variation like where you rest the rifle on the foreend from shot to shot, will make a big difference at long range. Rifle torque/roll during recoil can also 'throw' your shots off.
Good luck with developing this rifle. Shooting LR is not easy. That is why it is so addictive and fun.
Jerry