Are larger caliber rifles naturally less accurate?

[Custom66]

When I have these kinds of problems I simply tear the rifle down, go back to basics and rebuild it from the bottom up. Check all mechanical connection points, clear the barrel channel to provide a LOT of clearance for the barrel, carefully bed the action, torque in proper sequence and correct torque specs.
Then I will often shoot a box of factory rounds so I have a base line to start from. Then I work with a series of loads that vary .1 - .2 grains, ten rounds of each.
Got to keep in mind that comparing the accuracy potention of a .223 to a .308 is like apples and oranges. The smaller caliber can do incredible things at the not so long ranges but it'll fall apart when you try to reach out with it.
I'm a bit curious about the bullet weights you're using to do load development. I might lighten up just a bit and see how a 250 grain performs. Try the 250 grain Sierra MK HPBT.
To the question as to whether larger caliber rifles are naturally less accurate, here's whay my .284 can do:

Do you consider a 6mm Creedmoor a small caliber? These are my 3 shot groups at 1000 yards. The 3 shots in the center are mine. 107 Sierra MK, and H-4350. I built this rifle, and believe I can do this with a .338 as well.

 

[Custom66]

Do you consider a 6mm Creedmoor a small caliber? These are my 3 shot groups at 1000 yards. The 3 shots in the center are mine. 107 Sierra MK, and H-4350. I built this rifle, and believe I can do this with a .338 as well.

Anything can be made to shoot well, some just take more effort than others.

 

[450 Dakota]

I might of missed it but if you have chronograph I would compare the speed of the factory rounds to your loads. If you are off by a fair amount you might adjust your loads to get it closer to the factory velocity. I have had good luck with Fed 215s in the larger cases. I use a Past shoulder pad on the larger stuff when load testing as well, takes some of the sharpness out of it.