That's not too bad but you might be able to tighten up your groups if you experiment with seating depth, powders and primers. You said these were "touching" the LaG's and sometimes Bergers like em' jammed in there .010", maybe even .020". If not, trying backing out .010" and .020".
I'm getting that kinda accuracy and better velocity with my 6.5-284 at 500yds. Nosler list 51.5gn H4831SC @ 2850fps with 26" barrel. I couldn't get above 3000fps with H4831SC, so I tried other powders. Finally settled on RL25 and groups are hovering in the 1" range at 500.
Your mileage may vary as my rifle is just a plain ole factory Savage w/26" barrel.