So besides shooting different ammo, what have you checked ?
Did you check the trigger pull weight ? Did you check the barrel for free floating and the action for proper bedding ? Did you take the rifle somewhere and borescope the barrel and chamber ?
You could have a cast made of the chamber and throat, it can be very revealing. See here for examples of what one can find Bellm TCs - More Chamber Misalignment
The examples are of factory TC barrels, but the truth is that similar sins are committed by all gun makers including Savage and Remington. Its just a numbers game after all, and "minute of pipe plate" is good enough for a great many buyers.
Unfortunately, when it comes to re-barreling a Remington, it gets expensive since one often cannot reap the benefit of a match barrel without addressing other inaccuracies of the mass produced receivers and bolts.
Lead times at barrel makers is very long, especially now. 16-20 weeks is the current norm and add any receiver work on top of that figure.
I suggest to check the easy items in the first part of the list. if nothing there gets you where you want to be, sell and buy a custom action like a Stiller or a Shilen (often you can get a unit from stock with no waiting list) then add a prefit match barrel (since the custom receiver needs no "trueing" and the barrel tennon can be a standard size. Then add whatever trigger and stock you want since it will fit anything that is Remington compatible.
Or if you can stand it, buy a Savage, screw on a $350 prefit, adjust the trigger and go shooting....
As you can imagine I have mainly Savage rifles and just got rid of my last Remington, but I certainly do covet a nice stiller action with a Shilen or Timney trigger which I can never have with my Savages.