Yours is not the only rifle that does that. I have several, and I deal with it by firing a fouling shot through it and dry patching before I go hunting. It will eliminate this problem and be dead on for the first cold bore shot on game.
The change it caused by oil or solvent left in the bore and only firing can remove all of it and in some cases seasoning the barrel with a little copper fouling will make the difference.
I have a 7/08 that does the same thing but after the fouling shot it will consistently group 5 shots under 1/10th MOA.
In other words, there is nothing wrong with it. It is just an individual and likes what it likes.
J E CUSTOM