Do you happen to be a left handed shooter? I see this regularly with light weight rifles, especially with larger calibers and heavier bullets. You may be seeing the effects of increasing recoil and slower bullets. The heavier bullet causes more reaction in your shoulder and, being slower getting going, it is exposed to more movement. If you happen to be a lefty, heavier bullets tend to move farther to the left and sometime they tend to hit higher than lighter bullets. If you have access to a fixed rest, like a lead sled, see if you find the same reaction when the gun is not allowed to reciol at all. That can eleminate one potential source of your frustration.
Best of luck sorting it out.
"When working with the public, there are two things you need to remember. - 1. The public is a bunch of ignorant morons. - 2. YOU and I are one of them!"