Stupid answer time.
Have you hunted with the rifle recently prior to the trouble starting?
I have a really lightweight Kimber in 25-06 that started doing a similar thing. Right after a hunting trip it decided to print the first shot way off POA, subsequent shots went into the group at POA.
Tearing down the rifle after checking rings, bases etc, I found a twig had lodged itself between the barrel and fore-end.
After it was removed, no more POI shift or fliers.
Sometimes things shift also, so a tear down and re-torque on the action screws may help.
Cheers.