On the windage issue, what type of scope do you have? Sometimes the adjustments stick, time for a repair or new scope. I simply will not tolerate a scope with inconsistent adjustments. LR hunting depends on the performance of your scope.
Secondly, how was the wind? A light breeze will blow you that far. Use the pond as a big wind flag. Any ripples indicate a breeze and must be adjusted for.
Final thought would be to make or invest in a solid portable bench and proper rests. When testing, remove all possible variables. A wobbly rest and inconsistent shooting will simply drive you crazy and tell you nothing of your progress. There will be plenty of time to practise from field conditions AFTER the rifle and load are dialed in.
Still, you are getting very useable results. Just you will never know what to tune if you can't tell what caused the problems.
Most definitely bed that action. Remove the action screws, move the action in the stock. I bet you can move it around a bunch. A few thou is too much. Just imagine the stress and potential movement during recoil.
The action should be stress free in the stock once bedding has been completed. It will fall into the stock with a 'klunk' and not move in any direction. If fact, it should feel so solid that you can shoot without action screws. A glove fit is the goal. Also, ensure that the action does not move or bend when the action screws are tightened. A common problem.
Zero movement when the action screws are tightened and loosened. I bed under the first 1" of barrel. Some don't. With a light sporter barrel, both methods work just fine.
Free float the barrel after the bedding is done. I like at least 1/16" (very thick business card) around the barrel.
Now resight your rifle and rework up your loads (oops factory ammo). good chance the sweetspot will change but will now stay more consistent.
Proper rests, good form, calm conditions or wind flags/watch that pond, practise, practise, practise. Start thinking about reloading if you want the best out of any rifle (odds of factory shooting as well is not that great).
Sounds like you are well on your way...