Well unless you've made your ammo directional some how (like it knows right from left), then it is likely something else.
Maybe an existing stress in the gun.
This can be difficult to locate, and sometimes you can't fix it. A tube or chassis gun can put you there.
I hope you realize that temperature transitions and grouping don't go well together.
That hot grouping can print different from cold grouping, and a transition from one to the other is worst.
With some guns you may need to choose one or the other, and never between.
I've had great hot grouping guns that could not be relied on for cold bore accuracy.
And I've had horrible grouping guns that were fantastic for hunting shots.
Of course there are guns that do both, or neither, well.
All that aside, I've seen the resting & hold cause a consistent direction trend. The gun carving (from torque) into sand bags (depending on fill).
A bipod can dig in while you go, and with either you end up twisting or changing hold on the gun a little differently for each shot.
To counter, I fiddle with rests while group shooting to more or less reset each shot. Otherwise, if I have to counter a rest with my hold, that's not good.