Without seeing it, I would venture to say it's because you are pulling the gun into yourself. Your trigger hand (assuming right handed) gives the gun a fulcrum to lever off of and move to the right. But that's just a guess.
When I shoot off of the bipod, I like to preload the bipod with just the weight of my body being pushed forward either by how I come up on the gun with my weight forward so I kind of 'fall' into a preloaded bipod, or a slight push off of the tips of my toes puts me into the gun.