From most any position, if the sights jump to the left when dry firing, the trigger finger is too far to the left when it's pulling the trigger. After the sear's released, some of that force moves the rifle to the left during trigger travel during backlash when it comes to a stop before the bullet leaves the barrel.
If you're right handed, try putting only the first finger pad (beneath your fingernail) on the trigger, then do some more dry firing. Note carefully where the sights jump when the firing pin falls. If it jumps to the right, you don't have enough finger on the trigger; to the left means still too much finger on the trigger. Southpaws have the reverse situation.
After you're able to snap an empty chamber without sideways jump, have someone help you go through a "ball and dummy" session. That's an old military term where someone loads your rifle then you shoot it. You don't know if it'll go "bang" with real ammo or "click" with a dummy round. This helps one learn that they do in fact flynch if they jump when it clicks. Once you're able to hold still through the "clicks," then you should be able to shoot well with live ammo.
If you're convinced you don't flynch with live ammo, then it's your finger position on the trigger that causes the sideways jump.
Oft times, folks subconciously flick their trigger finger forward when they feel the sear release. This will move the whole rifle away from where the shot was aimed. You gotta keep the trigger finger pulling back until you stop moving from recoil. It helps to hold the pistol grip firmly when trigger pull weight gets over 2 pounds. Have someone watch you and let you know if you're flickng your finger forward when the rifle shoots. Then learn to hold it back until you stop moving from recoil.
