First let me welcome you to Long Range Hunting. I have been hand loading for a few years
( 30 ) and have developed some opinions about powders. Usually Varget is best in smaller capacity brass than your 30-06. My 6mm Rem likes it very much.
H 4831 is going to be a better bet for extreme temp variations in your 06.
H 4350 may also work well with higher velocities.
Something else that may be happening if your ammo has been loaded for a year or so, the bullets may be bonding to the brass in the neck area. If so, just seat the bullets a little deeper and you'll break the bond and have more consistent pressures.
Then again, maybe you'll have more fun re-barreling your 06 to something more interesting like a 280 or a 6.5-06.