I doubt it's a primer problem.
Not because it isn't possible, but because it's least likely.
Primer ignition is a tricky thing. It's more about impact speed, than impact force.
If primers aren't seated to a preloaded condition(crush), they'll seat further on impact
-slowing relative impact speed.
If your pin spring weakens enough in cold temps, or the internal(to bolt) grease thickens, drag can slow impact.
Yesterday turned sad for me when I discovered my firing pin catch had loosened, causing occasional misfires. Like you, the problem was intermittant, and re-firing would eventually set em off. The dimples left on mis-fired primers were plenty deep, but my pin was not pulled back normal distance, and that mean't slower strikes at these temps. A clue to this discovery was that mis-firing occurred only when the GUN got cold enough (I kept my ammo warm).
Anyway, I reassembled the bolt, locktite & a couple mods to prevent future failures,, back to the range and find velocities higher, gun out of tune, a weeks worth of seriously cold load developement invalidated...
Check your bolts