Pretty much each rifle has it's own set of rules. I have been lucky enough to own some rifles that didnt have much of a problem down to -10 F. Some others of mine were very unpredictable in 40-50 degree F., others had cold bore shot issues that were more predictable. The best thing to do is get to know each one of your rifles.
Long range shooting is a process that ends with a result. Once you start to focus on the result (where the shot goes, how big the group is, what your buck will score, what your match score is, what place you are in...) then you loose the capacity to focus on the process.