To answer the OPs question, they may or may not change within the season. As several others said antler growth is triggered by photoperiod, therefore, a individual that is moved between hemispheres would adapt to that new photoperiod. This happens relatively quickly, but if they are moved during a period of current antler growth they it may take a whole cycle to adjust. There have actually been some really interesting experiment testing this.
Weather and the environment do not have an effect on antler growth, but it can have an indirect effect on the breeding season (and rut). For example in AL the state restocked it's deer population with deer from the Delta region of the state and now the vast majority of the state has a delayed breeding season. The reason for this is that those deer adapted to a delayed fawning period because of the spring/early summer flooding present in the Delta region of the state, therefore the peak of the rut in the majority of AL is now late December and early January. Historically, meaning prior to restocking, it was in late November (similar to surrounding states) and January only in the Delta region. But as I mention, this doesn't influence antler growth. Those same deer with grow antlers, shed velvet, and cast antlers during the same time period and surrounding states.