Good brass is likely already annealed. So no need to anneal before fire forming.
Then decide whether or not to anneal. I think if you have a tight chamber and you dont overdue sizing, and its a hunting gun, annealing probably is a waste of time.
If your going to anneal, I would anneal every time for consistency. If your going to anneal, I believe the process needs to be absolutely consistent. Like an amp annealer.
I've seen my benchrest dasher groups change when I used a standard shell holder vs a +.010 shell holder in my amp annealer. I cant imagine how much variability one could induce with less precise types of annealing. I also know a guy that has never annealed his 6 bra and has 35 reloadings.
I do anneal my custom hunting guns and my benchrest gun because I always have. If I didnt have an annealer and i know what i know now, I think I would just skip it altogether, even on my benchrest gun.