Although the neck and shoulder get the most work done on them, the entire case to a degree is also worked. I anneal every other firing with my long range rifles, and after 3 firings on everything else.
The reason you are seeing more pressure signs is because as the brass hardens it doesn't 'spring' as much, either when it expands to grip the chamber, increasing bolt thrust, or when you size it back to smaller dimensions, annealing brings 'spring back' back to spec, increasing neck tension and decreasing some of the bolt thrust.
Obviously you are only annealing the neck, but I allow the heat to flow about a third down from the neck into the body, just like factory Lapua brass looks like. This reduces bolt thrust a reasonable amount and the brass lasts longer.
I have heard people say they get 20+ loads from their brass, that's fine, but if you get to the point where the 5th trimming is necessary, that brass is toast.
There are plenty of videos on youtube on annealing rifle brass, if you're unsure, have a quick look before you start.