Lee Collets are notorious or infamous about sizing the neck with very little bullet grip. I ordered undersize mandrels from Lee one time and they argued with me that increased bullet grip would increase runout. They sent them anyway at $5.00 each
Some have chucked them in a drill and held and abrasive against it (sandpaper, krochus cloth?) but I did not have good luck with that. The entire mandrel needs to be exactly the same diameter and perfectly round since you have to pull it back out of the neck after sizing.
Your brass gets work hardened and will have more springback after several firings. When you size the brass onto the mandrel, a work hardened case will spring back more and have a slightly larger ID. Perhaps you could anneal the case and that would help.