A couple of things could be going on here. If your powder burn rate is too slow, it is not providing enough pressure to seal the brass to the chamber, and / or, your brass is work hardening. ( or a combo of the both of them). It also can be caused by a more then liberal chamber, and again, any combo along with the chamber, will cause it. You may want to experiment with another brand of brass, or maybe another lot of the same as what you are using first. Annealing may or may not help if you have a over generous chamber. I have a 308 that does that, but it shoots fantastic groups, so I don't care if the chamber is generous or not, I'm not changing anything on that one. Good luck in your search, but if it's shooting good groups, don't go nuts trying to fix, or figure it out.
PS, what powder and bullet are you using?