New cases as they come from the factory are annealed for optimum hardness. This level of hardness enables the case neck to have the right amount of springback to hold on to the bullet. This serves a two-fold purpose: it makes the cartridge sound so you can load and unload it without it coming apart, and fosters uniform combustion of the powder. If the necks are too soft, they will not have any springback. If you seat a bullet into a neck with no springback all that will happen is the bullet will expand the neck and then fall out. This lack of grip also destroys combustion uniformity and ruins accuracy. I always know when I overcooked the necks when the accuracy goes to hell. It turns a half inch load into a two inch load every time. And the extreme spread quadruples. When you anneal, you have to do it right. After annealing my brass using a variety of torches, I have come to a conclusion that none are precise enough. Their flame length varies as does the temperature due to heat soaking of the tip. If you search YouTube you'll run across a video of Norma factory brass production. If you pay attention closely, you'll see their annealing process. They use a long and wide torch head with a series of small flames. The cases pass through the flames at a certain speed and are annealer uniformly. It is far easier to control a small flame than a large one. The best solution for the home annealer seems to be to use two torches with a pressure regulator, preheating, and minimizing the dwell time of the brass. Induction annealing is the most precise method, having the ability to control the dwell time to a milisecond. But it's also very expensive at this point.
If you don't anneal, every time you fire/reload the neck gets harder and the springback increases. This affects bullet seating/bullet release but you can compensate for it with specialty dies having the ability to adjust neck dimensions. After several cycles, however, the neck gets too hard and fails.
Now, depending on your load, the primer pocket may fail long before the neck does. So whether annealing will benefit you will depend on a variety of factors.