I recently did a test of my own on the primer subject.. Fed215's vs RWS LR primers to be exact. The research I-did (before-hand) showed the Fed215 to be the most abrupt and hottest ignitions of all large primers. What I did to test it was load single-charges for my caliber in one-grain incriments (with each primer & each brass-brand) until I hit pressure. Then, once settling on a safe max loaded five of each to to record what I saw for difference...
The way it went was as expected. Fed215 showed roughly 50fps faster loads with tighter ES in velocity (vs the RWS) in ALL the brass brands I tried, however, the group-sizes themselves weren't effected like I thought they'd be. Whether that is because I settled on an ideal accuracy node or what, and the velocity spread between primers didn't sway enough to take the load outside that node, I don't know. But to answer your question, depending on your bullet/caliber & how the load itself was dialed in, a primer change may not hurt your group at all-- but attention should be given to re-verify things in a proper manner
In your case it's very possible a softer igniting primer will help come (slightly) off and away from that pressuring ejector mark.. but take that as a possibility only! Load a few up and try it to see. Good chance it'll work.
...only other thing you could do is try a different batch of brass. Neck tensions almost always vary between "lot's" of brass (even of the same brand) & that will alter pressure, as will the capacities of the different brass. The pressures are without a doubt effected by those two things and sometimes it is drastically* But that does open a whole subject in-itself. If a guy's going to ride that line on max, you have to have it-all down to an art, it's not advisable, but it is doable.