any time any weight is added or removed from the barrel especialy at the end their is a good chance that your groups will change , this comes from the change in the barrel harmonics. If you just remove 1/4" of barrel length at a time you will see a differance in group size.
I have seen guns shoot a particular load very well without a brake and when a brake is put on the groups would open , now it only took a minnor change in the load to get the groups back down. I have also seen a gun that shoots 1" groups with a load drop the groups down to sub 1/2" when a suppressor was added.
So the bottom line is yes their is a good chance that you rifle will change its grouping habits with loads worked up for a barrel with a brake and without, but adding a brake to a rifle "PROPERLY" would make one shoot any worse , and generaly if its a good desgine that doesen't blow alot of crap in your face your groups will shrink becasue the gun kicks less