I make brakes to play with and test. I made one the other day that vibrated so bad the brake came loose. I tightened and fired it again and it came loose again. The rifle vibrated against my face like an advance martial arts guy striking with a bamboo cane. They abruptly stop a couple inches from the target and the cane beats the tar out of the subject with three or four whacks.
This last one was for an accuracy test. My gunsmith told me if you leave too much at the muzzle end, you loose accuracy. I left about 3/4" of metal on the end. It didn't vibrate as bad as the other but did vibrate. I cut another slot in it to see what would happen. Adding the forth slot cut the vibration in half or maybe even more more. So much for verifying the 'smiths theory.
I will make one with holes all the way around for the accuracy test.
I apologize for getting your post off on a tangent. I will take a stab at your problem and perhaps some more knowledgeable folks will chime in. Obviously it is an exaggerated harmonics issue. I have seen slow motion videos involving brakes with top ports that cause a disconcerting amount of barrel flex. Check out some of the videos on; jecustom.com
I don't know what configuration your brake is but this is one possibility.
Thanks to everyone that thinks I can solve the problem. I will try.
There are several things I would look for.
But first, the length past the barrel has nothing to do with the vibrations. people shoot suppressors
that are 12'' or longer and don't experience anything like that and large brakes actually dampen barrel harmonics.
1 - I would suspect you are having bullet strikes because=
If the bullet strikes the brake you will have all sorts of vibrations and that could be from an improper bore diameter It should be .020 to .030 larger than the bullet(To small for the bullet to clear if there is any misalignment of the threads), an improper barrel tenon (Not concentric with the bore) poor thread concentricity with the bore.
And with the bullet striking the partitions between the holes or ports it would get worse as the bullet proceeded down the Brake because the error would get worse and shave bullet jacket at every baffle.
And as far as the brake unscrewing, if the bullet is striking the brake, bullet rotation from barrel twist may be unscrewing the brake.
To be sure, I would have to actually look at the brake and the installation. But if you unscrew the brake you should be able to see bullet strikes looking from the back of the brake.
Muzzle work after the bullet is long gone so there shouldn't any vibrations if installed correctly
and are machined correctly to prevent an imbalance in the gas discharge.
I once repaired a ported brake that was installed with all of the bore holes drilled to proper size except the one nearest the muzzle.(The drill/reamer was not pushed through far enough to pick up the last hole and it was 1/4" and the bullet was a .308.
So check the bore all the way through for everything I mentioned and maybe you will find the problem.
If you can send us a picture of the brake and maybe that will help.
PS ; don't shoot the rifle until you figure it out what the problem is.