I haven't ever taken mine off for cleaning purposes. I figure that if I don't have to clean a centerfire rifle suppressor because the muzzle blast is supposed to keep it from fouling, then something similar likely applies to my muzzle brakes. On the brakes with large vertical slots on the side, I do try to pass something through and clean and dry, but it is mostly because the solvent and powder I push into the brake while cleaning has a tendency to keep fouling my patches. This makes it harder for me to tell when the bore is clean.
I'm not saying taking the brake off to clean it and the muzzle would be bad, I just haven't notice any ill effects from not doing it.