300RUM, with muzzle break on it shoots 2.5-3" groups. Muzzle break off, it shoots 1.5" groups, but also shoots 5" high. This is using Hornady factory ammo 220 eldx.
The fact that the point changes is not at all surprising with it on and off since harmonics are changed.
As far as grouping, it could be the same issue and you need a different load or there could be a problem with the brake. Has it always acted that way or is this something new?
Since hanging a weight on the end of the barrel changes the barrel harmonics, you may have to try some other types of ammunition to see if you can find something your rifle likes.
The other possibility is the muzzle brake is interfering with the bullet.