How much to "notice" can't say. Everyone handles recoil differently. You could be a 500lbs 4'9" tall butterball of a man (spherical but quite pointy in places) or you might have a physique more like mine and every other average guy, 5'10" @ 170lbs and strong but not overtly muscular or anywhere in between. Your rifle could be a fixed emplacement weighing many hundreds of pounds or it could be a 6lbs maker of recoil formed human tacos. When shooting something like a 10lbs tactical .308win I will EASILY notice the difference between a 110gr and a 150gr and a 150gr from a 180gr. I cannot really tell a 168gr from a 150 and I can only just discern it from a 180.
When shooting a suppressed .338wm I can't tell the difference between a 250gr and a 225gr but a 200gr and a 250gr are notably different to each other.
So, I'll say when looking at power levels that are up in high power rifle territory, 50gr of bullet weight is really quickly noticeable. Noticing so much that you start praising the Flying Spaghetti Monster for reaching down with his noodly appendage and taking the pain away, that's going to come down to the ratio of heavy bullet recoil to light bullet recoil and what recoil velocity is produced. Recoil Velocity matters more than recoil energy here... You can take a million pounds of recoil if you do it over a long enough time. The shorter the time the more violent the sensation.
So let's calculate recoil velocity: Add up the total weight of the powder and bullet. Divide that by the weight of the rifle. Multiply the result of that by the velocity to get recoil velocity. Compare the recoil velocity of one against the other. Now you know how fast your rifle will slap your shoulder and that should be a decent idea of how much it'll hurt compared to a different load in the same rifle.