It's a wee bit more complicated than that. If you take a 12 ga., and a 20 ga., and the guns both weight the same, and both have the same amount of powder (FPS of the load is the same in both guns), same wad weight, same pellet weight, and same forcing cone length, then the 12 ga will kick less due to the larger bore size.However, you are right in stating that the recoil impulses with a 20 gauge has less magnitude that a 12 gauge, less force applied to a smaller object. Simple physics really.
Then toss in gun fit. A shotgun with more drop at comb and heel vs. a shotgun with minimal drop at comb and heel will usually have more felt recoil because of it wanting to pivot up and into the shooters face.