It is probably the stock and/or the recoil pad. One day I shot a Ruger 7mm mag and a Remington 30-06. They should have had very similar recoils, but it felt (perceived recoil) like the 7mm was a lot more. I believe that you may be experiencing the same thing.
"When working with the public, there are two things you need to remember. - 1. The public is a bunch of ignorant morons. - 2. YOU and I are one of them!"