First off, the SPS is not designed to be a precision rifle by any stretch of the imagination. If you call Remington and tell them you're getting 3" groups with your SPS, they will not be real concerned about that, especially if you tell them you've been handloading. It could be your scope and it could be a little flinch, but if you have 20 years of handloading experience, I'm guessing you know how to mount a scope properly... maybe the scope itself is taking a beating? What scope do you have?
On the recoil.... I use a slip-on recoil pad with my Sendero 300 Rum (which is 1 1/2 lbs heavier than the SPS) and I can shoot my RUM all day, no problem.
Back to the basics... the SPS is NOT a precision rifle and Remington doesn't advertise as so. They do build some good shooters like the Senderos, but for the most part, they are not concerned with sub MOA performance. I have a Sako 300 WSM that shoots like crap, so I know how you feel. At least you paid less for your headache than I did.
If you want that Rem to shoot, get rid of the barrel and get rid of the stock, and maybe get rid of the bolt... and maybe get rid of the trigger - I got my Rem triggers breaking very nicely at About 1 1/12 - 2 lbs.
Probably not what you want to hear... but there it is...