You are getting 1'' groups with factory ammo at times. You definitely have a great factory rifle and I want to echo the other replies about handloading for maximum accuracy. If you are going for ultimate accuracy you really don't know what you have until you reload your own ammo tailored to your rifle!!!!!!!
I had purchased a used rifle that shot 1" groups with factory ammo. After taking the time to produce high quality match ammo the group sizes shrunk to .25" inch groups. .25" groups out of my 7mm-08 is one ragged hole :-) I definitely didn't change anything in the rifle after that as I didn't want to spend alot of money and screw up my rifle.
I would spend the money on glass, then a mc millan stock
or equivalent that you like. Custom reloading requires an FFL so not many around.