Re: Very sensitive barrel.
Have you by chance pulled your rifle apart and checked your stock? My dads 06 did that though not as bad, turned out the stock was cracked right behind the recoil lug. Going off of that it seems to want to return to where the 1st shot went after cooling I'm betting you scope is fine, and your barrel is fine. Gut feeling is something is loose some how, and it's at, in or very very near your receiver.
Keep in mind the animals we shoot for food and display are not bullet proof. Contrary to popular belief, they bleed and die just like they did a hundred years ago. Being competent with a given rifle is far more important than impressive ballistics and poor shootability. High velocity misses never put a steak in the freezer.