It could be the scope but probably not or there wouldn't be consistency in your group patterns. I have seen this once before, and it was a barrel shoulder issue(not your problem but similar symptoms). My advice is to not baby it, but heat the hell out of it to see if it gets progressively worse. 10 rounds or so as fast as you can accurately hold MOA or better and see if it strings farther down or goes all over. It may stabilize after a certain temp or you might eventually get a internal burr smoothed up and fix itself by fire lapping. If it is a bad barrel then there is only one fix unfortunately, but you really have nothing to loose and it will take more than a few 10 shot strings to burn up a barrel.