Make sure your mags are for a 6.8 spc, you may have to try different brands. Then try this test, with gun on a bipod or rest lock the bolt open. Load a mag with three rounds only. With rifle pointing in a safe direction load the mag in the rifle then press the bolt release and see if you get...
Yes remove pin first also Brownells has a tool that fits in the extension bolt lugs. I usually take a impact wrench with the tool installed and remove the extension. Or better yet there are a lot of great deals on AR barrels right now if you do not have the tools and reamer and a lathe.
I have a old 742 30/06 carbine that would not shoot well so I took it apart and cleaned the bbl. Just by chance I measured the id on the muzzle with some of my chamber reamer bushings and found the bbl was belled at the muzzle end. So I cut the bell out which was about .250" long and recrowned...
Nvision Halo LR if the budget allows it or Super Hogster or Super Yoter from Bering Optics are good for the money.
Also call and talk to a human for the best price on whatever scope you decide on.
Once you get a can you will find you need more, but a .30 cal is a good start. You might as well get one for a .22 rf as well, they are just too much fun to ignore.