Personally, I would look at some better layering gear if I were you. Get some of the Under Armour 4.0 Base Layers, then throw on another solid layer like a tighter fitting 1/4 zip. If you have wind resistent outerwear coupled with this, you should stay plenty warm. With all the technology in the layering gear, I don't see any reason to be buying a size larger in your outerwear.
Here is the UA 4.0 at Cabelas: Under ArmourŪ Base 4.0 Crew : Cabela's
That stuff is WARM!!!