Probably the biggest reason, at least on this forum, is that there aren't a whole lot of smiths that work on them, and most guys doing the LR gig are going to want work done. Having said that, for a factory gun they are nice.
The Vanguard line is a Howa action and barrel with an upgraded stock. Howa's are probably the best bang for the buck IMO in a factory gun. I have had a few 1/2 MOA shooters straight from the factory, some even better. The problem for many guys looking at Weatherby is that the Howa can be had for much cheaper and if you plan to do a stock upgrade or spend the money to bed/float it why spend the extra money for a Weatherby.
The Mark V's are very nice and very tough actions. But they aren't perfect out of the box and often need some work to get the real tight groups desired for the LR game. If you are going to start spending that kind of money the semi customs like Cooper and even a full custom gun starts competing much more.
Also, there are very few heavy barreled versions in any line which keeps them from being pure LR rigs.
My take anyway.