I use a switch barrel system that is probably what you are envisioning. It started out as a Remington 700 in 243 with a varmint weight barrel. I have had 3 other barrels done for the rifle in various calibers.
There are positives and negatives to the concept. The biggest difficulty associated with a switch barrel setup based upon anything other than Savage actions is that it still takes a gunsmith to initially set up the barrel. The second drawback, as has already been mentioned, is that the rifle effectively becomes a single shot without quite a bit of additional time and money expended, at least most of the time.
It is also a bit of a pain to be switching barrels, stocks, and scopes on a whim because of the problem of rezeroing. Once I even grabbed the wrong ammunition because I forgot that I had switched barrels the previous time out shooting.
The only way I would consider it a viable option is as a special purpose target rifle similar to the function filled by the Tubb 2000 rifle. For anything else it such a pain that I would much prefer to have multiple rifles. In case you were wondering, my switch barrel setup is a tactical style stock similar to the McMillan A4
except for being made from a laminate blank. The action is a Remington 700 short action with a single shot carrier welded in place. The barrels are the factory 243 varmint barrel, a 22-250 varmint barrel, a 22-250 Ack Imp 1-8 twist target barrel for 600 yard target use, and a 6.5-08 (260 Remington) 1-8 twist barrel for 1000 yard target use.