I agree; lighter is preferred, but not at the expense of accuracy. Physical condition is critical. If you are in good shape, it is a whole lot easier to carry anything. I guide hunters and most can hardly carry their own body when they come to the Wyoming higher elevations. I end up carrying rifles (and everything else) for a few folks every year.
What I do personally depends upon the hunt and terrain,. My dad always taught me to carry my rifle in my hands. That's the absolute best way to protect your rifle and scope. I have seen a lot of falls through the years where scopes and rifles were damaged. And as has been stated, always carry your rifle in your downhill hand. That way, when you fall - and everyone eventually falls in tough country, your rifle is away from the crash and you can put your uphill hand down to "save" yourself.
But in steep country or on long hikes, I slip my rifle into a pack. It is a lot easier to carry it in your pack. I also have one of those rifle covers that I put over the rifle if I'm just hiking to a place to hunt. But that is a matter of preparedness. If you are going to need to shoot quickly, having the rifle in a pack really slows down the process. And as has been said, in really tough country, using trekking poles mandates that the rifle be in a pack.
That's the long answer. The short answer is it depends. Depends upon you, your hunt, the terrain, the rifle and your preferences. Best of luck.