Depends on the rifle. Something nice, and traditional or vintage, then, yes, it will lower the resale value. For the most part, people who will pony extra up for very nice wood, or a classic or vintage rifle in good shape, are into the aesthetics/traditionalism and don't want to see a a brake on the muzzle. For example, a Winchester M70 pre-64 in very good shape - for sure, putting on a brake/suppressor will lower its resale value. The same can be said for a nice old Sako or Weatherby.
Whether you are willing to take the hit is a different question and is solely your decision.
Of course, there are always exceptions. And, people are always happy to trash someone else's rifle.
JMHO