mmk, your process of elimination has left only a few options. I suspect the barrel is installed crocked in the receiver. This a common problem with production rifles. It's easy to confirm this problem when the impacts are left or right. I simply place a metal straight edge across the receiver and measure the distance to the barrel 8-12" down the barrel. I do that measurement on the left and right side, and then calculate the difference.
The recoil lug makes this measurement difficult for vertical misalignments. You can try to do a measurement off the top of the receiver (with the rail removed) to the center of the bore. Do that measurement at two locations down the barrel with a 36" aluminum yard stick.
If the rifle shoots tight groups, I would keep it and just change out the rail. The aluminum rail on that model rifle is pretty cheap. I would replace it with a steel one.
It's remotely possible that the bullet is impacting the brake, or there is asymmetry in the brake vents. Pull the brake and inspect the interior carefully. Then shoot the rifle without the brake and check for a large change in POI.
A 338 LM without a brake will not hurt you if you use some extra padding. I've shot a 340 Wby without a brake prone and sitting. The 340 Wby MV is only about 100 fps slower than a 338 LM. Just shove a folded pair of leather work gloves between the buttstock and your shoulder, and you will be fine.
