Seems to be several parts missing in the puzzle
1) what is the scope magnification range
2) Does it have an AO or a side focus ?
3) what rest system is being utilized
4) What is used as the aiming point on the long distance target ? For that matter what is it on the 100 yd target ?
Generally, MOST non parallax adjustable scopes are SUPPOSED to be parallax free at 100 yards. That happens to be the most commonly used sighting in distance and the manufacturers may assume that the users are also unlikely to take a shot much further than that.
I have personally bought a 3-15 Minox scope that was quite expensive and it had no parallax adjustment and it was way off at 100 yards. So much so I got a headache trying to "split" the crosshair shift to even shoot a group. So if one can have that in a $700 scope, it is entirely possible that you may have a significant parallax problem at 400 or 500 yards. To check for it, you need the rifle placed on a firm rest system so it will not move and pointed at a small aiming point at the distance of interest. Now without touching the rifle at all, move your eye left-right and up-down and see where the cross hairs go. If the crosshairs move relative to the target it means that the image plane and the reticle plane are not coincident and therefore when you move your eye position, the reticle moves relative to the target image.
Parallax can be a big deal, there is basically no limit to how far you could be off (other than the field of view of the scope) if the parallax is off far enough. After my Minox experience I will never again buy a scope that does not allow for parallax to be corrected. The Nikon Monarchs I had prior to the Minox have a side focus and the Vortex PST's I have now have the same.
The next point is that for target shooting, you should try to use an aiming point of consistent subtention. I use a 1" black circle on a white sheet at 100 yards. At 320 yards, I have a 12x12 white painted steel plate and I spray a black 3" circle on it for an aiming point. So the view of the aiming point is the same in the scope at both ranges. If your eyesight is better than mine and your reticle thinner, you could use a smaller aiming point, but try to keep its size in MOA constant. Otherwise you might have a very hard time seeing the aiming point at 500 yards, or if for example you use no aiming point (hold for the "center" of a plain target) one usually will not shoot as consistent groups.
If the magnification of the scope is low (I see a LOT of tactical type outfits with a 1-4 or 1.5-6x scope) and they certainly do not help to get the best hold on a tiny target at 500 yards. I myself have a 1.5-4.5 scope that I use for night coyote hunting and my groups at 100 yards suck compared to if I was to use a 16x scope. But this is clearly a compromise between the ability to see the reflection in the eyes of a coyote with a red predator light, and having an acceptable field of view (situational awareness is very important at night in the inky blackness). For any task requiring more range and better accuracy, higher magnification is the way to go. Its not accidental that most of the long range scopes are 20x.
Often one will use a different rest for 100yd vs 500 yd shooting since in most cases they are not at the same venue. If you are lucky enough to have a 500yd lane at the range where you usually practice, then you are truly blessed. Me, I have to go out to a farm and shoot from a prone position, since there is no suitable shooting station. So I shoot from a bipod, from an unimproved "field" (top of a little hill) and have to pick a spot between the sheep and horse manure...