I thought I'd update this with results.
After the ladder test and seeing your input I decided to load the following:
165gr Gamekings 2.830" OAL (.027 off lands)
1- 45.7gr = 2666 fps
2- 45.7gr = 2677 fps
3- 45.7gr = 2682 fps
4- 45.7gr = 2688 fps
100 yard group resulting from these shots: 1.037"
(I was also going to load up some at 44.1gr also, but ran out of Varget so I put it off. Glad I did because I've since decided to stick with the heavier .308 bullets)
180gr Gamekings 2.820" OAL (.027 off lands)
1- 43.1gr = 2510 fps
2- 43.1gr = 2506 fps
3- 43.1gr = 2515 fps
4- 43.1gr = 2510 fps
100 yard group resulting from these shots: 0.692"
5- 44.3gr = 2594 fps
6- 44.3gr = 2588 fps
7- 44.3gr = 2583 fps
8- 44.3gr = 2588 fps
100 yard group resulting from these shots: 0.650"
After looking at these results and plugging all velocity and ballistic coefficient info into dropcharts (shows heavier bullets have significant downrange advantages) I'm giving up on the 165gr bullets.
I then loaded up another set of the 44.3gr rounds with OAL's .02 longer and .02" shorter - the original 2.82" load outperformed both, so I'm sticking with the OAL of 2.82".
I took this ballistic data and printed off a dope chart and loaded up another 20 rounds. Took out the 10" metal gong target and shot at 300, 400, 500 and 600 and the dope chart is scary accurate up to the 600 yard max range that I shot it at.
So, I've got a good, accurate 180gr hunting load and long-range confidence.
Since the Berger Hunting VLD's have had good reports on game, and a bit higher BC than the Sierra Gamekings, I figured I'd work with some of the heavier VLD's. Did a similar ladder test and load workup with the 185gr Berger VLD's and have worked up an even more accurate load averaging 0.561" at 100 yards. I've ordered some 200 gr Gamekings and some 210gr Berger Hunting VLD's - when they get here I'll do a similar ladder test and load workup on both of those.