What you are experiencing is fairly normal for the most part. Whenever I go to the range to practice, or when developing a load, I always take a few " fouler" shots before getting serious. Most factory, and even some custom rifles
are that way also. As far as hunting, and with that knowledge that the first shot, cold and clean is going to be out of the group, just sight your rifle the day before, or just shoot a few foulers. That way, the bore is " slightly" fouled. I have done that for years anyway. I wouldn't worry about it, or think into it at all. Enjoy!!!!