Most LR match shooters won't mix lot #s for primers (or anything else for that matter). Some will even endure the tedium of weight sorting primers for a match.
I expect that very very few shooters could ever actually realize a demonstrable difference, but it makes guys feel better. To "feel better" is not necessarily a bad thing in match shooting; confidence is important.
For the purposes of hunting, I don't think it matters a whit, but staying within the same lot # is easy enough to do if it makes you feel better.