Yep, you'd miss.
I've run into the same thing shooting at chucks from the top of my truck, off a bipod. Just the opposite shooting off the hood, I never miss from there.
One bag firmness to another completely changes things as well.
At a local annual shoot we're challenged with hitting a 1"bull at 200yds, with a single cold bore shot. With each hit, you move to the next round(and the back of a big line). A miss, and you're eliminated. One winner takes all eventually. Simple.
Except, you can't use a rest other than a feed sack thrown up on the table by the event holder. He changes up from sand, dirt, to wheat, or whatever, and you never know what to prepare on. So you never know where exactly your gun will be shooting until you try it that day. You also can only guess about temperature beforehand.
I went this year with the same gun/load that I won with last year. Fully prepaired to shoot off a sand filled bag(as before). I could nail the bull with 50 cold bore shots if needed. But when I got there the bag was full of wheat, and my gun hated it. Knocked out in the 4th round..
A Savage won with Hornady ammo! That gun just loved that bag.