wondering what is the difference is in ballistic performance
Entirely dependent on barrel length, action, bullet weight, barrel twist, etc. There is such a wide range of all those out there that there is no way to broadly compare the two without going into at least some level of detail on intended use.
77gn 223 Rem from a 16# 26" barreled bolt action will most likely be a lot more accurate than a 55gn 5.56mm from an 5# 11.5" barreled AR. But that has nothing to do with the cartridges and everything to do with the platform.
most surplus ammo is no good for reloading.
Lake City cases can be used for reloading, but loaded ammo in the common civilian-labeled variants of M193 and M855 are pretty lousy in terms of accuracy and precision. I have a 6 Mongoose that uses 5.56 LC cases that shoots exceptionally well and proves out that LC brass is capable of good things, but I don't typically reload those cases as a standard .223 Rem or 5.56mm NATO other than to run through ARs. Easier to buy Lapua and start off without a crimp that has to be cut out, cases known to be matched from the same lot, all the normal reasons factory fired brass isn't the ideal source for any reloader in any chambering.
I know that varmint hunters seem to prefer the .223 because it is more accurate (so I have been told) and it appears to be more small game hunting ammo available in that caliber.
Base accuracy is more likely a by-product of the 5.56mm being primarily used in ARs and not in bolt action rifles. It's cheaper and easier to make a bolt action more accurate than an AR. Either platform can be built to shoot exceptionally well, as proved in Service Rifle competition and by countless varminters using ARs, but those aren't the rifles you find on the wall being sold as mass-produced sporting rifles. At equal price points for fully assembled rifles, 223 Rem chambered bolt actions are likely higher quality than 5.56mm chambered ARs. Also there is a slim-to-none chance of finding a 223 Rem bolt action with a chrome lined barrel, which is somewhat common in 5.56mm ARs.
Hybrid and wildcat chambers like the 223 Wylde, 20 Practical, and several other AR-specific variants have eclipsed the 5.56mm/223 Rem in the AR platform with significantly improved performance and part quality, to the point I don't see a reason to ever build a 5.56mm NATO chambered barrel.
Ammunition availability would favor the 223 Rem as there are more hunter-focused loads for it versus 5.56mm with it's sporting rifle focus, but you can use 223 Rem in 5.56 so there's not really a mark against the 5.56mm in terms of availability.
Do these calibers share any uses such as coyote hunting, or other small game.
Yes, they can be used for predator hunting, up into smaller ungulates, and can be great on feral hogs specifically. 223 Rems don't seem as popular in predator hunting as the 22-250, 204 Ruger, 6 Creedmoor, etc that shoot faster and flatter, but does still function perfectly fine in those roles.
My personal 223 AI is a 7 twist designed to shoot 77+gn bullets out to 1000+ yards, and I would have no reservations shooting anything up to Texas whitetail with it. Since it's a fast twist it wouldn't be my go-to for varmint or predator hunting, but if had been built in a 12 twist it could be ideal for that.
What I am really after in all this is--how do these rounds differ in performance?
Practically speaking, they don't - if you compare apples to apples on quality of rifle and ammunition. A well built rifle with quality ammo will shoot as good or better as the shooter can, in either loading spec, in a bolt or AR platform. There is simply not enough difference between the two to matter at all, especially in the hands of someone who can load for them.
