But then TBH why would WOTC really protest if they publish Mercer worldbooks and it's probably biggest advertisement of D&D ever
Because, according to US IP law, if you don't defend trademarks, you lose them. So the moment WotC turns a blind eye to one group using beholders, everyone can use them.
Beholders fall under copyright, though, not trademarks, and copyright isn't lost if ignored. It's also very easy to file the serial numbers off of beholders and illithids and avoid the problem, so even a victory wouldn't win WotC much.
Eyeball monsters go back to at least The Trollenburg Terror from 1958, and possibly further back. Beholder-like creatures, minus the extra eyestalks usually, are ok. Just don't call them beholders or put eyestalks on them. IP infringement avoidance 101. Same with illithids/mindflayers. Squidhead monsters go back at least to Lovecraft and probably further back too. Again, just don't call them mindflayers and so on.
Its not that you need to file the serial numbers off TSR critters. You just have to find the source and use that or do like TSR oft did and rename something.
There's also just the fact that, if you're not explicitly trying to sell D&D, there's no real need to use those specific monsters when much more commonly known ones like dragons, vampires, zombies, trolls, ogres, giants, demonic dogs, etc. are available.
I have no idea what turned up in later arcs of Critical Role, but all the stuff in season one of the animated series was pretty bog standard critters you'd find in any fantasy world (i.e. dragons, undead and giants) and some civilized races that you frankly don't see in bog standard D&D (wolf-men members of the rival mercenary gang in the first episode for example) so it'd be very far fetched for WotC to even think about trying to hit the show with anything.