Yes, the visual cues this game provides are fairly misleading at various points! While being physically bigger does a lot to indicate that you're more powerful in some way, there's no way to really tell whether you're safe or not per creature. (Though like so many things at this point in game history, it could simply be a matter of seeing which ones are safe in the manual.)