As a SF editor (with a sister fantasy magazine, so genre is pretty rigid), I think about this topic a lot. And I realized something this morning about monsters/tentacled beings/etc:

If they’re on another world/in space: science fiction

If they’re here on earth: fantasy

If they’re overly violent/bloody: horror

I’m not sold on the idea, but it seemed… odd but fitting.