Rather than xenophobia, I think, good cosmic horror and weird fiction practice something closer to exophobia–this fear of things outside human perception or outside what it means to be intrinsically people.