If you swim in, you can enter Xtabi through the cave - it's very easy.

In high seas the ladder might be rough but again the cave is going to be an easy way to get up.
For what it's worth . . .