No problem swimming at Xtabi, using the loungers, restrooms, etc. Like mentioned, have a few drinks or some food while you are there.
I wasn't much of a swimmer there since the sea was constantly forcing you back against the sharp coral. I like the idea of the blow up raft. Do the bartenders deliver?