I was doing a lot of swimming and snorkeling around 3 Dives last week (ouch that hurt to say that since I'm back in Frost-bite Falls).
I saw at least one staircase down to the water at 3 Dives - I missed the cave though.

I was staying at Xtabi and there were outside people coming in every day to swim.
Gerryg even had his raft stored there and "ground" on Omar when he couldn't find it.