on the cliffs: Seastar, Rick's, Oasis, Sunset on the Cliffs, Xtabi, and Samsara.

on the beach: Shields, Travellers, NBCC, Fun Holiday, and Seasplash.

Those are the places I have personally swam at that as long as you have a drink or two or a meal have no problem and welcome visitors.

A really good snorkeling spot too right off the cliffs is that little cove area at Rockhouse, again just order something, but their pool is for guests only.

Some places I have not mentioned I have swam at, but since they officially don't allow visitors as overall policy, even if exceptions are made, I try not to advertise that.