These are the places I have been able to swim at if I have had a drink or two..or a meal..most places are okay with you using their pool as long as you ask first and purchase something:

Cliffs: Seastar, Xtabi, Sunset on the Cliffs, Rockhouse, Oasis, and Samsara

Beach : Shields, Travellers, NBCC, Fun Holiday, Seasplash, and Treehouse

There may be more of course, but those are the ones I have swam at.