Here are the places I would hang out at or swim at on the cliffs:

Xtabi - very friendly, good food, prices medium to high

Samsara- I didn't find them unwelcoming to me, ordered a drink, but that pool and jacuzzi really needs work..jacuzzi was nasty.

Rockhouse - As long as you buy a drink or two or meal, they are fine with nonguests swimming in the cove at the restaurant area, prices were high, and while I have always had good meals on my past trips, do NOT recommend their "lobster roll" or whatever it was..blechh..

Negril Escape - Got a drink and went swimming, they had no problem with it at all.

Whoopie's Hammock Park - Really nice laid back place, there was a place down to go swimming from too.

LTU - good views, good food, prices medium to high

Firewater - Right on the cliffs past Rick's, FANTASTIC VIEWS, ordered a drink, priced decently

Rick's - What can I say, it's a tourist trap, but still worth checking out, ordered a few drinks and swam around a lot, prices are high, only would get a drink or two not really my scene for food, and it's a great place for people watching

Seastar - not right on the cliffs, but still more than worth going to, good food, good vibes, good folks, nice pool with swim up bar, some of the best bartenders on the island imho, and definitely need to go to the Saturday show and buffet!

Sips and Bites - Authentic Jamaican food, prices VERY affordable.

DeBar - Okay, I admit it, my favorite spot to hang out when on the West Side. period.