Y'all nailed it if you want peace & privacy = Cliffs (can't say quiet cuz some of the places on cliff road can be noisy at Night).
I stayed at Banana Shout in 2008 and had the Red Rooster room at it was too close to the road, could hear cars and dogs barking all night!

I prefer the beach because I like to be able to get up from my lounger and walk a few miles (safely) and I like the action of the beach too.

I loved my stay at SeaStar but wound up going to the beach everyday anyway. Cliffs are great if you want to stay put and chill otherwise you have to take a taxi everywhere. That being said I would stay on the cliffs again but I'd prefer being right on the sea (nothing against SeaStar).

It really depends on what u want out of your vacation, IMHO.