Y'all nailed it if you want peace & privacy = Cliffs (can't say quiet cuz some of the places on cliff road can be noisy at Night).
I stayed at Banana Shout in 2008 and had the Red Rooster room at it was too close to the road, could hear cars and dogs barking all night!
I prefer the beach because I like to be able to get up from my lounger and walk a few miles (safely) and I like the action of the beach too.
I loved my stay at SeaStar but wound up going to the beach everyday anyway. Cliffs are great if you want to stay put and chill otherwise you have to take a taxi everywhere. That being said I would stay on the cliffs again but I'd prefer being right on the sea (nothing against SeaStar).
It really depends on what u want out of your vacation, IMHO.