You could always look into taking the Southcoast Express bus to/from Kingston and then hiring a local driver once in Kingston. However, the travel time each way on the bus shows 5 hours (but they do say it could be less, they just account for traffic and stops).

http://southcoast-express.com/services.html

Whether you hire a private driver or take the bus, it's going to be a long day for sure. One of these trips I'd like to spend a good week on that side of the island and do some exploring. Have a fun trip.