I too have never been to Port Antonio but if you think you got the urge, then do it. WORST that could happen is you don't like it as much as Negril. But you will get to see another side of Jamaica and her people. If you don't go, you will always wonder. If you don't have a lot of luggage and feel adventurous, maybe you can explore various options for transportation.

This looks interesting... http://www.portantoniojamaica.com/ivey.html
http://www.portantoniojamaica.com/maroons.html

I think I just might have to look into going there myself.