Outside of the AI's, this is standard restaurant wear. When eating dinner at a nicer establishment (like Ivan's or Le Vendome), I usually throw on a collared polo shirt. However, you will never catch me wearing shoes or long pants in Negril. After all, it is the Capital of Casual.
As far as drivers go, don't feel bad about not using the ones affiliated with the resort. They understand that many people have their favorite drivers and will make their own arrangements. Plus, you will usually get a better rate using any of the drivers on the "TRANSPORT" link at the top of this page. I can personally recommend Linston, who will provide you with a cell phone. I have also used Michael's On Time Taxi, and he is very good.