All legal Taxis have red plates.
Route Taxis will charge the flat route taxi fare for pickup\drop off anywhere along that taxi's route. They will pick up other passengers along the way so it is not a private ride and they will not deviate from their route. If they reach the end of their route then you will need to get out and find a taxi that drives the next route in the direction you are going. The route of the taxi should be marked on the side of the car. You can think of route taxis like buses.
A private taxi is any taxi that you hire to give you a ride based upon the terms you both agree upon. The ride is typically private and you can negotiate a ride to\from anywhere on the island. You can even negotiate a ride with a route taxi. Anytime you negotiate with a route taxi driver you are actually hiring them as a private driver for the time you need them. There is no need to negotiate with a route taxi driver if you just want a ride along their route and pay route taxi fare.
In my experience route taxi drivers have smaller older cars but you can typically negotiate a private ride for a little more than standard route fares would be for the same trip if its local. Taxis drivers that don;t drive a route usually have nicer bigger cars\vans but may not be as negotiable. If you're walking or standing waiting for a ride it is most likely a route taxi that will stop and ask you if you need a ride.
To be safe you should pass on any taxis that drives a route outside of the area you are traveling in (i.e. make sure the taxi drives a Negril route if in Negril).
Hope that helps