Ya, Priority Pass only gives you access to the lounge, not the fast track on arrival. Which pretty much makes it useless for arrivals.
You also can't book it ahead of time using Priority Pass. When departing you're stuck crossing your fingers and hoping that they're not full. It's nice in there if you can get in, though.