Some taxis drivers have set up a makeshift drop off/pickup area for taxis near the Scotia Bank...every so often the taxis are kicked out of the parking lot by authorities...but eventually make their way back there.
Most tourists tend to go to the HiLo, Scotia Bank area or up to the West End rather than travelling the other way towards Sheffield, Little London, Sav La Mar etc. The driver may be assuming that's the direction you would choose and is offering a more convenient stop for you and for him...this reduces a bit of driving time.