I find it's because everyone is saying to me "Oh, yeah, before you go on vacation I need you to do one thing for me..."

I have learned to schedule an extra day off before I go, just so I can cancel it and actually come in to work to do all those extra last-minute requests. So if I'm leaving Friday AM, I book Thursday off, then actually work Thursday to do all the stuff that came in Wed. and it's not crazy.