You don't have to give every time, or at all... or just give what you would over a series of "hat passes", like 2 bucks at a time for a total of ten.
I always throw in a few dollars each pass, and maybe a little more if it's a guest singer I really like. Sometimes I wait until the end and put in 10 bucks. I've seen the money bucket poured directly into a backpack before, which make me think the money goes right to the band.
I definitely see your point of view, though.