The swipe fees (paid by the merchant) are composed of a switch fee (goes to the network), plus an interchange fee (which goes to the bank that issued the card). They can be quite substantial, ~1.9% on average.

Interchange fees in the US are some of the highest in the world, and rising. They are a lucrative revenue stream for banks (iirc, ~$40 billion annually). See this piece by Felix Salmon for more: http://blogs.reuters.com/felix-salmon/2010/01/05/the-interch...

Thanks to the passage of Frank-Dodd, they should be dropping soon.