I don't think there's a big technical issue in scaling SMTP sends; you can have N machines doing it in parallel and it will work (almost) N times faster than one machine. There IS the question of minimizing your cost per unit email, and the real spammers address that by building SMTP senders that don't comply with the standard.

What does bug me about email is deliverability. It's pretty much impossible to send e-mail to AOL members, for instance, if you (1) don't pay ice to AOL, or (2) are big enough that AOL is afraid you'll sue them. AOL has shrunk a lot lately, so this isn't as big a concern as it was years ago.

It's not hard to get burned by other organizations as well. For instance, I sent an email shot on the the behalf of a campus organization at a university from off campus... an opt-in list of 2500 subscribers, really just chicken feed, but only 1999 went through -- after 2000 connection attempts, they firewalled my IP address and for all I know that address is still firewalled today. I knew the guy who runs email for that school by name and refused to talk about the whole affair... That's what you're really up against.

We used two small Sun E250s to send upwards of a million emails a day using qmail and a tweaked version of the Solaris TCP settings to close connections as quickly as possible. These two servers also handled the bounces as well as the usual spam attempts. If I was doing something similar again I'd still used the Solaris/qmail combo as it was very reliable.

Our biggest problem was with people marking us as spammers even though they had to sign up to groups to use the service but that's a constant headache in this game. We also had some shell scripts that culled SMTP connections (either incoming or outgoing) that had been help open too long as that helped us deal with DOS attacks.

The site also throttled SMTP traffic to give HTTP traffic priority if required as that's OK for SMTP with its store-n-forward architecture but not for HTTP

Solaris is very good, and the x-BSDs often do better if you're a free software fanatic.

Practically you might find your CPU utilization is 2%, but the load average is 50 and you do 'ls' on a directory that has 20 files in in it and it takes 30 seconds.

We made heavy use of Mirapoint. But we also made use in different places of qmail and postfix. At the high end of scaling SMTP routing starts becoming similar to routing IP traffic. You're just not going to beat a dedicated device like Mirapoint with some Unix box you make from scratch. You need dedicated and differentiated mail routers for inbound, outbound and mail storage. You monitor their usage and when they get overloaded you just add more. Since everything is load balanced you can scale close to linearly. We used SAN for storage. Local storage just doesn't work.

Mail hits an inbound SMTP router and that router does an LDAP lookup to find which storage box actually handles hat account. Then it forwards it. That's all it does and it does it fast. Each storage box handles mail for the accounts it stores and IMAP/POP3 access for those accounts as well. The outbound routers just take mail from the storage routers and spool it for outbound.

I could go on but I think you get the idea. The main idea is to put everything behind a load balancer that you can so that you can scale it linearly as much as possible.

Can you explain why?

I imagine Gmail does something similar. I bet if every Gmail user suddenly started using 100% of their email storage Google couldn't cope. But Google knows this won't happen.

Over subscription is how you make money in the end. Getting better at figuring out exactly how much you need when you need it directly affects your margins.

I am sure every emailing receiving server has its own darn rules, so how do large cos manage sending emails to all such services without getting blocked or by adhering to the random rules of each of the email receiving servers?

There's a size range between the little guy, who has no need for large, scalable email delivery, and the huge players who don't get shut out because of their name. For these people, it involves a lot of footwork. You have to identify where the bulk of your email is headed and establish relationships with those entities. This means talking to them and understanding their limits.

Or you simply outsource part of the job (e.g., Return Path) or the whole thing (e.g., Send Grid).