I/O is about channels. Channels are fixed bandwidth queues which generally have a 'signaling' rate, and a 'bandwidth' the one that most people are familiar with is a SATA port.

A SATA port can do some number of IOs per second (this depends on how the controller is designed for the most part) and the SATA data pipe is 6 gigabits/second or 600 megabytes per second these days.

A SATA Disk (the spinning rust kind) is different, it can pump out some maximum amount of data per second (usually about 100 Mbytes/sec and has an average seek time of a few milleseconds. So if you are doing random i/os to the disk, so that every I/O needs a seek, and the seek takes 10 mS, you can do at most 100 I/O operations per second (IOPS) if you transfer 512 bytes with each IOP you get a throughput of 100* .5K or 50K bytes per second. If you transfer 100Kbytes per transaction you get 10 megabytes per second. Anyway the bottom line is that an I/O channel is bound both by its bandwidth and the number of IOPS it can do.

A PCIe 'lane' can do 250Mbytes per second and with Intel's 5500 chipset about 2 million IOPs. (Although those IOPs are spread across all PCIe channels) A "big" server chip set will give you 32 or maybe 48 PCIe "lanes" which can be 12 by 4 (or x4) lanes or 2x16 + 8x1, or 2x16, 1x4 and 4x1 (this is a common desktop config for two video cards and misc peripherals)

The bottle neck is exactly like networking, each I/O device can consume a certain amount of channel bandwidth, since they don't run 24/7 you can "oversubscribe" which is to say potentially have more I/O devices than you have bandwidth to service them (this happens a lot in like big disk arrays) and ultimately the transaction rate of the entire system is constrained by the cross sectional bandwidth of memory and I/O channels such that you get I/O limited in terms of things you can push around.

Anyway, a mainframe has thousands of channels. Lots and lots of them. And each of those channels has a lot of raw bandwidth and can sustain a high transaction rate. This makes a mainframe a transaction monster.

In the taxonomy of computer systems, 'supercomputers' have a monster amount of memory bandwidth and high transaction rates (low latency) between their compute nodes (or cores). Mainframes have a monster amount of I/O bandwidth with high transaction rates between their nodes and I/O devices. General purpose computers achieve good economics by limiting memory and I/O bandwidth to stuff that will fit in a single piece of silicon. (or at most a couple of pieces)

Thanks for the well-thought-out and detailed explanation.

Mainframes tend to have prodigious amounts of expensive, unusually fast, specialised RAM. And then they have extremely smart controller hardware that the OSes can hand off almost everything off to.

And this is a theme replicated throughout mainframe design. Everything is replicated, all pathways are wide as possible, everything is integrated from the hardware up to the application layer.

Take virtualisation, for instance. IBM mainframes have PR/SM, a hardware-level capability to virtualise everything in the system down to the lowest level of granularity. And they've done this since 1985.

Plenty of other machines have Amazing IO in this day and age now. It is a fairly old notion now that Mainframes were much better at IO than other machines (this /was/ true).

That said, they do deal with IO pretty well. They have dedicated offload processors for transferring data, which means for the same (IO intensive) workload, the CP utilisation of a zSeries machine would be much less than that of an x86, pSeries (or other) machine. And in terms of the disk being a bottle neck... If you're pushing to a disk array that has 192GB (pretty standard) of write-back cache... then no, the disks are not a large concern.

But... At the end of the day, does this really weigh up to the cost of Big Iron, or for that matter the additional licensing of software on top of it?... Not really.. is my answer.

Back in the day mainframes had an I/O advantage because they offloaded a lot of I/O processing to coprocessors (e.g. a minicomputer would just have one processor while a mainframe would have one main processor and one or more I/O coprocessors). That advantage appears to be gone now because I think they are using normal PCIe cards and normal cores as I/O coprocessors (which you could also do on a non-mainframe server).

Of course, mainframes are also faster simply because they are more expensive (e.g. 100 PCIe slots are obviously faster than 10 slots), but that's not an intrinsic advantage (in theory you could develop a $10M x86 server — call SGI if you're interested).

Think throughput...

Lots of cache memory and very fast buses to very large external disk arrays. An old mainframe trick was to use the previous generation hardware as the IO controller for the current generation...