Perhaps because I am inexperienced, I don't understand how software is much more complicated than other stuff, like electronics engineering or industrial engineering. Those are expensive to debug and have complicated interactions, yet engineers somehow can find ways to make them robust without the cost skyrocketing.
Perhaps because software is now the interface between different systems, and we are desperately trying to abstract away the underlying system yet the details are eventually somehow leaked and cause other issues? Perhaps because complexity is similar to multiplication, physical systems are limited by 3D space and softwate systems can become entangled without bound? Just some naive thoughts.
I think it's actually because in a modern environment, software is cheap to deploy, debug and update. That leads to under-investment in design and testing. For many software environments, I think the root cause of unreliability is that it's perceived to be cheaper to build and observe failures in production than it is to invest in adequate design review and QA.
It's instructive to consider how things went in parts of the software industry where failures are more expensive, for example:
- Avionics firmware gets a lot more scrutiny than a trendy website.
- In the days when consoles shipped games on cartridges, a lot of time went into QA. Today, a day-one patch is considered table stakes -- if the game even has a physical release.
Here's a shot at an answer, lifted from 'The Mythical Man Month', Fred Brooks:
"Software entities are more complex for their size than perhaps any other human construct, because no two parts are alike (at least above the statement level). If they are, we make the two similar parts into one, a subroutine, open or closed. In this respect, software systems differ profoundly from computers, buildings, or automobiles, where repeated elements abound."
Part of the issue is that there is "no cost" to changing something in the software stack, while there are very tangible costs and barrier to entry to modifying a physical structure. This tends towards a more conservative culture, where controls are placed on designs and changes. This culture also exists in safety-critical software, like that used in medical devices, aviation, and industrial automation.
Otherwise, robustness is relatively expensive because it requires the organization to value the long-term quality and function of the system, at the expense of short-term velocity and malleability. If you are competing with others who can hack together an MVP with 90% functionality overnight, then waiting for the engineered product may be problematic.
Software is more malleable than, say, a bridge. This leads to a lot of input about how to best do things, which generates a lot of rework, which generates a lot of billing hours, and keeps us all employed.
Does it need to be like this? No. But until the VC MBA gods get their hands off the tiller and we face actual energy scarcity, expect more stupidity.
Perhaps because software is now the interface between different systems, and we are desperately trying to abstract away the underlying system yet the details are eventually somehow leaked and cause other issues? Perhaps because complexity is similar to multiplication, physical systems are limited by 3D space and softwate systems can become entangled without bound? Just some naive thoughts.