Reminds me of: People said you can’t write a financial trading system in a garbage collected language. Turns out you can as long as you don’t garbage collect during the trading day by carefully managing allocations and manually running GC each day before trading starts.
Or the old story of the junior engineer who finds a memory leak in a missile’s guidance system and the senior engineer says the memory leak is fine as long as you don’t run out of memory before the missile completes its mission.
This is why we need a Software Engineering license.
When the point of the product is to kill someone, you can't just stochastically measure shit like this. If the device is ever out of control before it ceases operations, you're looking at Geneva Convention level offenses.
Even software designed to save lives can't get away with this sort of thing.
Well no it doesn’t but it does however assure the public that all certified teachers have a duty to care for your children in a safe environment. In which teachers are also duty bound to report anything illegal or potentially harmful as they are liable in court if something does end up happening
Doctors have licences and that doesn't necessarily make them any good at doctoring either, but that doesn't mean you should prefer one without a license.
>you're looking at Geneva Convention level offenses.
That's a joke. Russia is intentionally bombing Ukrainian civilians, you think they're going to have a single War Crimes charge put against them? How many civilians did the US unintentionally kill across Afghanistan and Iraq, hundreds of thousands? See any charges there either?
Just because charges don't get brought doesn't mean that you didn't commit a crime.
I don't know how I could possibly get one but I'd love to read an independent comparison of this sort of behaviour between US operations and Russian ones. My belief is that the US aims not to kill civilians but is often careless and nets a lot of collateral damage, whereas Russia doesn't care at all and will happily bomb schools if it thinks there's a target in there. But of course I mostly read Western media and writing so my view is potentially very biased.
You've just asserted that it's stochastic with no basis. It could easily be (and in fact is much more likely to be) a periodic operation that leaks memory.
This sort of analysis is completely normal in software. E.g. in safety critical software you often analyse maximum possible stack depth to check that your stack is big enough (one of the reasons why recursive code is sometimes disallowed). This is exactly the same class of analysis.
Isn't the reason for Software to flourish in last 20 years is that we have had no regulation? Adding regulation would then lead to contraction of software industry as we know it. May be the regulation should be not on employees, but the products in categories where they are subjected to life/death situations.
If you think getting a license is the way to fix quality issues, I have a bridge to sell you.
Presumably undefined unless specifically designed to fail safe but a logical thought would be it keeps burning at present trajectory until it falls and blows up at end of trajectory if device uses an impact fuse. I would be very interested in a more precise answer if you can find one.
> People said you can’t write a financial trading system in a garbage collected language
People have been writing HFT systems in GC languages (Java, OCaml) for a while, and as you mentioned, you need to know when to pause/unpause/prevent collection in the critical section.
This [1] is an excerpt from a talk with Dave Lauer [1], who has done it in Java - it took his system ~40 μs from receiving data to producing market order, and he explains in great detail what amount of work the system had to do. And this was >10 years ago.
Well, for ballistic missiles a large portion of the flight will be the mid-course phase (which is typically unpowered) and the terminal phase may use control surfaces for maneuvering. For cruise missiles that's a good rule of thumb.
Yeah, a ballistic missile's bus still does a lot of horizontal maneuvering during the mid-course phase to actually get on target; the main boost phase is mainly just for getting it pointed in the general direction and proving verticality
Reminds me of a sales call to a military contractor. They were interested in an Open Source OS because of the licensing costs. The embedded OS they were using had a license that said you could run as many instances as you liked on a "single device".
Their problem was that when they operated their device it was indeed a single device, but later became many more. If they paid for the extra licenses they didn't need it would be ripping off the government; if they failed to pay for the extra license at some point they would be in breach of their contract with the government as all these unlicensed devices were released.
It rapidly became obvious that their application was a MIRV and that they weren't really going to change but had asked us to talk with them as part of their process of negotiating a deal with their current supplier.
It's hard to imagine really worrying about your license agreements if there actually were a global thermonuclear war.
That reminds me of a short sci-fi story I read about an alien invasion, in which humans were losing horribly until we learned about their legal system and sued them into submission
> Turns out you can as long as you don’t garbage collect during the trading day by carefully managing allocations and manually running GC each day before trading starts.
The surprising thing here is that it is considered worth it with those limitations.
My hope/guess is that because of the GC hype there wasn't any good alternatives.
With the alternatives gaining maturity now I'd hope that that decision wouldn't be made today (when starting from a clean slate).
> The surprising thing here is that it is considered worth it with those limitations.
It shouldn't be that surprising. Languages like Java and C# are perfectly sensible languages for writing the vast majority of a lot of trading systems. So it's eminently reasonable to use them, or equivalent, and deal with the bits that are GC-phobic as special cases.
I've worked with/on trading systems written is mainstream languages including C++, Java, C# and Python, as well as less mainstream such as APL and Smalltalk. I can safely say that there are way more critical concerns than whether or not it uses a GC.
How? GC does its work in large batches once in a while, whereas for example Arc (the only serious alternative to what GC brings, which is the ability to have arbitrary graphs of objects) does its work much more frequently, with additional effort expended on atomic operations in a multithreaded environment, and with additional memory writes to boot. Somehow I doubt that GC "just increases your energy bill".
[Citation needed] Rust encourages reference counting, a particularly inefficient form of GC which actually turns reads into writes(!), and it cannot move objects to compact memory except when it can prove that every object has only a single reference.
I usually hear those sorts of rationalizations in companies that have already become feature factories. We just have to go fast. We can't make people slow down and learn discipline, or if we have, we can't go back and fix the past because we have to go go go.
In retrospect I think I should have put more effort into breaking the feature factory culture than the discipline culture. It's hard to get clean if your drug of choice is ready at hand.
> it is considered worth it with those limitations.
These people make decisions based on Excel spreadsheets that are not properly documented. I wouldn't trust their judgement, despite the fact they managed not to crash the global financial system too frequently.
For many processes where the lifecycle is limited, freeing memory can be waste of time; most notably freeing memory just before exiting is complete waste.
Is that really an old story? There's usually no memory allocation after initial startup in safety- and mission-critical systems. Rolling over past the max value of an integer data type is a thing though, and they do reboot for that.
FTA: Editor's note: An earlier version of this article mentioned the Boeing 787 CCS uses a Wind River VxWorks real-time OS product at its heart. While this is true, Wind River has been in touch to remind us "the CCS is made up of 80 to 100 applications," as well as VxWorks, and said the bug described in this article is not the fault of its operating system.
"The functions of VxWorks have nothing to do with the data issue you are highlighting in the 787," a spokesperson added. We are happy to clarify our coverage.
VxWorks is the only software in the entire system that I would not question. Sadly, somebody has already added this thing about the airplane to VxWorks' wikipedia page.
I didn't mean to imply that the 787 ran critical infrastructure on Windows 95, just to give an example of a system which was affected by a timer overflow of about that duration.
I immediately though of the GetTickCount win32 function, which, as you said, wraps around to zero every fifty days or so. A lot of programs would crash if you left them open for this long.
I worked at Microsoft on the NetShow Encoder, later known as the Windows Media Encoder, in the Windows 95/Windows NT4 era. I remember carefully writing millisecond code that would deal with differences, instead of absolutes, so that we could run longer than 49 days. Honestly I don't recall whether the limitation was GetTickCount, the recording API, the file format, or all three.
In our debug build, I added a registry key that would start the concept of time at 49-ish days, so that we could test that everything was working without waiting 50 days. However, before we shipped, the intrepid test team did actually run the encoder for 50 days to prove that it worked. (On Windows NT, of course. Windows 95 with good hardware and drivers was perhaps more stable than its reputation, but still...) Though TBH we definitely wouldn't have delayed a release by 49 days if it had failed at the last minute.
I did an ultra running race that took 37 hours last month. I got maybe 10minutes sleep. I had hallucinated before during (shorter) long races but I had no idea that auditory hallucinations [1] were a thing. it was trippy "hearing" things for a solid 12 hours - mainly off-trail "voices" and the sound of "runners" approaching behind me. No matter how many times I stepped to the side to allow a non-existant runner go by, I kept doing it. It was that realistic
I've never experienced auditory hallucinations before. I did however do a 42 hour self navigated run last year with 0 sleep. After about 30 hours I was visually hallucinating pretty much consistently. Amongst the things I saw was Darth Vader, a giant lego man, our cat sleeping on the trail, spectators that weren't there, a banker sitting at a desk staring at me, a church building that wasn't there, signs that didn't exist, and more.
I think it has to do with the way the human visual system effectively does edge detection and then groups those into objects and attaches meaning. When you are extremely sleep deprived, the edges gets grouped differently and you start seeing things that aren't there.
I could judge what was real or not based on likelihood of me seeing said things in the wilderness, or if I got close enough I would be able to tell what I was really looking at. Another runner later informed me that I had told him he wasn't real when he passed me.
I once saw death (exactly as most of us imagine it, dark robes and a scythe) while driving on a highway late at night, I understood the message my brain was sending me: I just took the next exit and slept for an hour (it was too cold to sleep longer, but it made a big difference on my concentration anyway).
I can't believe physical exhaustion is not at least as important a part in this. Personally, I skip a night's sleep every now and then to readjust my sleeping patterns, and I've never experienced anything strange after something like 32-36 hours without sleep (but note: no significant physical exertion involved).
It doesn't even feel that bad once you get past the slump in the 20-24 h region.
It might be. I ran 30h race earlier this year and had no hallucinations. I've not done _exhaustive_ research on this, I'm just recalling some undergrad neuropsychology. It could also be the environment. I suspect it's a lot less likely that you'll be seeing stuff that isn't there when you are in a familiar environment as opposed to running alone in the bush.
How do you rationalize running for 37 hours while having hallucinations, I am legitimately curious. At what point do you say "I might be pushing myself too much"?
Well, I don't think it was a rational thing to do, but that's not a reason not to do it. The race [1] took place on circular hiking route 118 miles with 19,000ft of climbing. I'd hiked sections of the route (which typically takes 7-10 days for hikers) but was really drawn to the idea of doing the route in a "single day" (actually starts at 6am Friday, cut-off is 10pm Saturday - 40 hours). It was also my fourth time attempting to finish the race. After each of the first three times I failed, my immediate reaction was "It is too hard. I'm not going to sign up again." That is usually followed within 24-48 hours with an intense desire to sign up and try again. The hallucinations were not something I sought, and as I said, the auditory was not something I even know existed. The overall feeling was trippy rather than discomfort. Unlike the many blisters on my feet. I'm definitely older and chunkier than most doing the race I'm proud as hell that I persevered and finished especially after failing repeatedly. During the years of training and attempts I learned a lot about myself - good and bad.
I’m not a runner but pushing yourself to your limits is the point of many sports. Humans like seeing how close they can get to death without succeeding.
Your body goes into an autopilot trance. It's happened to me on long bike marathons where as long as your water, electrolytes and fuels are still coming in your brain is in this primal mode where your body knows what to do, and your brain shuts off. You will hear conversations in memories vividly, or the 'singing' of the frame of the bicycle and its connection with the pavement through your tires becomes readily apparent.
Later Hominids are meant for this. Without being able to pass long time chasing our prey to the moment of its exhaustion, we'd not have been able to secure the cholesterol needed to further evolve our brains.
"exhaustion chase" never lasted that long, I don't remember the times exactly but animals that don't sweat are exhausted pretty quickly, a couple hours at most.
Doesn't make any sense to me either. Not a biologist, but AFAIK animals known for being sprinters are able to outperform you in bursts, with refractory periods of extreme exhaustion and vulnerability. Especially if already wounded. You turn the chase into a game of hide-and-seek.
Small fish/shrimp can appear to "teleport" since they dart away so quickly, but if you harass them enough, you exhaust them into a state of shock. It's no longer a chase, it's a harvest.
Even in manhunting, you get nowhere chasing the subject since you move at the same speed and the element of uncertainty tilts in their favor. So you try to anticipate where they're going and ambush them there. You turn the chase into a trap.
We're apex predators for a reason. Exhaustion chases don't make sense.
Your analogy made me laugh, but also made me think about how common fatal mental illness is among humans, and contrast that rate with the fatal plane crashes attributable to computing errors.
It's a pretty amazing accomplishment for humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them.
> humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them
I would not compare human brains to tech. Maybe you can call them simpler and thus more reliable. The human brain is processing far more data than any processor humanity has made. The only comparison is humans handle various types of data whereas computers are designed to handle just a small batch.
> It's a pretty amazing accomplishment for humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them.
Load the airplane's "brain" with algorithms for how to survive, cooperate, adapt and replicate under every conceivable condition and you'll end up with a comparable system more complicated and worse-performing than ourselves. Not really a fair comparison.
Self-driving cars (and robotics) would be better examples, since they attempt to emulate a broader swath of our sensory capabilities.
> It's a pretty amazing accomplishment for humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them.
Airplanes are not reliable. That's why they require frequent and regular maintenance.
Ignoring the poor analogy linking brains/humans to human built systems, if every human maintained their mental and physical health as much as airplanes get, then they'd all be much better off.
You need more scare quotes around “reliable” than brain. The airplane requires constant almost daily maintenance by dozens of highly trained professionals and yet none of its chips will survive decades save for a few lucky passive components.
Whereas going decades without seeing a qualified doctor has been the default state since the dawn of homo sapiens with nothing more than a good night’s sleep.
Chapter 1 of "The Inmates are Running the Asylum," by Alan Cooper (1999) opens with the riddle: "What do you get when you cross a computer with an airplane?"
This was my first thought. Surely there are other systems on the plane that require maintenance or checks that are even more disruptive. Rebooting is a switch, button, or command. Inspecting engines, pipes, or compartments requires somebody to get out of a chair.
This seems like some kind of overflow problem, but 2^32 milliseconds is 49.7 days. Is it possible that the 51 day number came from a combination of rounding and an off-by-one error in the reporting?
I'm just thinking about a Liam Neeson movie in a 787 with on flight tank refill and the reboot after 2^32 milliseconds because the plane could crash and it is the president pilot in the middle of something bad like a WW3..
Can you imagine being a pilot halfway through a flight and while you are at 40,000 feet your airlines tech support asks you "Have you tried turning it off and on again?" :-D
One time I was flying inside China, and as the plane reached cruising altitude it felt eerily quiet, as if the engines were off and we were just gliding. I honestly wondered "Is this it? Am I going to die? Surrounded by people I won't even be able to communicate with?".
Of course since I'm still here, it wasn't it. After a while I felt the vibration of the engines again and was relieved. Maybe the pilots had to reboot the plane.
I have lab equipment that needs a power cycle after 48 hours of power on time for it to continue to produce high quality data. This is because the internal calibration code only runs upon boot. Maybe these planes have something similar.
So, to me, the power cycle thing is not super surprising. That fact that the FAA felt the need to order Boeing to do it seems a bit suspect though. That's probably the real story.
The only reason this seems to be a problem is that apparently operators (or the manufacturer) didn't know about it, since I don't think there's an actual usecase for the aircraft to be powered on continuously for that long.
FTA: It is usual for commercial airliners to spend weeks or more continuously powered on as crews change at airports, or ground power is plugged in overnight while cleaners and maintainers do their thing.
Ryanair a private ULCC of Europe and one of the largest ULCC of the world having a TAT of 25 minutes does not have a seatback pocket.
Wondering why?
Back in 2004, removal of seat back pockets greatly reduced the time for which the Ryanair aircraft stayed on the ground. That’s is because it takes more time to clean an airplane with seatback pockets.
So, that’s 25 minutes from touching down to being airborne again. Chances are the plane is standing still for less than 15 minutes. Long-haul flights take longer (up to two hours, according to https://simpleflying.com/aircraft-turnaround-process/)
Also: it wouldn’t surprise me if you could not refuel an airliner while it is rebooting, all lights may be off for a while, etc.
I learned programming in the late '70s on a Wang 2200T. The manual had a procedure it would recommend called "Master Initialization." Yes: turning the power off and on.
I’m pretty sure most of the computers can be reset during flight. You may lose flight protections, but you can fly an airplane on basic instruments. You can even handle unreliable speed by flying at a specific angle of attack and a proper thrust setting. (According to the Air France 447 report)
On some (all?) Airbus planes, the basic flight controls don’t require the flight computers to function.
>On some (all?) Airbus planes, the basic flight controls don’t require the flight computers to function.
I would honestly hope that would be true about any airplane from any manufacture. Fly-by-wire makes some of that more difficult when the wires go through the computer, but even then I'd have assumed some sort of 100% manual capability. Like when Y2K nutters were predicting airplanes falling out of the sky because of 2-digit years rather than the logical the planes would behave according to laws of physics for an intact air frame.
Airplane always want to fly, give them enough speed, they will take off on their own.
Thats why you have bumpy landings sometimes because pilots didn’t slow down/flare properly to slowdown the aircraft instead forced it to ground. With enough wind some general aviation aircraft are capable of takeoff/land vertically like a chopper.
If you take off on day 50 23 hours type of thing is more of what I was getting at. It's very easy to just hit skip when presented dialogs like "3 more days to reset password" or "3 more days until scheduled reboot". As long as it doesn't end during your flight, why not push it for someone else? Otherwise, you just make it part of every flight's routine to reboot.
I think this is one of the areas that airlines are super good at. Every single plane needs strict regular maintenance and tons of checklists in order to take off, so something like a 50-day requirement for a reboot would be in that list of items, and if the people aren't following those lists of items you're going to have things like turbine failures, so they get done.
Maybe if there's a really badly run airline it might be a problem, but there are much more severe problems than software in a case like that.
Boeing gets away with this software issue because it's not really a big issue. The scary point the media can tell you because you don't have context to understand why it's mundane.
>Boeing gets away with this software issue because it's not really a big issue. The scary point the media can tell you because you don't have context to understand why it's mundane.
Yes, like the mundane software overriding pilot's intent from software written specifically to not retrain pilots on the system? Yup, sounds mundane to me.
When I said I don't trust anything from Boeing, it had little to do with this 50 day reboot, but much more in line with the Boeing is allowed to self certify and has been shown to not do that in a good way. They have ruined that trust because of greed. You might think that's a broad brush? At this point, every little bit is just another instance of death from a thousand cuts.
Usually these aircraft are power cycled often. There is no way they are flying these planes 51 days straight with no power cycle. Maintenance checks would necessitate a restart.
I would not be so brazen as to call the engineers behind this "average" or their work so. Unless you can make the argument that a plane would fly for 51 days straight, being required to reboot the on-board systems is not a major deal-breaker, especially as the larger Boeing's take 60 minutes to refuel.
The engineers behind Boeing planes have built an incredibly sophisticated machine which omitting a very nessescary but not so inconvient task of rebooting every n days, performs its primary directive well.
one bug implies two bugs only in the sense that any given software has at least one bug and any given software has at least one more bug than you know exists ;)
Or the old story of the junior engineer who finds a memory leak in a missile’s guidance system and the senior engineer says the memory leak is fine as long as you don’t run out of memory before the missile completes its mission.