"How can we be sure that our radar sensor does not trigger an unintentional break due to a plastic bag blowing in the wind? We collect 10000h of real world driving evidence."
10,000 hours of real world driving data is probably several orders of magnitude too little? Would that really give you confidence that a system can handle a plastic bag blowing in the wind?
I would have thought that this specific scenario would also depend more on factors such as: likelihood of the object being a person or animal vs inanimate, estimated cost of impact to the object and car, and secondary consequences of a rapid deceleration of the car.
Here are some ways to learn deeply and be relevant:
1. Pick a data structure (such as a hash table or LSM-Tree) then read all the literature there is to read, every single paper that's great, following the best conferences year after year, and implement a 10x faster or more scalable version for the std lib of your favorite language.
2. Pick a fault model (such as storage faults, network faults, cryptography faults) then read all the literature there is to read, every single paper that's great, following the best conferences year after year, and write a fault injection or fuzzing harness to break some of the most respected storage/network/cryptography systems (for examples, see the work done by Remzi and Andrea Arpaci-Dusseau on storage faults, Kyle Kingsbury on Jepsen, and Guido Vranken on Cryptofuzz: https://github.com/guidovranken/cryptofuzz).
3. Pick a software field (such as web applications, mobile applications, native applications, file formats such as Office Open XML, or protocols such as SMTP, MIME, HTTP, QUIC) then read as many CVE reports and bug bounty reports as you can find, and then start participating in bug bounty programs within this field. Pick a target and give yourself a goal, e.g. DoS, RCE or read/write access, and do the work to make it happen. Chain as many steps as you can. Automate and enumerate. You'll find a way in if you keep at it. There's nothing like crafting an exploit to change the way you think about programming.
As you gain experience in data structures, storage/networking/cryptography, and security, you'll find this translates well to most software engineering work. You'll gain a speed/safety/security way of thinking, you'll have fun being curious and learning along the way (and hopefully you'll earn a bounty or two and get some CVEs under your name).
3. Read blogs like "The Morning Paper" (https://blog.acolyer.org) but skip fields that are outside your scope. You don't have time to follow more than one or two (or three) major fields.
4. Use Google Scholar to find the most cited papers, or to find papers that build on papers you think are good.
5. Keep an eye out for the conferences where these papers were presented. Then read the other papers that were also presented.
6. When you come across an amazing paper, read other papers by the same authors or supervisors.
7. If you're lucky you might also find good "survey" papers that cover and reference the state of the art.
8. Lecture notes from Stanford or MIT or another university can also be a great way to get a big picture of the evolution of techniques for a given data structure or problem. For example, these lecture notes are just brilliant for getting started with stuff around memory hierarchies: https://www.eidos.ic.i.u-tokyo.ac.jp/~tau/lecture/parallel_d...
These are a few tricks that I find useful. What else?
Yeah similarly “review” “survey” in google scholar will work. Identify major authors (they keep cropping up). Find the big textbooks. See who those kinda people cite, follow that trail.
Start with the foundational paper(s) on the topic. Then use google scholar or your bibliography tool of choice to see who's cited it. It takes a little work but just burn down the list reading abstracts as well as noting how heavily each of these papers has in turn been cited. Those are likely the most influential derived works. It's common for grad students to do survey papers as well, so keep an eye out for those as they often give you a great roadmap to the current state of things.
Also pay attention to authors. If someone has done an influential paper on a topic, it's likely there will be additional work or tech reports in that area on their homepage or with their research group.
To find conferences/venues just note where the more recent papers are published.
I think this comes down to realizing that everything in life is eventually hard. Provided you're doing something worthwhile [1], when the going gets hard, don't look for easier pastures. They're all difficult. Just keep plowing away. And don't doubt. Where there is no doubt, the wind is your companion.
[1] Ecclesiastes is a pretty great book to help figure out for yourself what exactly this is. It's not for nothing that the book uses the same word for "meaningless" or "empty" or "vanity" or "vapor" or "breath" as its refrain. It's also a beautiful book in itself as literature: https://en.wikipedia.org/wiki/Ecclesiastes#Influence_on_West...
(disclosure: I am not religious). Yes. Ecclesiastes has some great and often quoted lines: “For everything there is a season, and a time for every matter under heaven: a time to be born and a time to die…” “…There is nothing new under the sun”. It’s not a typical old testament book, which I think is why I finished it.
(going really off topic here)
I was inspired to look at the Biblical version after reading the awesome “A Rose for Ecclesiastes” by Roger Zelazny, in which a poet on a mission to Mars uses it as a basis for communicating with the ancient Martian elders. Incidentally, a copy of Zelazny’s 1969 story was landed on Mars in 2008 on a DVD carried by the Phoenix probe [0]. If 'The Martian' was the perfect story of engineering on Mars, 'A Rose for Ecclesiastes' is perhaps the perfect poetic look at the planet, at least on a par with "The Silver Locusts".
"just cranking out variations of that formula as papers."
You'd be surprised how often variations on a formula are all it takes to destroy security. Security is enumeration at the end of the day. If it takes a paper mill to "count all the ways" then I'm grateful.
As someone who loves C, Zig has replaced C for me and I don't plan to go back.
There are just too many benefits, not least developer velocity and readability.
Zig is also very approachable to anyone who is familiar with TypeScript, in a way that C isn't. This means more of your team can get excited and get involved, at least in terms of reading the code.
And with Zig, where you have to spend a little extra maintenance time paying for the language being pre 1.0, that's still nothing compared to the upfront and long-lived costs associated with C. You can develop so much faster in Zig compared to C, that you're ahead even pre 1.0.
In the general sense, no, not yet, of course, but in one sense, Zig is probably more production ready than Rust, for example on embedded systems with obscure architectures where OOM-safety is essential.
Here is a talk on using Zig in exactly this kind of environment in production:
> In the general sense, no, not yet, of course, but in one sense, Zig is probably more production ready than Rust, for example on embedded systems with obscure architectures where OOM-safety is essential.
I mean, FWIW, Rust is fairly ready for embedded systems, even when OOM-safety is essential. There are crates dedicated to providing fallible memory allocation primitives for exactly those use-cases.
With that said: I do believe Zig would work a lot better than Rust in severely resource-constrained environments. Rust has a few pitfalls that can easily cause a code size explosion (anything using core::fmt, for instance). And zig's constexpr appear more powerful than Rust's current stable const fn, which can be critical to writing maintainable, small code.
I've worked on a Rust reimplementation of a bootloader that had to fit in 8KiB of RAM, and I often went over accidentally by having a panic handler pull core::fmt. I haven't tried Zig for the same use-case (yet), but from a cursory glance, it appears to be less prone to this kind of code-size explosions.
Raft is essentially Multi-Paxos under the hood, with some overstrict invariants that can impact availability and latency. If you need a consensus algorithm, then it's worth doing a survey of the literature beyond Raft.
Other Multi-Paxos variants are Viewstamped Replication Revisited, which has deterministic leader election for lower latency in the common case and thus no issue of split votes, and which I personally find easier to understand than Raft, also making less demands of the underlying hardware (i.e. VRR can do in-memory leader election, so you're safe in the event of disk failure, corruption, lost writes, and misdirected writes, which Raft does not deal with).
For all these Multi-Paxos variants, you can also look at Flexible Paxos, which is a simple technique for making leader election quorums slightly more expensive (since leader elections are infrequent) in order to make replication quorums much cheaper (since they are the common case critical path). You can use Flexible Paxos to save 20% in the number of hardware nodes for the same common case failure tolerance. Flexible Paxos is a game changer.
I don't think it's a question of "boring" or "proven" vs "latest" or "newer". It's not even an 80/20 split between "use proven technology" and "explore new tools".
What it all comes down to at the end of the day is "quality".
You need to have an eye for quality, you need to be able to recognize good design decisions when you see them, appreciate their worth and weigh their potential. You need to be able to make up your mind for yourself.
If you can spot quality in a language, framework (assuming you even need one) or database, then "boring" or "proven" become meaningless phrases.
Quality transcends age and hype.
When you can spot quality far off, you can paddle out to the swell, be there as it breaks, and enjoy a long quality ride that's not cut short because of irrelevance. You won't need to see thousands of surfers on a wave to know it's a quality break.
With technology, the earlier you can spot quality, the sooner you can start riding the wave and benefitting from everything that comes with that (long term experience with that technology, deep understanding, the opportunity to shape or introduce key design decisions at a critical stage, learning from new ideas that can compound earlier into your own output, new connections and marketing opportunities). That's how it always works. There's no point catching a wave because everyone else is riding it. By then, you probably need to be paddling out for the next big swell.
To be clear, this is more than "skating to where you think the puck will be". This is looking for quality and having an eye for it, and being sure of it when you find it.
I would say quality is undervalued when it comes to technology. Technology suffers from a plague of bloat, and it's getting worse as high level languages pile on the dependencies and abstractions, all the while justifying the waste with "quality is overrated".
I find it hard to believe that you can ship a high quality product without caring about the quality of the components going into it, or that this won't cost you down the line. It only takes a few "good enoughs" in terms of probability theory to end up with a multitude of poor product tail results.
It's also not enough to get the job done, you need to maintain what you ship. You don't need to choose the absolute best tool, but choosing quality tools pays dividends over time.
10,000 hours of real world driving data is probably several orders of magnitude too little? Would that really give you confidence that a system can handle a plastic bag blowing in the wind?
I would have thought that this specific scenario would also depend more on factors such as: likelihood of the object being a person or animal vs inanimate, estimated cost of impact to the object and car, and secondary consequences of a rapid deceleration of the car.