Unfortunately Immich doesn't (yet) support object storage natively, which IMHO would make things way easier in a lot of ways.
You can still mount an object storage bucket to the filesystem, but it's not supported officially by Immich and you anyways have additional delay caused by the fact that your device reaches out to your server, and your server reaches out to the bucket.
It would be amazing (and I've been working on that) to have an Immich that supports natively S3 and does everything with S3.
This, together with the performance issues of Immich, is what pushed me to create immich-go-backend (https://github.com/denysvitali/immich-go-backend) - a complete rewrite of Immich's backend in Go.
The project is not mature enough yet, but the goal is to reach feature parity + native S3 integration.
This is most likely a strong requisite for such a big scale deployment if DDOS protection and detection - which explains their architectural choices (ClickHouse & co) and the need of a super low latency config changes.
Since attackers might rotate IPs more frequently than once per minute, this effectively means that the whole fleet of servers should be able to quickly react depending on the decisions done centrally.
The article mentions that this Lua-based proxy is the old generation one, which is going to be replaced by the Rust based one (FL2) and that didn't fail on this scenario.
So, if anything, their efforts towards a typed language were justified. They just didn't manage to migrate everything in time before this incident - which is ironically a good thing since this incident was cause mostly by a rushed change in response to an actively exploited vulnerability.
I think (would love to be corrected) that this is the nature of their service. They probably push multiple config changes per minute to mitigate DDOS attacks. For sure the proxies have a local list of IPs that, for a period of time, are blacklisted.
For DDOS protection you can't really rely on multiple-hours rollouts.
Ironically, this time around the issue was in the proxy they're going to phase out (and replace with the Rust one).
I truly believe they're really going to make resilience their #1 priority now, and acknowledging the release process errors that they didn't acknowledge for a while (according to other HN comments) is the first step towards this.
HugOps.
Although bad for reputation, I think these incidents will help them shape (and prioritize!) resilience efforts more than ever.
At the same time, I can't think of a company more transparent than CloudFlare when it comes to these kind of things. I also understand the urgency behind this change: CloudFlare acted (too) fast to mitigate the React vulnerability and this is the result.
Say what you want, but I'd prefer to trust CloudFlare who admits and act upon their fuckups, rather than trying to cover them up or downplaying them like some other major cloud providers.
@eastdakota: ignore the negative comments here, transparency is a very good strategy and this article shows a good plan to avoid further problems
> I truly believe they're really going to make resilience their #1 priority now
I hope that was their #1 priority from the very start given the services they sell...
Anyway, people always tend to overthink about those black-swan events. Yes, 2 happened in a quick succession, but what is the average frequency overall? Insignificant.
This is Cloudflare. They've repeatedly broken DNS for years.
Looking across the errors, it points to some underlying practices: a lack of systems metaphors, modularity, testability, and an reliance on super-generic configuration instead of software with enforced semantics.
I think they have to strike a balance between being extremely fast (reacting to vulnerabilities and DDOS attacks) while still being resilient. I don't think it's an easy situation
I would very much like for him not to ignore the negativity, given that, you know, they are breaking the entire fucking Internet every time something like this happens.
This is the kind of comment I wish he would ignore.
You can be angry - but that doesn't help anyone.
They fucked up, yes, they admitted it and they provided plans on how to address that.
I don't think they do these things on purpose. Of course given their good market penetration they end up disrupting a lot of customers - and they should focus on slow rollouts - but I also believe that in a DDOS protection system (or WAF) you don't want or have the luxury to wait for days until your rule is applied.
I hope he doesn't ignore it, the internet has been forgiving enough toward cloudflares string of failures..its getting pretty old, and creates a ton of choas. I work with life saving devices, being impacted in any way in data monitoring has a huge impact in many ways. "sorry ma'am we can't give your child t1d readings on your follow app because our provider decided to break everything in the pursuit of some react bug." has a great ring to it
Cloudflare and other cloud infra providers are only providing primitives to use, in this case WAF. They have target uptimes and it's never 100%. It's up to the people actually making end user services (like your medical devices) to judge whether that is enough and if not to design your service around it.
(and also, rolling your own version of WAF is probably not the right answer if you need better uptime. It's exceedingly unlikely a medical devices company will beat CF at this game.)
Most hospital and healthcare IT teams are extremely under funded, undertrained, overworked, and the software, configurations and platforms are normally not the most resilient things.
I have a friend at one in the North East right now going through a hell of a security breach for multiple months now and I'm flabbergasted no one is dead yet.
When it comes to tech, I get the impression most organizations are not very "healthy" in the durability of systems.
I have never seen an Ops team being rewarded for avoiding incidents (focusing in tech debt reduction), but instead they get the opposite - blamed when things go wrong.
I think it's human nature (it's hard to realize something is going well until it breaks), but still has a very negative psychological effect. I can barely imagine the stress the team is going through right now.
Depending on the tech debt, the ops team might just be in "survival mode" and not have the time to fix every single issue.
In this particular case, they seem to be doing two things:
- Phasing out the old proxy (Lua based) which is replaced by FL2 (Rust based, the one that caused the previous incident)
- Reacting to an actively exploited vulnerability in React by deploying WAF rules - and they're doing them in a relatively careful way (test rules) to avoid fuckups, which caused this unknown state, which triggered the issue
I honestly think it is.
The amount of people who thinks Europe and EU are the same thing is really concerning.
And no, it's not only americans. I keep hearing this thing from people living in Europe as well (or better, in the EU).
I also very often hear phrases like "Switzerland is not in Europe" to indicate that the country is not part of the European Union.
TL;DR: They forked Chromium to avoid being detected as bots because of the automation
Unfortunately this will probably just trigger a cat and mouse game - but I'm curious to see where that ends.
Currently, it's already painful that some companies such as CloudFlare are relying on privacy invasive features [1] to fight bots and spammers (but I totally understand their reasons).
Maybe Proof of Work is the only solution, but it's environmentally bad and stupid.
You can still mount an object storage bucket to the filesystem, but it's not supported officially by Immich and you anyways have additional delay caused by the fact that your device reaches out to your server, and your server reaches out to the bucket.
It would be amazing (and I've been working on that) to have an Immich that supports natively S3 and does everything with S3.
This, together with the performance issues of Immich, is what pushed me to create immich-go-backend (https://github.com/denysvitali/immich-go-backend) - a complete rewrite of Immich's backend in Go.
The project is not mature enough yet, but the goal is to reach feature parity + native S3 integration.
reply