What is necessarily provocative about the term? Both the noun and verb forms of the word "desert" fit well here, especially in the context of the history that this article talks about.
"If you don't have a car ... you are going to have a significantly limited life in so many ways that having to drive to the supermarket just seems like a triviality in comparison." Except that everyone needs to eat, and we know that a poor diet can lead to many, many bad health outcomes. You're correct that there are other disadvantages, such as access to employment opportunities. But if you didn't live in a food desert, you could get a job at the grocery store, right?
"people just like, have cars, it's normal"
Except that owning (or leasing) a car is a significant, ongoing expense. I barely use mine (<1,000 miles per year) and it still costs me $2k in insurance, registration, and maintenance. Some people just can't afford that. And what if you are young, old, or are unable to drive for any reason?
It's provocative because the implication is that it's a "problem" that needs to be "solved" when in reality there's a free market, if you think that an area needs a grocery store then start one up, you'll probably find that there is just isn't the demand.
Everyone needs to have a life, to have a life in the United States outside of a few very specific areas requires a member of your family to have a car.
> I barely use mine (<1,000 miles per year)
Right, so you obviously don't live in a food desert, presumably you're in NYC or San Francisco or the DC area or some other place where you can just use the metro for everything, this is not the case in the vast majority of the US, it's not even the case in most of Britain or Europe.
My grandma has to walk 15 minutes to get to her nearest food shop, one day she won't be able to make that walk, does she live in a food desert? No, she's just old, we'll help her out!
Thank you! I think this is the first time I've logged in for 10 years.
I don't mind the current UI change, but I wish significant changes would come with a toggle button to let me look at the old and new renderings. I don't think it's possible to mentally "place" the improvements without side-by-side comparisons and affordances for finding edge cases where the new UI may be lacking.
I'm going to guess somewhere between 50 and 400 people. Smaller than 50: this is probably an org where "we maintain the source code and documentation. Running, debugging, and scaling your cluster is YOUR problem". Larger than 400: the org probably fulfills other functions besides "running thousands of production Cassandra clusters."
This is a lot, and is one of the big things that surprised me when I first joined a large organization. But here's an example breakdown in no particular order:
- 10 to 30 people for managing, contributing, and maintaining the open source aspect. Just a handful of engineers could contribute features and fixes to the project, but once the open source project and community gets larger it becomes a full time job. System component maintainers, foundation boards, committees, conferences, etc. add up.
- 10 to 70 people for "operations". As you mentioned, the load here tends to scale with the number of clusters (customers). At the large end this is several teams, with say a team dedicated to (a) fleet-management of the individual machines in a cluster, (b) cluster lifecycle management, and (c) macro level operations above the cluster level. Alerts can't all go to this team, so some of the work is writing alerts that go to customers in a self-serve model.
- 10 - 40 people for "scale projects." At this scale you have 1 - 10 customers that are on the verge of toppling the system over. They've grown and hit various system bottlenecks that need to be addressed. And you'd be lucky if they're all hitting the same bottleneck. With this many customers, it's likely that they've all adopted orthogonal anti-patterns that you need to find a fix for: too many rows, too many columns, bad query optimization, too many schema changes, too many cluster creates and destroys, etc. So you probably have multiple projects ongoing for these.
- 10 - 30 people for "testing infrastructure". Everyone writes unit tests, but once you get to integration and scale testing, you need a team that writes and maintains fixtures to spin up a small cluster for some tests and a large cluster for scale tests (which your "scale projects" teams need, btw). And your customers probably need ways of getting access to small test Cassandra clusters (or mocks of the same) for THEIR integration and scale tests, since Cassandra is just a small part of their system.
- 10 - 30 people for automating resource scaling and handling cost attribution. These may not be one function, but I'm lumping them together. "Operations" might handle some of the resource scale problems, but at some point it's probably worth a team to continually look for ways to manage the multi-dimensional scaling problem that large cluster software systems inevitably create. (Is it better to have few large nodes, or many small nodes?) You need some way of attributing cost back to customer organizations, otherwise you're paying $50M because one engineer on the weather team forgot to tear down a test cluster in one automated test 6 months ago and... You need to make sure that growth projections for customers are defined and tracked so you have enough machines on hand.
- I'll add that it'll be worth adding whole teams for some of the more complex internal bits of this system, even if the actual rate of change in that sub-system is not very high. At this scale organizations need to optimize for stability, not efficiency. You don't want to be in the situation where the only person who understands the FizzBuzz system leaves and now dozens of people/projects are blocked because nobody understands how to safely integrate changes into FizzBuzz.
- Things not covered: security, auditing, internal documentation, machine provisioning, datacenter operations, operating system maintenance, firmware maintenance, new hardware qualification, etc. Maybe there's an entire organization dedicated to each of these, in which case you get it for free. If not, some of your time needs to be spent on these. (Even "free" might have a cost as you need to integrate with those services and update when those services change.)
Spot on, and thank you. My second team was ~40 (might have peaked at ~50) split across four sub-teams, for software that ran at similar scale and was designed and developed to rely heavily on other in-house infra. Maybe half a dozen people on adjacent teams (including customers) who had more than trivial knowledge of our system. Some in our team were almost pure developers, some were almost pure operators, most were at various points in between.
I think the reason you and I (we know each other on Twitter BTW) are so at odds with some of the other commenters is that they haven't maxed out on automation yet and don't realize that's A Thing. Automation is absolutely fantastic and essential for running anything at this scale, but it's no panacea. While it usually helps you get more work done faster, sometimes it causes damage faster. Some of our most memorable incidents involved automation run amok, like suddenly taking down 1000 machines in a cluster for trivial reasons or even false alarms while we were already fighting potential-data-loss or load-storm problems. That, in turn, was largely the result of the teams responsible for that infra only thinking about ephemeral web workers and caches, hardly even trying to understand the concerns of permanent data storage. But I digress.
The point, still, is that when you've maxed out on what automation can do for you, your remaining workload tends to scale by cluster. And having thousands instead of dozens of clusters sounds like a nightmare. There are many ways to scale such systems. Increasing the size of individual clusters sure as hell ain't easy - I joined that team because it seemed like a hard and fun challenge - but ultimately it pays off by avoiding the operational challenge of Too Many Clusters.
- There's nothing scarier than "the automation had no rate-limiting or health-checking". Of course, what do we mean by automation? At some point it becomes impractical to slow every change down to a crawl, so some judgement is required. But "slow enough to mount a human response to stop it" is the standard I've applied to critical systems.
- Thankfully I've avoided having to support "real" storage systems. The challenges of "simple" distributed databases storing is enough for me. :-)
On the "pets vs. cattle" metaphor, I think most people fail to grok the second component of that. I don't think there are many successful cattle ranchers that will just "take the cattle out of line and shoot it in the head." The point of the metaphor is: When you have thousands of head of cattle, you need to think about the problems of feeding and caring for cattle operationally, not as a one-off.
Despite what https://xkcd.com/1737 might make one believe, people don't just throw out servers when one part goes bad, or (intentionally) burn down datacenters. What the "hyperscalers" do is decouple the operational challenges of running machines from the operational challenges of running services (or at least try to). Of course this results in a TON of work on both ends.
Combining a traffic circle with stop signs is a recipe for confusion.
One thing that Seattle has going for it here is it's very narrow streets, probably 1/2 the size of that intersection in Sacramento: https://goo.gl/maps/A1epH93AE8R28rZH7
That said, we also have 4-way intersections with no stop signs and no roundabouts. This is complete madness.
> China currently has two Panamax ports, with a third on the way. Vietnam has none and is building its first. The U.S., by comparison, has 42 by my count. [5]
I'm curious how Panamax or New Panamax berths make ports more economically viable? And whether US ports are more heavily import focused (and thus want a Neopanamax berth available should the need arise) vs. Chinese or Vietnamese ports? The reason I ask is because I spend too much time looking at ports on Google Maps, and to my eye no single N.A. port compares to most Chinese ports in terms of land area (and apparent capacity). [1] Do goods just sit idle in yards at Chinese ports?
Also, it looks like a single user has added ~30 ports to that list for China. [2] I'm not sure what to think of that.
[1] With the exception of Long Beach. Also this is only for container capacity, so I'm not thinking about roll-on/off, bulk and tanker ports.
A Chinamax ship is larger than a Panamax port. Surely if the standard is named after your country you have many ports large enough to handle them. I imagine the list was(is?) inaccurate.
I know the dry dock list on Wikipedia isn't accurate because it doesn't even include my shipyard!
Her Hainish novels (SF) may be more to your liking, but I think all of her works are informed by the kind of mysticism that informed her translation of the Tao Te Ching. What made A Wizard of Earthsea dry for you?
Much of it was written in the way of "The wizard went here and this this, and it was pretty damn cool". It wasn't necessarily dry, but it was so differently written from the fantasy I normally read I couldn't get used to it on my first time reading it. It should probably try again at some point, however. The setting was really cool.
It definitely is distinct. Personally I find it refreshing after all the 500-1200pg fantasy I read. There's something poetic about compressing so much into so little words. I find that it flows nicely. It's more poetic than dramatic.
I read The Left Hand of Darkness and The Dispossessed at the beginning of the year, just before she passed away. This summer I the first four books of the Earthsea series and will probably read The Other Wind and Tales soon. I've really enjoyed both universes--with Earthsea I keep wishing someone would make a game with this kind of magic system. The Lathe of Heaven is such a wonderful title, I'm looking forward to reading that as well.
I read her translation of Tao Te Ching while waiting in 2 hours of traffic for a ferry and it was wonderful and really informs the rest of her works.
I'm currently reading Cadillac Desert by Marc Reisner, on the history of U.S. water policy in the West. Frustratingly dry.
Also re-reading On the Road but unlike 16 years ago I'm reading the 'Original Scroll Edition' this time: no chapters, paragraph breaks, invented character names, censors, etc. The book you read in high school is, in fact, the novelization of the original, which Kerouac wrote in twenty days. I've kept my beat-up paperback copy of the 'novel' nearby and I've preferred the scroll edition for the most part.
On the nightstand:
- Annals of the Former World by John McPhee 300 pages into it, but I got a bit tired of the redundancy inherent in combining multiple books into one.
- The Vulgar Tongue by Jonathon Green. 40 pages in. I've been looking for a copy of the dictionary.
- The Path to the Nest of Spiders by Italo Calvino.
Ten years ago I thought we were all going to solve this with OpenID and strong protections on the provider side. [1] What happened? I know a bunch of providers moved on to OAuth which let users authorized 3rd parties to do stuff on their behalf. And then at some point we decided v1 was hopelessly broken and moved to v2. But today it's still the rare event where I can login with my {Gmail, Hotmail, Yahoo!} account.
[1] U2F, authorized devices, predictive phishing protections, etc.
"If you don't have a car ... you are going to have a significantly limited life in so many ways that having to drive to the supermarket just seems like a triviality in comparison." Except that everyone needs to eat, and we know that a poor diet can lead to many, many bad health outcomes. You're correct that there are other disadvantages, such as access to employment opportunities. But if you didn't live in a food desert, you could get a job at the grocery store, right?
"people just like, have cars, it's normal" Except that owning (or leasing) a car is a significant, ongoing expense. I barely use mine (<1,000 miles per year) and it still costs me $2k in insurance, registration, and maintenance. Some people just can't afford that. And what if you are young, old, or are unable to drive for any reason?