"I would absolutely love to know what it is about Rails that Twitter found stopped them from working on it together. Please, if you work there, I am dying to know."
I don't work at Twitter, and I'm whitewashing a bit because I'm at work, but the consistent line I've heard from engineers there[1] and from the press[2] is performance. Plus, Scala has shares some linguistic niceties with Ruby that Java doesn't, so it's a little bit more of a natural fit.
From my own experience with Scala, the reason to go in a service-oriented direction is, to say the least, make your compiles go faster over smaller isolated bits of functionality in your app. Scala type safety is awesome. Scala compiles, much less so.
Ruby still runs at Twitter, but my understanding is that the more recent engineers there are afraid to touch what's there for fear of blowing things up.
If I recall, Rails got the bad rap early on because Twitter thought it was the cause. It turned out they were making synchronous calls to a Jabber server, and something in that connection was getting overloaded and causing their apps to hang.
I'm not saying Ruby doesn't have a performance issue over something like a JVM-based language, Go, or Haskell, but the Twitter example works better as an example of the need to debug, benchmark, and test all aspects of your application. Don't just leap to conclusions about problems without information.
> If I recall, Rails got the bad rap early on because Twitter thought it was the cause.
IIRC, it got a bad rap because of one blog post by a Twitter dev that seemed to suggest that Ruby and/or Rails was the cause, which was later followed by a more in depth explanation from Twitter of their reasoning, which didn't place the blame there, but by that time Ruby/Rails haters had permanently latched on to the "Rails was killing Twitter and that had to leave to survive" meme and facts were no longer relevant.
They also had that big huff about ActiveRecord not supporting more than one database system in a replicated environment. Humorously, someone from outside of the organization supplied a patch a few minutes later.
To make matters more complicated, Twitter publicly abandoned Starling, their Ruby-based message queue, for something written in Scala. People took that to mean they abandoned Ruby and Rails completely.
As far as I know they are still using Rails to generate the HTML pages. If not, it is a recent change.
I don't think they leapt to conclusions. According to [1], they were running into problems with garbage collection issues in long-running Ruby processes. They put a lot of work into doing performance measurement, if Zipkin (https://blog.twitter.com/2012/distributed-systems-tracing-zi...) is any measure.
+1 on this, Recently at my company we encountered performance issues in one of the products, (mongodb, mongoengine ORM, django)...We dove in, isolated separate modules and let them run, figured out some metrics, setup some reporting in AWS and the problem quickly presented itself to us. For the performance we needed, we had to go with a different language (c++) on certain modules. Monitoring like this is something we're going to include from now on, makes the actual debugging and integration testing process much easier.
Let's also remember that this was a while ago. Ruby has picked up some critical performance and GC improvements along the way, including completely replacing the old AST-walking interpreter from 1.8 and before with the bytecode interpreter of 1.9 and newer.
> From my own experience with Scala, the reason to go in a service-oriented direction is, to say the least, make your compiles go faster over smaller isolated bits of functionality in your app.
Compile times are not a reason to move to a service oriented architecture.
> Does the JVM provide better raw performance than Ruby? Yeah, okay, you got me, it does. So let’s also mention that one of the languages that runs on the JVM these days is Ruby, and it gets a performance boost when run there.
We can mention that, but let's be careful to be clear that JRuby is a bit faster than Ruby, and not even remotely comparable to straight Java (or Scala) performance. To the extent that I'm not even sure why you would mention this, when it's a wildly misleading point.
> Perhaps you’re talking about factoring tools? Maybe? I don’t see how that stops people from working on it together though.
> Source code control doesn’t care what language you write in, and that is the main tool we use to allow multiple people to collaboratively modify code so I’m feeling stumped.
For me, I'd be talking about tools like 'find me everywhere in my enormous codebase where this version of this method gets called, quickly'. I'd be talking about all the tools that make it so the maintenance programmer who has never seen your code before can alter it with a high degree of confidence that they're not going to accidentally break something.
The idea that if you have source code control you've got all you need is mad - it might be the case for small projects with low employee turnover, but it certainly doesn't generalise.
I think Ruby is a wonderful language, but it doesn't surprise me that you might want to switch to something else when you need to scale up both your operations and your enterprise size.
I will say that Java, Scala, or any compiled JVM language will run code a lot faster than Ruby. So will Go, so will C#, so will C++. Ruby isn't about fast, it's about expressiveness and programmer happiness.
If you wanted fast, you'd use a compiled language. If you want fast and ruby syntax, you might use Mirah, which is great btw.
In either case, I don't think that twitter made a mistake at all. Everything I've read says that 1 java server =~ 10 ruby servers and twitter isn't the only one who has said this. So, if you are talking about going from 10,000 machines to 1,000 machines that's saving 9,000 x $1,000 or about $900,000. At that scale machine cost does matter.
Most developers will never work at that scale so it is hard to understand that set of requirements, but at a certain point it's less about how fast you can write CRUD ops or other forms over data type things and more about making sure everything runs as fast and reliably as possible.
>>Everything I've read says that 1 java server =~ 10 ruby servers
I believe that is roughly correct. I'd add two points:
- This goes about the same for all the scripting languages and the faster compiled languages. (+)
- You should consider, and might even earn time, by first doing a scripting language implementation and then port it to C/C++/Java/etc, instead of going to C/C++/etc directly.
(But all this is more or less conventional wisdom, afaik.)
(+) Some lisp variants might be able to be as expressive as a scripting language while almost as fast as a compiled language. I don't really know that world since quite a few years.
I think that the general idea is that many companies that employ ruby based technologies have been making a switch away from them. The reason is simply scale, it's not a matter of "oh java's the new thing because everyone's switching back to it" Java has always been there and it will be around for a long time. People are just now realizing what a great platform the JVM is and since languages like Scala, Clojure and Groovy exist on-top of the JVM it allows large complex systems to be created in a language agnostic manner because they will all run on the same platform. Interop between other languages and java on the JVM aren't always nice but the fact that it's possible lets you leverage the many java resources that exist. For example SummingBird https://blog.twitter.com/2013/streaming-mapreduce-with-summi... . SummingBird is a Scala platform built on top of Storm (a Clojure based real time computation system) and Scalding (a Scala adaption of Cascading (a Java library) which is used for writing map-reduce jobs easily) and they work beautifully together. So in my opinion their switch to the JVM and embracing of it was a smart move.
I stopped reading when he said "TIOBE"; nothing invalidates an opinion on programming languages more clearly. There should be a law like Godwin's you can freely yell on the internet every time you see it, resulting in general agreement to doing so.
Are you sure? Seems to me he referenced TIOBE as a counter to the claim that Java ruled the Web, and wondered where Wired was getting their data, since the TIOBE data contradicts Wired's claim.
Let’s look at some numbers.
The TIOBE Programming Community Index shows us information regarding the relative popularity of languages. Java is high up there, Ruby also features. Scala and Clojure just don’t have enough people using them yet to break the top 20 which still includes COBOL, pilloried by all as an abomination to work with.
As of today Scala gets a mention at number 42 with 0.3% share.
This is unfair though, these numbers are not segmented by industry. We really need to look at numbers for web sites only.
Before we leave these numbers though it is interesting to point out that the Java ranking is very obviously trending downwards in the TIOBE index.
An intriguing part of the article asks about the Twitter Ruby-to-Java migration and posits that any language would be fine for a services architecture.
I'm not much of an object-oriented hacker, but when I was at larger IT firms, what several Java codebases shared was a common use of design patterns. It seems that this sharing of what may be considered a common vocabulary essentially makes it much easier for Java devs to jump into any project and follow the "accepted" usages in a way that perhaps other languages do not? So you are scaling up your dev team, it's infinitely easier to recruit acceptable Java candidates than say, Scala hackers? I do want to mention that my impression of Twitter is that there is definitely a polyglot JVM environment there (Scala, Clojure).
This feels like a bigger issue to me than I'd like it to be - in particular that Java programming opportunities vastly outnumber, say Ruby, Scala, Clojure, etc, hacking jobs.
I believe that strong typing is a much bigger factor than design patterns in supporting big codebases, and allowing people new to the codebase to contribute productively. Once you go beyond a small project, it's really nice to have the compiler on your side here!
In terms of the pool of total developers, the argument is that "A-grade" developers use Scala and/or Java, "B-grade" developers only use Java, so a Scala developer is rarer but more likely to be an A-grade developer. But of course, once this is known then the B-grade developers learn Scala. This is probably what happened to Ruby - it was little known, so all the Ruby developers were 10x. Now everyone learns Rails, and Rails + Ruby teams are rapidly reverting to the mean.
But yes, I do agree that we've figured out what good Java code looks like now, although it took us 10-15 years to do so. And we haven't hit the 10 year mark on Scala, Ruby, Clojure etc...
Once you go beyond a small project, it's really nice to have the compiler on your side here!
Absolutely. Mobile dev brought me back to compiled languages after a long stint in Ruby and the advantages of static typing for performance, correctness, and tooling are so overwhelming that I doubt I'll ever work in a dynamic language by choice again.
My experience with Java vs. Ruby is that Java applications are full of common design patterns because they "have to" be. Design patterns that are often considered necessary in Java, are often pointless or considered bad practice because of unnecessary complexity in other languages.
It may be easier to recruit java devs, but because more people have learned Java, not because it's easier to jump into new projects. That may be a valid reason to pick Java (I've picked PHP in the past, despite preferring Ruby, exactly based on the availability of "good enough" developers versed in the language).
As for the Twitter example, pretty much everything that has been publicly discussed about their original RoR app made it sound like a god-awful abomination in terms of architecture. OF COURSE it didn't scale. It seems to me at least that blaming Rails or Ruby or both is just a convenient way to avoid responsibility for the architecture and/or a convenient excuse for people that wanted to switch to Java/Scala/Clojure to use to justify the rewrites.
I've posted examples in the past on how they could have scaled it using Ruby, based on my own experiences with pure-Ruby queueing systems that would've been "simple" to scale even to their current size. As I've also pointed out, the primary thing they were trying to scale - messaging with potentially massive fan-out - is a well understood and "solved" problem: Turn your flat distribution lists into trees where each recipient is either a follower or the location of a subtree, and you can scale it arbitrarily. In practice, only a very small percentage of their users are "special" enough to require the "tree" treatment, to the point where you can handle it as a special sub-system.
[Heck, you could scale the foundation of Twitter with a bunch of mail servers and shell scripts to manage forwarding lists (no, it would not be costs-effective, but I have done high volume message queueing with mail servers before).]
Note that I'm not saying that another language might not be better for them - at their scale, it pays to optimise things that'd be a rounding error in a smaller environment - but at least their original RoR complaints were ludicrously off the mark. They might very well have valid reasons for switching, but the reasons they used to give were not it.
People try really hard to not accept this, and blame "architecture", but yeah, no.
You can read up on templating benchmarks from the very early days of Rails. You can easily spend tens of milliseconds just in template rendering.
The reason ActiveRecord, DataMapper, et all went with multiple Database round-trips had nothing to do with efficiency for the database and everything to do with Ruby's extreme performance deficits. A language so slow that a few thousand objects, with a dozen fields, and a relation, causing a couple million method calls during materialization would bring the interpreter to it's knees.
And now you've filled your response cycle with forced synchronous roundtrips to external services.
You don't even have the option to think about pure architecture. You're working under real constraints that don't just go away. Everything's got 'em, but the point where you have to start worrying about those kinds of issues in your average app in Scala is 100X later.
> As for the Twitter example... OF COURSE it didn't scale.
> It seems to me at least that blaming Rails or Ruby or
> both is just a convenient way to avoid responsibility for
> the architecture and/or a convenient excuse for people
> that wanted to switch to Java/Scala/Clojure to use to
> justify the rewrites.
Perhaps it was more responsible to avoid investing millions of dollars and years of effort in an over-architected system and just ship something quick and dirty, and see if people like it. All of us developers fantasize about the "write once, runs forever" system, but the reality is that the real requirements of any system are constantly changing, and we have to keep adapting, rewriting and re-architecting just to keep up.
Twitter Engineering recently posted about their move to the JVM in detail. It's worth a read.
It wasn't just about a move to a service oriented architecture. They also got well over 10X performance gain out of each machine by moving to the JVM. (They attribute it to better concurrency.)
I think the article's methodology is fine. It's not about the most popular languages in terms if developer head count. It's looking at what powers the biggest sites.
Great rant, with the most important takeaway that no software written by two guys in a garage ends up scaling to millions of users without major changes to its architecture:
> More importantly Twitter re-architected their system
> when they found performance problems...
I'd say a major advantage of the JVM is the lack of a GIL. Ruby has one. JRuby is slowly eroding that advantage.
That being said, it's much easier to write performant code on the JVM with the right tools. Static types and the discouragement of metaprogramming (except in Groovy, and let me tell you all about how fun that is) help with that.
Groovy would be the JVM's Ruby equivalent (if JRuby didn't exist). It's cleaner, faster to write in, and very non-obvious what is and isn't performant.
I think an SOA Ruby architecture would have helped Twitter, but as a startup built on the JVM (and experience with Rails/Ruby) the JVM's performance is a key factor for us. Not to say that Ruby or Python can't be performant too, but there's a reason it's used. http://www.techempower.com/benchmarks/#section=data-r6&hw=i7...
While these benchmarks aren't conclusive, Rails is definitely built to be friendly and encourage developer productivity, not raw performance. Many JVM frameworks are built to be faster at the expense of more time developing. We use Dropwizard and get a lot more bang for our buck when it comes to servers.
I don't think most developers really consider the performance aspects of downstream services ... top-level services rarely see the brunt of load, that happens a few levels into your SOA graph.
I sympathize with your anger at the Wired article. I too read it and cringed at the misrepresentations. It was either carelessness or a desire to dumb down the subject matter.
To my mind, the article is a demonstration of how poorly the media cover just about everything. When you or I read articles about subjects wherein we are not experts, we (or at least I) assume that they are well-researched and accurate. But when we read articles about subjects in which we have expertise, I find I'm routinely dismayed at the inaccuracies. I am no longer surprised by this, but still bothered. It casts a huge cloud of doubt over everything I read in the media.
Therefore, I found your rant delightful.
Nevertheless, I feel some of your points are subject to debate.
You criticize the characterization of JVM as the star of the show affording Twitter and others the ability to scale more easily, considering these efforts were necessarily coupled with a rearchitecture of their entire infrastructure. Of course there is truth in your point, but minimizing the role of the JVM is unfair. The JVM, and especially its higher-performance languages—which include Java, Scala, and Clojure, is a big star in our web application frameworks benchmarks charts [1].
With respect to high scalability opinions vary, but I lean toward the side that believes that while very high scale is ultimately a matter of horizontal addition of servers, a great deal of complexity and pain can be avoided by servers simply being fast at what they do. When platforms perform not merely a few percentage points apart but an order of magnitude (or more!), it's not reasonable—again in my opinion—to ignore raw performance. Simply being faster—massively faster—is like opening a pressure relief valve that gives you a lot of architectural breathing room.
Yes, Ruby runs on the JVM, but not all languages on the JVM have proven to reap the benefits. We have not yet observed JRuby demonstrating a compelling superiority to MRI Ruby.
Also, as others here have pointed out, many believe that maintaining a large system is aided by a language with type safety. I personally am considerably more productive with a modern statically typed language than any dynamic language. When I make typo-grade errors in dynamic code, and that code makes it to a testing round, or even worse a staging or production environment, I have lost huge amounts of productivity. I end up feeling embarrassed and I long for a static language to prevent this kind of silly failure.
Type safety preference isn't a matter that will be settled any time soon or ever, but I believe that at least the general consensus is that for large systems, stricter languages can make maintenance less painful.
> To my mind, the article is a demonstration of how poorly the media cover just about everything. When you or I read articles about subjects wherein we are not experts, we (or at least I) assume that they are well-researched and accurate. But when we read articles about subjects in which we have expertise, I find I'm routinely dismayed at the inaccuracies. I am no longer surprised by this, but still bothered. It casts a huge cloud of doubt over everything I read in the media.
This is informally called the Gell-Mann Amnesia effect.
But journalists are journalists. They have to have a superficial acquaintance with a large number of subjects that is at least mostly accurate. What you've taken years to learn they only have the luxury of a two week or three week deadline to learn. The reason they are not experts in your field is because, well, they are not experts in your field, and they have the unenviable task of communicating difficult subject matter about which they are not an expert to the general public in a short amount of time.
I don't think it's accurate to say that the news is mostly inaccurate. We just subject material about subject matter we know to a higher threshold of scrutiny than material about subject matter we don't know. We tend to see the inaccuracies as crowding out the accuracies because, thanks to our precise technical knowledge, not getting it exactly right is the same as getting it completely wrong. But to the layman, this is not true. 85% right is not the same as completely wrong and completely useless. 90% right is a good try, not a worthless hack pontificiating about a subject matter he could never possibly in a million years actually understand.
As usual it's all relative. But let's not let our cognitive biases make us too cynical. People generally are trying their best.
Thanks for the feedback. I never made any point regarding type-safety so I'm just going to gloss over that. I have no issues with static typing.
Also, I did not mean to say that you should not move to something that inherently runs faster, that would be perverse. However if I have a system written in a language that already works I would probably start by looking at algorithms and architecture before pulling the New Language card.
Once you have a set of interoperating services then re-evaluate which, if any, individual services are my performance bottlenecks that could benefit from the effort of re-implementation in a new language.
The really interesting discussion of course would be one that included engineers from Twitter and their recall about what happened.
I have been in enough design meetings to know that their strategy didn't emerge fully formed from the mouth of one person.
Lots of words, but once again they boil down to your intentionally skewed benchmarks and your personal opinion that performance overrides any other concern and that dynamic and non-JVM languages have no advantages. This is oversimplified debate-club propaganda presented as fact.
What's particularly funny is that today, Java is the poster child for performance. I remember when I was giving a talk in 1999 where I said that HotSpot is a game changer, and its lacking performance will not be an issue any more, I was laughed at.
I don't think Java is as bad a language as some people think it is.
Personally I write very concise Java code that is full of functional ideas. Tools like Guava, Spring and language features such as static imports, as well as just having the value that verbosity is a problem, can lead to Java code that is fast to write and easy to understand.
When people are doing rapid prototyping in a REPL, I'm writing unit tests in my IDE, and rather than scrolling off the screen and into oblivion, these tests are incorporated into my project right away and always watching my butt.
If you want a REPL, Groovy is great. For instance, one time I was trying to figure out what was going wrong with a Hazelcast cluster, and I was able to create a Hazelcast client in Groovy just by puncing in the maven coordinates for Hazelcast and it was a lot like working with the MySQL or mongodb db shells.
I think the main problem with Java is the total complexity of the ecosystem, and the fact that it requires wisdom to use well more so than intelligence.
Personally I have been bouncing back and forth between Java and other programming languages for years. I was programming it back when it was in beta, and back then it took me about a year of experience to be able to write reusuable OO code that was worth reusing.
More recently I got back into with two gigs for companies that had huge maven systems (30+ projects) and it was stressful a while, particularly when a large number of SNAPSHOT version releases, my internet connection was slow, and everything was half-baked.
In a situation like that where you have to get on top of everything that's specific to a large codebase, you've got a 20-minute build process, no real project management (in one place the project manager quit the team after she got bitched out by my boss) and, if you're really unlucky, you're the last person to touch the code before it goes to QA, it's stressful.
When I went back to coding for my own account, I read the maven documentation from cover to cover on my nook when I was working out at the gym and now my build process is faster than my car can drive the quarter mile and I intend to keep it that way.
I think it is tougher for new programmers to get on board of it and a lot of the problem is that companies are trying to look for 10x programmers when they really should be building a 10x organization.
For instance, a 10x change can be had by getting a 20 minutes build process down to 2 minutes. It's not science fiction. Sometimes team members and the boss will say that this "isn't a customer requirement" or "you're not being a team player" if you want to speed up the build, but I know the customer wants the job done quickly and the job done right, and speeding up the build process 10 times is the fastest way to attain either of those goals.
> When people are doing rapid prototyping in a REPL, I'm writing unit tests in my IDE, and rather than scrolling off the screen and into oblivion, these tests are incorporated into my project right away and always watching my butt.
THIS!
Many programmers tend to prefer the cowboy style of coding: jump to their favorite IDE/editor and code straight away with less consideration of anything else (productive workflow, etc).
But you're right, the initial complexity (complexity of the ecosystem) to setup a Java project is a world of hurt for people who have limited experience/knowledge vs Python/Ruby ecosystem.
With Java, it's more of a love-hate relationship probably because the language has been around for far too long :).
Love it because it's stable, well-known quantity, have standards (can be a double-edge sword) that many people have to support, and awesome collection of tools and libraries that no other platform/language ecosystems can match.
Hate it because setting things up (IDE, projects, etc) and writing certain part of code could be a bit verbose/more work.
PS: I also go back and forth between various programming languages (JS+Python mostly)
The major issues for a lot of people are simply cosmetic clutter: having to specify types everywhere rather than some basic type inference, for example.
Other problems are deeper: no first-class functions, no tail-call recursion support in the JVM (Clojure has a hack to deal with this), no map/fold/etc. equivalents that can potentially aid with parallelization. I think 1.8 addresses some of these.
Personally, I can live without the modern language niceties. I would really, really like type inference though. Oh, and unsigned ints too.
Java does have its own form of type inference that revolves around Generic functions and once you make peace with it, life isn't so bad. Rather than writing
var xyzzy=new ArrayList<Foo>();
you can write
List<Foo> xyzzy=Lists.newArrayList();
Similarly, the lack of first class functions can be worked around by various means. The Guava toolkit contains map/fold and other functional operations, and when worse comes to worse you can write
for(Thing that:things)
fn(that);
In cases where you might be tempted to do something that's more verbose than if you tried to do it the cool way.
There is a lot to like about JDK8 but unfortunately I'm coding to JDK6 these days instead of JDK7 because it's a waste of time dealing with trouble tickets caused by people who are running the wrong Java version and don't know what is going on.
Unfortunately this is the state of journalism today. In the length of an article it is only possible to make one point, at least outside of a dense article aimed at a specialist audience. Additionally the article should seem to be relevant, that means that its single point need to be exaggerated until it seems earth shattering. And journalists are not really payed to research a topic in depth, but to write articles.
So this is at least the impression I get from every article, where I actually know something about the topic. In theoretical physics every article describes one paper, and calls it a break through towards quantum gravity. Independent if this is a cute toy model, a interesting trick to calculate some technical problem or a actual attempt at quantum gravity, in which case there is certainly no data to back up the theory and the theory is almost certainly wrong. Another thing is any reporting about nuclear energy, either the article is wildly exaggerating the dangers of nuclear energy or it is wildly overstating the necessity of nuclear energy. ( Nuclear energy is IMHO a particularly nasty example for articles, because the dangers are extremely model dependent, so to inform a lay audience one would need an entire book.) And for programming, well the wired article did not strike me as a particular bad offender.
This would be tolerable, if it would not bias the political process. The current German president, Joachim Gauck, is a nice example. [1] In 2010 the, in this context center-left, SPD had no chance at electing their own candidate, so they attempted a PR coup by nominating a guy who is a hero for conservatives, while the conservative CDU did elect a career politician. Much to the dismay of conservative talking heads. Last year, the president had to resign [2] and chancellor Merkel did nominate Gauck, knowing that now the SPD is on the hook because he was their candidate. And all political commentators in Germany did just dust off their opinion pieces from two years ago.
In conclusion, I have no idea how to fix journalism. But I think that better journalism is badly needed. Or else the control loop for the political process has less and less input from actual facts and deep thought and is only controlled by some self consistent PR bullshit.
Edit: Clarified the conclusion.
[1] In Germany the president has a largely ceremonial role and is elected by an assembly of the parliament and (some members of) the state parliaments.
As someone working close with tech writers, I suggest (project) that the thing that precipitated this article is the realization that the JVM can run multiple languages. That's the kind of thing that is just science-fiction-y and just break-the-rules-y enough that it blows tech writers' minds and to some extent their audience.
It speaks to the part of them (us) that wants to believe things like, "I knew that big company Sun couldn't do everything perfect, because they're BIG and they're a CORPORATION. I knew there was a deeper way to use technology, and isn't it just inspiring/trippy/deep that a VIRTUAL machine can so something it's totally unintended for?!?!?! lolomgcyber those hackers are breaking the rules established by The Man so we can have our super-slick free text messaging capabilities #mind #blown"
I don't work at Twitter, and I'm whitewashing a bit because I'm at work, but the consistent line I've heard from engineers there[1] and from the press[2] is performance. Plus, Scala has shares some linguistic niceties with Ruby that Java doesn't, so it's a little bit more of a natural fit.
From my own experience with Scala, the reason to go in a service-oriented direction is, to say the least, make your compiles go faster over smaller isolated bits of functionality in your app. Scala type safety is awesome. Scala compiles, much less so.
Ruby still runs at Twitter, but my understanding is that the more recent engineers there are afraid to touch what's there for fear of blowing things up.
[1] http://blog.redfin.com/devblog/2010/05/how_and_why_twitter_u...
[2] http://techcrunch.com/2008/05/01/twitter-said-to-be-abandoni...