It seems to me that TDD works against both bottom-up and exploratory programming.
Bottom-up programming develops tiny functions that are each trivially correct. These tiny functions form a base on which the next layer of tiny functions is built, and so on -- each layer written in the layer below. Most of these tiny functions begin life as tests, which are then generalized and encapsulated in a function. So in that sense BUP is sort of test-driven. But if it's necessary to write further tests to determine the correctness of these tiny functions, You're Doing It Worng.
In exploratory programming you're not sure where you're going yet. The development path during exploratory programming looks like a crooked tree with lots of stubby dead-end branches. If you first developed tests for each of those stubs, only to find that that's not where you're going, you may never stop writing tests, assuming you even know what to test for. Plus, testing during exploration would kill the flow. You can't be a pioneer and a settler at the same time.
I'm not saying not to test. And I'm not saying not to use TDD. TDD is probably great for top-down projects where you already know where you're going. But if it preempts the process by which many of the great programs were written -- if it preempts hacking -- then it's not the final word.
My experience is that TDD helps you write code that consists of tiny functions that are trivially correct. For the simple reason that this is the easiest way to write a test for.
As for the exploratory programming argument, a key element in TDD is to write one test and the make it succeed. You are not supposed to write a lot of tests and then the code that makes them succeed. So unless you are so exploratory that you don't even know what you want your next function to do, this would not hinder you either.
Does it seem weird to anybody else that about half the list of prominent projects using TDD he listed are testing tools?
That's like saying type systems are good because they help you write sound proofs about type systems. I know they can be really useful, but it's an incredibly circular argument.
It seems really weird. I've been trying to get an answer to the question "what popular apps have actually been built using TDD?" for a while, without success. All of the entries on his list are either libraries, programming languages, or testing tools and frameworks. Can anyone name actual apps that TDD has blessed us with?
Much of IMVU's client and website (tens of millions of customers) were written with TDD. I've long been a proponent of TDD once you pass the prototyping stage and you need to write a production-ready component with [most of] the edge cases figured out.
The "red green refactor" loop is more like:
1) What's the next problem this software needs to solve?
2) Write a failing test that demonstrates the missing need.
3) Implement said behavior.
4) Refactor until you don't see anything else to refactor.
I think step #1 is one of the most important and yet I don't see much discussion of it. Pick the most "key" tests to write first, whether that be your happy path or the functionality that has the most risk.
On refactoring... I work with people who TDD all of their work and other who write unit tests afterwards. In my experience, those who test-drive are more comfortable refactoring otherwise. When I was test-driving imvu.task ( http://thespeedbump.livejournal.com/63798.html ), I spent a great deal of time just staring at the code after it was written... it's very continuation-heavy code... and suddenly I saw a refactoring that collapsed it by 1/3. We've hardly touched the code since. It solves the right problem with the right amount of code and I think it's only had a couple very obscure race conditions.
In summary, TDD is one of my favorite tools for focusing my efforts so we build what we need, but only what we need.
More importantly, that were done with TDD as opposed to unit testing, testing with a repl, etc.? A lot of claims about TDD's benefits seem to take credit for testing in general. (Automated testing is a clear win, IMHO.) Insisting on writing tests upfront does mean that you test consistently, but then it's hardly the primary cause.
I'm sure it's a highly useful tool in some cases, but the people pushing it uncritically make it harder to tell when it's actually a good fit. In my experience, it's very useful for safely making changes in a legacy code, but I suspect it can encourage over-engineering surface details or getting caught in local maxima when used for initial design. (I usually just design with a REPL or notepad/whiteboard.)
> All of the entries on his list are either libraries, programming languages, or testing tools and frameworks.
I've never used or looked much into TDD, but it seems to me that that's where TDD would be best: areas where input is easily done programmatically & where output can be easily checked for whatever property. How does one test a big GUI app - write some sort of binding to xlib to move the mouse around and go through routines? I'm sure it's possible, but I suspect it's pretty difficult to come up with useful but reliable tests.
That's true of automated testing in general, really. It helps to design the GUI as a thin layer around a bunch of code that can be tested, scripted, etc. in isolation, rather than tangling the major functionality with the GUI code itself.
It's hard not to see that as a list of clients of consulting firms rather than a list of serious open source software built with TDD.
I know the Rubinious project was built aiming to use TDD and it doesn't seem to have gone anywhere YET. As far as I can tell, a compiler is one beast that needs a strong abstract design first rather than a set of tests first - but I'm prepared to be surprised.
Pivotallabs lists Twitter first but I would be rather surprised if someone could verify that a substantial part of Twitter's app has been built using TDD (go ahead, surprise me).
One of the comments on the original post claims open source apps don't count because they "are not subject to many of the challenges the average developer faces in a corporate environment" so it appears TDD fans can't win either way.
But since you're looking for open source success stories there was some very positive noises about TDD from a Twisted dev recently:
"I would strongly recommend at least practicing some TDD, too. Learning how to write tests was useful for me, but practicing TDD really expanded my design skills in a bunch of surprising directions that were almost unrelated to testing. I originally felt as you do, that I wasn't disciplined enough to do it; now I feel like I'm too undisciplined not to do it ;-)."
Though he specifically states that Twisted didn't start out TDD, which he regrets, but maybe that disqualifies it.
Regarding Rubinius (and JRuby that was listed in the original post) I'm not sure trying to create a spec for an existing language implementation and then try to meet it would count as "true" TDD if people care about such differences. But maybe they do TDD for other bits lower in the stack, I'm not sure.
I've only scanned the Rubinius code and noted their blogs/announcements but ... it seems like they shifted from one basic model of their virtual machine to another in the middle of their process and I'm quite skeptical that this either indicated they doing well or that it would be considered a victory for TDD. Looking from the outside, it looked more like "we thought that TDD would let us get by without design but we were wrong...".
I've heard TDD described as "write a test, write a function, repeat" and THAT is not enough. You definitely need a guiding design as well.
I worked as a consultant at Twitter for a while [summer 2008] and saw no evidence of TDD in action. More like - someone monkey patch the running server, we're on TechCrunch again!
The circularity-argument for type systems is actually correct, but that is an aside. Type systems on programming languages are proof systems, so they can be used as tools for building better type systems.
There is a big lack of TDD in highly succesful Open Source endeavors. Part of this is perhaps because a failed project never get attention. But also note that defects in Open Source projects are good. They instill new programmers to come to the project and work on it because there are some easy low-hanging fruit. Furthermore, as exemplified by Keith Packard recently (see http://lwn.net/Articles/354408/ -- which unfortunately is behind the Iron Curtain for 24 hours more). Keith mentions that he sometimes deliberately breaks some old code by introducing obvious bugs into it. Then, when nobody have complained for a couple of years, it can be removed on the grounds that it was not interesting.
Interesting point - The social factors of Open Source have numerous subtle consequences. It reminds me of how someone noted that it can be easier to get help with (some) Open Source software by going on IRC and spouting misinformation (and getting corrected by a mob), rather than asking for help and getting treated like a stupid newbie.
My note about type systems doesn't have anything to do with whether it's correct or not - just, if the first example people give for the usefulness of type systems is more type systems, it makes me wonder how much consideration the speaker gives practicality. (Then again, I'm not a grad student...or an agile methodology coach.)
Was just about to right a similar comment before I saw yours. Also, productivity "gurus" who use their god-like productive capacity to... talk about productivity. Good times.
Probably not many, but I don't think that is necessary. He just has to show us a convincing list of projects (e), and demonstrate how these projects improved with the introduction of TDD, WITHOUT blurring the line between TDD and unit testing. Not student assignments, but real world projects where real jobs, real money, real stakes are on the line. The problem here is that if TDD is as effective as the zealots and XP gurus claim it is, then the evidence for it should be overwhelming.
Edit:
(e) Listing projects that are: 1. real (not just student, testing or open source projects); 2. large enough that real stakes are on the line (I want to see an example of a 2 MLOC system before and after TDD, not the 70 KLOC of Fitnesse) would make me pay attention. Then I want to see how TDD improved the state of the project (and that picture has to include things like productivity and cost analysis of test first testing everything). Then I want to see why the same results could not have been achieved with "test last" or "test whenever" unit testing. Then I would want to see an explanation for the rate of success other people have had without ever practising TDD - because if TDD is as effective as the zealots claim, and if the zealots make a living making those claims, then that explanation HAS to be there.
"An Initial Investigation of Test Driven Development in Industry" by Boby George and Laurie Williams, and "Realizing quality improvement through test driven development: results and experiences of four industrial teams" by Naggapan et al.
and I am not too impressed. The first study has 6 TDD teams and 6 control teams. Our null hypothesis is that these teams produce exactly the same quality of code, measured in defect rate. Now, any sane study would take a look into (statistical) power, ie the probability the test will reject the null hypothesis. If we further look at the box-plots, we see that the overlap between the "treatment" and control group samples is pretty big. Hence, we will need to test for any statistic significance before coming to the conclusion. In fact, it may even be that we can reject the idea that writing in a TDD style takes 16% more time on average.
The second study is much worse. It quotes studies secondhand, only has 4 studies (with 4 "controls" being completely different systems). It generally fails to the problem of lurking variables, or confounding: That some unmeasured external factor affects the study. For instance, it is put forth that the Microsoft teams in the study employ many methods for minimizing program defects including static analysis. The other grande problem here is that there are no assesment of the control-group systems at all. They will, invariably differ in more than one variable and the question is: Have we measured to benefits of TDD or have we measured the benefits of something else? We can't say for sure.
Finally, we only test statically typed languages with a subtyping construct (Java, C#, C++). I think (and I stress it is a hypothesis) dynamically typed languages would benefit considerably more with a TDD system in place, and I hypothesize more expressively typed languages (ML, Haskell) would benefit less. But I have no good idea of how to design an experiment that can show this hypothesis true.
Actually, now that you point me toward the real data...
It makes me even more skeptical of TDD.
I would expect most projects using just about any new methodology to look notably better in tests through the programmer's equivalent of the placebo effect: a new methodology imparts energy and optimism compared to just doing the same old.
This would lead me to deduce that once the "shine" wears off TDD, it would actually give no improvement in quality and result in considerably longer development time...
What I see far too often in these debates is the following:
Person A claims "Process P is good!"
Person B retorts "Sez you, can't prove it to my satisfaction!"
And the debate rolls on and on about whether we can prove P or not. I am not blaming Person B, but the question I want to put to them is this: Can you prove that whatever you are doing now works? Or is it just a case of you feel it works for you?
The debate would be more useful for all involved if it were re-framed in terms of asking what problems TDD is well-suited to, rather than arguing over whether it's (implicitly, always) a net win or not. (This is also true for many other techniques, I think.)
Another similar, and superficially easy, improvement in the level of conversation would be if, instead of claiming X "sucks" or "rocks" in absolute terms, people ranked things against other known items.
For example, if you think unit testing is generally good but TDD is a bad idea then give some context to that: is TDD worse than no unit tests at all? Is it worse than low unit test coverage say 20, 30, 40%? Or off at a tangent TDD makes Java programming feel like Ruby (or vice versa), just something for people to intellectually grab onto in terms of scale.
I ask because not unit testing at all seems (to me) to be further from unit testing than TDD is, yet from this current discussion it would appear that unit test fans have more issues with the latter than the former.
It would also be useful if people would be more explicit and specific about the techniques they are actually discussing. Consider, for example, the earlier comments about confusing or conflating TDD and unit testing.
Part of the reason the debate is muddled is that TDD forces one to use unit / regression testing. People promoting TDD often take credit for the benefits of those, when evaluating TDD alone concerns whether writing tests upfront as a design technique is actually better, provided tests are still written.
"Can you prove that whatever you are doing now works? Or is it just a case of you feel it works for you?"
Good Question.
Another question is "Do you go around calling legendary programmers who don't use your practices, yet ship product, "Stone Age Programmers"? Or the questioning of your favorite practice X as "X derangement syndrome"? If so why?
My comment said absolutely nothing in support of or in refutation of statements along those lines, so I am disinclined to pursue this line of discussion.
But if it amuses you, consider the question of whether people like Alan Turning and John McCarthy operated in the stone age of computer programming.
For that matter, consider the question of whether we are still in the stone age. I am inclined to say we are remarkably unenlightened. Some people ship without process, we don't really know why other than to make vague statement like "they're legends." Others have process but fail to ship and we again don't fully understand why.
Although our tools are concrete, our application of them seems to be a mystery with very little hard research or true science behind it.
"My comment said absolutely nothing in support of or in refutation of statements along those lines,"
Oh I never said or implied you did. That is why I phrased my comment as "Another question .
I unreservedly apologise if anything i wrote here I gave that impression.
", consider the question of whether people like Alan Turning and John McCarthy operated in the stone age of computer programming."
but those are not the people being addressed as "stone Age programmers". The people interviewed in CaW (which is how this hole brouhaha started in the first place) and Joel Spolsky are hardly "Stone Age" programmers just because they don't "evolve design" using TDD.
As per Bob Martin , people who don't use TDD are "Stone age". People who question the value of TDD are "deranged". This sounds like religious fanaticism to me.
To extend your original comment.
"Person A claims "Process P is good!"
Person B retorts "Sez you, can't prove it to my satisfaction!""
Person A replies "That is because you are a deranged Stone Age Programmer".
Well, Robert can say whatever he wants about people and you are welcome to disparage him for it, God bless you both. But the reason why your question is another question is because it has nothing to do with my question, not even when phrased as an extension to my original imaginary dialogue.
My question is about people who reject some things for lack of empirical justification while accepting others without such justification. If you want to piggy-back off my question to criticize famous consultants, how about asking whether Robert Martin is actually Person B and not Person A?
I haven't read his work in enough detail to comment one way or another, but I certainly am open to the possibility that some of the proponents of agile methodology do so on the basis of very scant evidence while rejecting other approaches for lack of substantial justification.
" But the reason why your question is another question is because it has nothing to do with my question, not even when phrased as an extension to my original imaginary dialogue."
I never said it did (I am saying this the second time). You have a right to your opinion.
You were using a rhetorical device to put forward an opinion. So was I.
I wasn't aware it was somehow offensive, especially since I made it clear I was borrowing the device and not attacking your points and apologised for any inadvertent impressions to the contrary. . If that still offends you ... [shrug] whatever.
"I haven't read his work in enough detail"
I have.
"but I certainly am open to the possibility that some of the proponents of agile methodology do so on the basis of very scant evidence while rejecting other approaches for lack of substantial justification."
Fair enough. With that, I exit this thread in the interests of harmony and not wasting time hairsplitting.
I didn't take offense, nor did I actually argue against your point. I was aiming for vigorously trying to drag the thread it back to my point. I can understand your desire to call Mr. Martin to task for making bone-headed statements. I would join you, however I have made enough of my own mistakes that I have very little room for other people in the part of my brain devoted to bone-headed writing.
The research I've seen on TDD seems to essentially point to two things: quality and productivity is positively correlated with the number of tests, and people who use TDD tend to write more tests. People who write the same number of tests but don't use TDD get similar benefits to people who do use TDD. So TDD is itself useful primarily in that it ensures that you actually do write the tests. (Unfortunately I don't have any citations handy to back that up; that's just my recollection of what I've read on the subject over the years).
There's that ever-elusive "it improves design" argument for TDD, but that one's far harder to prove either way, and I've personally seen it cut both ways: sometimes things end up more nicely decomposed, and sometimes they end up wayyyyy too decomposed.
As for the design argument, I think that the thing that TDD actually encourages in terms of design is testability (something that I do think is desirable, in general). If you have high test coverage without using TDD, you've probably either taken testability into consideration before you wrote your code or you made some changes after the fact to accommodate your tests.
I'd definitely agree that TDD leads to testable code, which is usually a good thing. If you know how to write testable code, it's not as much of a win (in my opinion), since you tend to write with testing in mind. If you don't have experience with unit testing, though, doing strict TDD for a while can really help you learn what sort of constructs make for testable code.
The places I've seen TDD lead to less-than-desirable design is when some larger problem is broken into such tiny pieces with so many interfaces that it becomes hard to read through the code and follow the flow of control. That's certainly testable, but often times I personally get as much mileage out of writing tests one level up, against some higher-level abstraction or API that those other things are all components of, and not being so religious about breaking up all the little parts. In my experience those slightly-higher-level tests (they're not even "integration" tests, since they're still strictly against one logical component) tend to provide the same amount of value in terms of preventing regressions while requiring fewer compromises to make the little parts testable and giving more flexibility as to future refactoring. Essentially, you're treating a whole bunch of classes/parts as internal implementation details.
I think we see eye to eye here. The sort of "one level up" thing is what initially attracted me to spec testing because that seemed to not only give you tools for performing your tests, but also helped direct you towards what would be useful to test: the behaviors of the code that you actually care about. So when I use spec testing tools, I tend not to think about making sure that the low level function traverses the directories in a particular way, but rather that all of the directories have been renamed (for example).
> I personally get as much mileage out of writing tests one level up
This is why I prefer the rebranded "BDD" as opposed to the original "TDD".
I find that the perspective that BDD offers makes me write much more useful tests, since the ultimate reason we write objects in the first place is to use them, right? I don't care about how it works inside, I care about how it gets used. There's nothing technically different between BDD and TDD, it's a matter of philosophy, and I find BDD's take on things to be better.
Bottom-up programming develops tiny functions that are each trivially correct. These tiny functions form a base on which the next layer of tiny functions is built, and so on -- each layer written in the layer below. Most of these tiny functions begin life as tests, which are then generalized and encapsulated in a function. So in that sense BUP is sort of test-driven. But if it's necessary to write further tests to determine the correctness of these tiny functions, You're Doing It Worng.
In exploratory programming you're not sure where you're going yet. The development path during exploratory programming looks like a crooked tree with lots of stubby dead-end branches. If you first developed tests for each of those stubs, only to find that that's not where you're going, you may never stop writing tests, assuming you even know what to test for. Plus, testing during exploration would kill the flow. You can't be a pioneer and a settler at the same time.
I'm not saying not to test. And I'm not saying not to use TDD. TDD is probably great for top-down projects where you already know where you're going. But if it preempts the process by which many of the great programs were written -- if it preempts hacking -- then it's not the final word.