My field (classical music) has a similar application screening problem (lots of applicants, very few good ones). Our chosen screening process, the audition, is imperfect for many of the same reasons given for why puzzle solving sometimes falls short when it comes to screening for developers:
- The ability to perform on audition day doesn't necessarily correlate with the abilty to perform consistently in concert settings.
- Great solo players aren't always great ensemble players.
- It is very difficult to predict how an auditionee will mature over time.
That said, our field has yet to come up with anything better, and I think the organization I work for has developed a particularly good screening process:
On audition day, applicants are asked to perform some excerpts from literature our organization plays regularly, a prepared solo selection, and to do some sight-reading. This is pretty standard stuff for the clasical music industry and this preliminary round is done from behind a screen to help combat biases on the part of the panel. The excerpts are important because that's closest to the work applicants will be doing on a day to day basis if selected. The prepared solo helps show the candidates potential to excel in solo situations. The sight reading is crucial because it is a basic and necessary skill and correlates very highly with the applicant's quality of musical training.
The second round (generally 5-15 applicants from the initial pool of about 100) is where our organization does things a little differently. It consists of more excerpts and sight-reading, but this time applicants are asked to make adjustments to how they performed the selections (a little faster or slower, a little louder or softer, etc.). Observing how the candidates respond to these requests shows how nimble the musicians will be in a rehearsal setting, and also shows how well-prepared they are and how well they will take direction.
In the final round (generally 2-5 candidates), applicants are asked to perform alongside members of our organization. This demonstrates more responsiveness, how well the applicant listens, and how well the candidate performs in a group setting.
With that in mind, I often wonder why those looking to hire developers don't use a multi-stage "audition" process.
Step one could consist of some thing along the lines of "Here's a piece of code like what you might see in our organization. There's a bug in it. Find and fix the bug and send the revised code back to us." Step one could also consist of a short puzzle-type problem. Step one could be 100% automatic and would require the same amount of time for 100 applicants as it would for 1000 applicants.
Step two could be where applicants are asked to show samples of something they've coded in the past. This could also be where they are asked to perform a more difficult bug fix and a more difficult puzzle. In this round, applicants are told that their code will be read for documentation quality and use of good style.
Finally a final round could be done in-person where an applicant is asked to solve a problem with existing members of the team. This could be done in conjunction with one on one interview. This would help tell if the candidate is someone the team actually would enjoy working with, and how the candidate would be able to contribute.
I know I wouldn't mind submitting to such an application process because I could see how each step is relevant to what the company is looking for. It also shows that the company takes hiring seriously. Hiring is supposed to be hard. Knowing that every one else on the team has been able to excel in those circumstances gives me greater confidence that I'd be joining a great team.
Do you really see many talented programmers submitting to the process you're suggesting?
Interviewing is a two-way street. While you're trying to figure out my coding skills, I'm trying to figure out if I'm interested in what your company does and if I'd like to work with the people there. Personally if I knew I'll have to spend an hour of my time on interview challenges just for the chance to get some face-time with an engineer, you'd never even see my resume.
Based on what I have seen, the turnover rate of musicians in a symphony orchestra is much slower than the turnover rate within software development organizations.
That is a very good point. Most top-tier orchestras have tenure for their permanent positions and it's not uncommon for musicians to stay 20-40 years.
As a result, the consequences for a bad hire are extremely high. Many orchestras have the audition winner perform with the group for a probationary period of a season to help guard against this. That said, I read a lot on HN about the consequences of a bad hire for a an early team member, so although the timeframes are different, the fear is similar.
The long tenure (and prestige of the organisation) probably also contribute to the willingness of candidates to subject themselves to such a process. Getting someone who is currently employed to sacrifice multiple days of (paid or unpaid) holiday for your hiring process is presumptuous, particularly if they're actively looking for a new position and thus probably going through this with multiple companies simultaneously.
As you say, it's all about making the relevance of the process obvious to the candidate. Also, companies should be as accomodating as possible in terms of timing of appointments, etc.
Totally off-topic, but I'm really curious about what brings you to Hacker News? I wouldn't have expected a professional musician to spend time on a software-tech/startup site.
I'm a recording engineer, which increasingly means "data wrangler". A couple years ago I taught myself bash and then python in order to write a script to convert our archives from wav to mp3 (and inject metadata from a crystal reports file). That project has evolved into a web interface for searching and browsing our audio, video and photo archives. I probably spend about 3/4 of my time making recordings and 1/4 writing and maintaining code.
I don't remember what first brought me to HN, but I stay to learn about open-source projects I wouldn't otherwise find out about. The next version of my archives site is built with Flask and Twitter Bootstrap, both of which I learned about here. Also, my transcoding scripts use Kenneth Reitz's Envoy for the system calls.
Also because I realize that as much as I love music, my decision to become a recording engineer was probably ill-advised. There are days I strongly consider giving it up to move to the Bay Area and try developing full-time, hopefully at a startup.
Was a recording engineer too and stopped because it broke my ability to sit down and appreciate listening to music: I had to analyse it "professionally". Mick Jagger said once he would only listen to classic on the radio. I can believe that. In some areas of human creation being an amateur is necessary. A simple example: China has an enormous corpus of classical painting and poetry, no part of it has been created by professionals.
Add: to me writing code is an activity that can be enjoyed professionally, unlike music.
- The ability to perform on audition day doesn't necessarily correlate with the abilty to perform consistently in concert settings. - Great solo players aren't always great ensemble players. - It is very difficult to predict how an auditionee will mature over time.
That said, our field has yet to come up with anything better, and I think the organization I work for has developed a particularly good screening process:
On audition day, applicants are asked to perform some excerpts from literature our organization plays regularly, a prepared solo selection, and to do some sight-reading. This is pretty standard stuff for the clasical music industry and this preliminary round is done from behind a screen to help combat biases on the part of the panel. The excerpts are important because that's closest to the work applicants will be doing on a day to day basis if selected. The prepared solo helps show the candidates potential to excel in solo situations. The sight reading is crucial because it is a basic and necessary skill and correlates very highly with the applicant's quality of musical training.
The second round (generally 5-15 applicants from the initial pool of about 100) is where our organization does things a little differently. It consists of more excerpts and sight-reading, but this time applicants are asked to make adjustments to how they performed the selections (a little faster or slower, a little louder or softer, etc.). Observing how the candidates respond to these requests shows how nimble the musicians will be in a rehearsal setting, and also shows how well-prepared they are and how well they will take direction.
In the final round (generally 2-5 candidates), applicants are asked to perform alongside members of our organization. This demonstrates more responsiveness, how well the applicant listens, and how well the candidate performs in a group setting.
With that in mind, I often wonder why those looking to hire developers don't use a multi-stage "audition" process.
Step one could consist of some thing along the lines of "Here's a piece of code like what you might see in our organization. There's a bug in it. Find and fix the bug and send the revised code back to us." Step one could also consist of a short puzzle-type problem. Step one could be 100% automatic and would require the same amount of time for 100 applicants as it would for 1000 applicants.
Step two could be where applicants are asked to show samples of something they've coded in the past. This could also be where they are asked to perform a more difficult bug fix and a more difficult puzzle. In this round, applicants are told that their code will be read for documentation quality and use of good style.
Finally a final round could be done in-person where an applicant is asked to solve a problem with existing members of the team. This could be done in conjunction with one on one interview. This would help tell if the candidate is someone the team actually would enjoy working with, and how the candidate would be able to contribute.
I know I wouldn't mind submitting to such an application process because I could see how each step is relevant to what the company is looking for. It also shows that the company takes hiring seriously. Hiring is supposed to be hard. Knowing that every one else on the team has been able to excel in those circumstances gives me greater confidence that I'd be joining a great team.