I'm not able to watch the video at my current computer
but its actually really typical that when something becomes an advantage for being selected to a certain pool
the success of the those in the pool after the selection will negatively correlate with that thing
---------
the really obvious example of this is the hockey birthday thing from Malcolm Gladwell's Outliers
people with the earlier birthdays were more likely to make it past each selection stage in becoming an NHL player
but those with the later birthdays who were able to be selected in spite of their later birthdays, were typically more successful after the selection
---------
The authors contend that the strategy might actually work against a team's success because they found that players born later in the year and drafted later actually had more productive hockey careers.
Deaner said the study showed that men drafted in the second half of the year were about twice as likely to have successful careers in the NHL ??? reaching benchmarks like 400 games played or 200 points scored ??? than those born earlier in the year.
"If the team wasn't making this mistake, they probably would have been more successful," he said. "The guys born in the first part of the year are much more likely to be busts."
I favour the simpler explanation: Campbell's law. [0] If programming contests are favoured, then people will optimize for them. Simplify/generalize even further and you get Goodhart's law. [1]
That's an excellent point! I can think of another example: In the movie "Hidden Figures" the black, female engineers at NASA are much better than their white, male counterparts simply because it was harder for them to get in. Perhaps the opposite is true today of engineering students at colleges with affirmative action?
Alumni preference is at least as big or bigger a factor in admissions than affirmative action. Many of the schools were segregated not so long ago, so you can guess how that affects the alumni pool.
This suggests that you think there is no perception bias that affirmative action is trying to counteract. You might be interested to know that when people try to correct for bias, they typically undercorrect.
Also, for people coming from more challenging circumstances, getting the same results is literally more impressive.
He probably meant that people who do programming competitions are more likely to pass Google interview (they'll be better at doing algorithmic questions quickly) but are not necessarily better suited to do the actual job.
Just like hockey players born earlier in the year were more likely to be drafted.
* Prior to officially selecting candidates based on performance in problems derived from programming competitions, candidates who excelled at programming competitions were likely to do well on the job.
* That correlation was observed on a wide scale by employers, so many companies -- Google chief among them -- started incorporating such questions into the official interviews.
* Candidates now observed the change in employer interviewing methods on a wide scale and adapted their preparation methods. This fundamentally changed the pool of people good at programming competition problems in such a way as to reduce the correlation between the original signal (good at algorithmic problems) and the goal (good at the job).
* Overall, widespread acknowledgment -- and all consequent changes in behavior -- of the original correlation between the signal and the goal significantly reduced the quality of the correlation.
I think it's simpler than this. I don't think there is much outright gaming of the signal.
Programming interviews and programming competitions are very similar, much more similar than programming interviews and real world software engineering. When you are selecting top programming competition competitors, you are implicitly selecting people who will absolutely smash your (non-design) programming interview questions. This has little to do with their effectiveness as software engineers.
So if you measure hiring candidates by their performance in programming competitions, everyone will manage their own skills towards doing better in competitions.
Yes but not just that. I think the bigger trend is that training oneself to excel at those types of problems meant (in addition to other things) one thing about you back then (that you were really into programming). Now that same behavior likely means that you want to get a nice job at one of the big tech companies, as a result of them publically selecting for that. These are fuzzy indicators to begin with, but they're definitely different fuzz.
> I think the bigger trend is that training oneself to excel at those types of problems meant (in addition to other things) one thing about you back then (that you were really into programming). Now that same behavior likely means that you want to get a nice job at one of the big tech companies, as a result of them publically selecting for that.
Thanks for the explanation! The idea being discussed didn't quite click for me until I read this
Was it a proper double-blind study substituting interview result for rnd() to make a hiring decision? And established correlation a few years down the line?
Because what you've stated: 'data shows' ... "successfully pass go on to be successful" - sounds like cargo cult science or pseudoscience to me.
Do you really need to do the double-blind study, though? The median person cannot program at all, so using rnd() couldn't possibly have better than a 50% success rate (certainly lower; 50% is just a round number that I'm sure is larger than the number of programmers). Is Google doing worse than that? I kinda doubt it.
Yes. You can do it at each stage as well. And we are not talking about hiring a general employee but a CS or programmer. So employ rnd() after a resume screen. Google has already found that GPA and brain teasers had no positive effect. This is similar and it would be an interesting experiment.
You've stated 'data shows'. So my question was - was it a proper double blind study?
Because it is definitely possible to do it properly. Substitute the results (or partial results) of the interview with rnd(), use it for hiring decision for a subset of candidates. Keep this information confidential. Establish if parts of your interview process don't perform better than randomness a few years down the line.
It's possible to do. Only I don't think this was done. And if it was not done, and the method was some 'data shows' with hand-waving - it would be under definition of pseudoscience.
You're calling most scientific journals psuedoscience -- even the hard sciences, like astrophysics. Nobody's doing controlled studies of supernovas, they're drawing inferences from observational data. I don't think your definition of science is sustainable.
Definetely. I arrived at the final stage in a Google interview which I failed and I can confirm that out of the 5 interviews three were based on puzzles that I later discovered are found in books for coding competions. On two I did a good job to work out a solution myself but I completely got the third wrong. Who prepares for this kind of competitions has a huge advantage in these kind of interviews.
That's how I felt trying to find a job in the bay area. If I wanted to compete with the top talent then I needed to prepare for the interviews and practice those problems.
Your interpretation is too weak. It is not just "not necessarily better", it is the stronger "on average are probably worse".
The reason is that programming competitions give more of a boost to your odds of being selected than to how well you'll do on the job. So people who otherwise wouldn't have gotten in now will, and will not perform as well as the people that they displaced.
Which is what happened with hockey players. Being born at the right time of year put you in a bucket with people who were slightly younger than you. Which improved your performance on the tests, but didn't matter once you all grew up. So slightly worse people at the right time of year displaced slightly better at the wrong time of year, and the average came out that people who got through and were born in the latter half of the year were actually better.
The hypothesis is that the older kids are bigger stronger but the younger kids are more skilled. And apparently this holds up all the way to the NHL where the skill edge overcomes age.
It's sort of like Simpson's paradox. It's relevant because the statistical "paradox" (contradiction of intuition) described in Outliers is similar to the one in the video.
but its actually really typical that when something becomes an advantage for being selected to a certain pool
the success of the those in the pool after the selection will negatively correlate with that thing
---------
the really obvious example of this is the hockey birthday thing from Malcolm Gladwell's Outliers
people with the earlier birthdays were more likely to make it past each selection stage in becoming an NHL player
but those with the later birthdays who were able to be selected in spite of their later birthdays, were typically more successful after the selection
---------
The authors contend that the strategy might actually work against a team's success because they found that players born later in the year and drafted later actually had more productive hockey careers.
Deaner said the study showed that men drafted in the second half of the year were about twice as likely to have successful careers in the NHL ??? reaching benchmarks like 400 games played or 200 points scored ??? than those born earlier in the year.
"If the team wasn't making this mistake, they probably would have been more successful," he said. "The guys born in the first part of the year are much more likely to be busts."
https://www.nhl.com/news/study-suggests-nhl-has-bias-in-favo...