> I’ve paid users on Mechanical Turk and later on oDesk to label data for me for some Machine Learning research.
What tasks (I'm assuming these involved huge batches if it was for a machine learning set) were economical to move from mTurk to oDesk? I've never used the latter but that seems to be a place to hire individual workers?
In this case, I hired people to perform sentiment tagging of status updates for the training of SVM regression models. oDesk allowed me to interview more carefully and ended up working better. I wrote a quick Rails scaffold that let people rapidly tag statuses.
Edit: Something like CloudFlower might also have worked well, but I didn't want to pay for it. The key was that I needed to ensure a baseline level of quality in all of the taggings. (Although, to some extent, I could take consensus labels.)
Anytime you require consistency with more involved training, go the oDesk route.
Mechanical Turk can produce quality results but you have to take measures to ensure that quality (gold standard question, triplicate entry, etc). Email me if you want to discuss further. jim.jones1@gmail.com
What tasks (I'm assuming these involved huge batches if it was for a machine learning set) were economical to move from mTurk to oDesk? I've never used the latter but that seems to be a place to hire individual workers?