The economics of Mechanical Turk
This is part of a series of Mechanical Turk blog posts – check out more at Mechanical Turk in Ruby, Mechanical Turk Pages in JQuery, and Building Mechanical Turk into Rails(coming soon).
In this post I want to go over just a few tips about deciding how much to pay and how to structure your HITs on Mechanical Turk for the best and most cost competitive results.
The business behind the HITs
At the heart of it, Mechanical Turk tasks are completed by real people, so unlike the other Amazon services, the pricing isn’t so simple. So when you start to place tasks on the network, it’s completion is dependent upon several factors, the largest being:
- Price
- Complexity
- Number of tasks
- Time of creation
The first two are obvious, but the combination of all three can make for a tricky time figuring out what to offer.
Number of tasks
The more task available the less reward you can offer. This works because a worker is looking at this from two angles:
- How long will it take to learn to complete the tasks?
- And how long is each task going to take to complete?
The first question is the biggest barrier to getting someone to work on your HIT’s. If the first few take quite a while to learn how to do, and their only 20 available, it’s not going to be very rewarding.
However offer 1000 and you can probably discount your price significantly, so submitting your HITs all at once has a tendency to encourage acceptance and completion.
Time of creation
Although there are people all across the globe working on Mechanical Turk, if your task requires a decent grasp of English, or knowledge of western culture(eg. Read this newspaper article and tell me if it’s positive of negative), you’re workers will primarily come from the US and Canada. This of course means that you need to schedule your HIT’s accordingly.
A reasonable person would expect that HITs in the system get randomly cycled through listing page, however the truth is that they are listed chronologically, and this means that if you don’t get them completed within the first hour, it’s unlikely they’ll be completed at all.
Complexity
There’s two types of complexity you need to worry about, but both have the tendency to chase away perfectly good workers.
The first is task complexity. Distill your task down to the most basic parts, and don’t expect everyone to understand your instructions. The wording should be simple, clear, and concise. You should be able to look over it and immediately understand what it is you’re asking for and how it should be done.
The second is UI complexity. A worker wants to be able to load up new tasks and complete them as fast as possible. What they don’t want is a website that slow, or javascript magic that doesn’t work in their browser. And don’t assume someone has the lastest browser when designing your external HITs. Make the design simple and easy to use. And for god sakes, don’t get crazy with the JS – the goal is a web application page that works well for the worker and allows them to use the key binding and events they are used to. Especially if it’s data entry, make sure the tab navigation works.
Price
The easiest way to figure out what your tasks are worth is to build a baseline guide. Start with a few(less than 20) and pick a price which seems inline with similar tasks. Also ask that each task be completed 4 times to ensure that it’s not just one worker doing it. Then posts the jobs, and check the response. If you don’t have them all completed within a few hours, cancel them and raise the price. Likewise if they’re competed quickly, it may be a sign you’re paying too much. You’re only going to figure this out through a little trial and error, and maybe a little gut instinct. You’ll get better an estimating the appropriate reward as you get more experience
Putting it together
It always looks easier than it is, but in this case we’ve got a lot of different variables to work with. The first step is to establish a price and build from that. You can also attempt to fit more than one task into a HIT; if they are simple, you may lower your cost per HIT by allowing workers to spend more time on the HITs and less on page reloads.
Integrating Mechanical Turk can be a daunting task – let us build a solution that’s easy to administer and get Mechanical Turk working for you and not just sucking up your time. We built and maintain the only open-source Ruby library for Mechanical Turk, so why not let us do the hard part. Give us a call!
blog comments powered by Disqus