Using Amazon’s MTURK for Multiple Waves of Data Collection: Part 2 (Qualifications, Follow-up Survey Invitations and Tracking Responses between Surveys)

Justin

PhD University of Illinois at Urbana-Champaign School of Labor and Employment Relations

You may also like...

5 Responses

  1. Hello,

    I’m not really certain how we can really be sure in accuracy of data that participants collected for us… Is there some another type of checking its consistency despite the qualification test? does the mTurk work in that field as well, per HIT on basis?

    Thanks ahead for providing the answer, If any

    Marko

    • Justin says:

      Good question Marco. There are other quality controls procedures. You can also use MTURK to filter for workers with specific approval ratings. All said, the verdict is still out on the quality of MTURK. Ultimately there are limitations with any sample. MTURK seems to work well for some purposes. See the following for a couple published perspectives:

      Barger, P., Behrend, T. S., Sharek, D. J., & Sinar, E. F. (2011). I-O and the Crowd: Frequently Asked Questions About Using Mechanical Turk for Research. TIP: The Industrial-Organizational Psychologist, 49(2), 11–17.

      Richard N. Landers, & Tara S. Behrend. (2014). An Inconvenient Truth: Arbitrary Distinctions Between Organizational, Mechanical Turk, and Other Convenience Samples. Industrial and Organizational Psychology: Perspectives on Science and Practice. Retrieved from http://www.siop.org/journal/Article_1.aspx

  2. jewel khan says:

    I’m not really certain how we can really be sure in accuracy of data that participants collected for us… Is there some another type of checking its consistency despite the qualification test? does the mTurk work in that field as well, per HIT on basis?

    • Justin says:

      You can implement quality control items and manipulation checks in your data (as in other survey research). The quality of the data has been discussed in a few articles (see below for a couple examples). I’m not sure I follow your last question. Could you expand on it?

      Barger, P., Behrend, T. S., Sharek, D. J., & Sinar, E. F. (2011). I-O and the Crowd: Frequently Asked Questions About Using Mechanical Turk for Research. TIP: The Industrial-Organizational Psychologist, 49(2), 11–17.
      Richard N. Landers, & Tara S. Behrend. (2014). An Inconvenient Truth: Arbitrary Distinctions Between Organizational, Mechanical Turk, and Other Convenience Samples. Industrial and Organizational Psychology: Perspectives on Science and Practice. Retrieved from http://www.siop.org/journal/Article_1.aspx

  1. July 11, 2016

    […] invitation only HIT to get paid for the screening test. To set up an invitation only HIT, look at this blog post BUT ONLY  POINTS 3, 4 AND […]

Leave a Reply

Your email address will not be published. Required fields are marked *