{"id":195,"date":"2015-01-04T16:25:03","date_gmt":"2015-01-04T22:25:03","guid":{"rendered":"http:\/\/justinwiegand.com\/blog\/?p=195"},"modified":"2026-03-10T22:32:01","modified_gmt":"2026-03-11T05:32:01","slug":"using-amazons-mturk-for-multiple-waves-of-data-collection-part-1-external-survey-hosting-and-random-number-generators","status":"publish","type":"post","link":"https:\/\/justinwiegand.com\/blog\/?p=195","title":{"rendered":"Using Amazon\u2019s MTURK for Multiple Waves of Data Collection:  Part 1 (External Survey Hosting and Random Number Generators)"},"content":{"rendered":"<p>Amazon\u2019s MTURK is a promising way to collect data in the social sciences . \u00a0The sample (as compared to other convenience samples), timeliness, ease, and cost are all strong positive attributes of the service (Landers &amp; Behrend, 2014). \u00a0However, there is not a convenient method to collect multiple waves of data over time from the same respondents, thus precluding\u00a0an otherwise helpful reduction of common method bias (Podsakoff, MacKenzie, Lee, &amp; Podsakoff, 2003).<\/p>\n<p>After personally carrying out two waves of data collection (with a one-week time lag) for 1,000 respondents (MTURK Workers) with surveys hosted independent of MTURK, I would like to share the steps and methods used to track workers between the surveys and encourage participation in the follow-up (Part 2). \u00a0First though (Part 1), I lay the ground work for those using external survey hosting.<\/p>\n<ol>\n<li>To start, create a separate HIT on MTURK for each wave of data collection you plan to carry out. The HIT should contain:\n<ul>\n<li>A starting link to your independently hosted survey (e.g. Qualtrics)<\/li>\n<li>A single text box for data entry<\/li>\n<li>Instructions (i.e. \u201cClick the link to be redirected to our survey. Please, LEAVE this browser window open as you will be asked to paste a code provided in the survey in the box below.\u201d)\n<ol>\n<li>Be sure to emphasize keeping the browser window open, as in the example provided above.<\/li>\n<\/ol>\n<\/li>\n<li>This can all be easily accomplished through MTURK&#8217;s &#8220;Survey Link&#8221; template.<\/li>\n<\/ul>\n<\/li>\n<li>Create separate surveys for each round of data collection using your respective survey service.\n<ul>\n<li>Have the survey provide a randomly generated number at the end using a random number generator. \u00a0Qualtrics uses their <a href=\"http:\/\/www.qualtrics.com\/university\/researchsuite\/advanced-building\/survey-flow\/web-service\/#ExampleRandomNumberGenerator\">own web services<\/a> to provide a random number generator and I recommend them. \u00a0Tyler Burleigh\u00a0offers a nice walk through <a href=\"https:\/\/tylerburleigh.com\/mturk\/survey-completion-codes-in-qualtrics\/\">here\u00a0<\/a>(and this will guide you with most other survey services as well). \u00a0Survey Gizmo mentions an internal random number generator feature in <a href=\"http:\/\/surveygizmov4.helpgizmo.com\/help\/article\/link\/hidden-values#calculated-value-options\">this<\/a>\u00a0help document. \u00a0Feel free to give that a try if you cannot settle with Qualtrics.<\/li>\n<li>As a buffer, in case a problem occurs with your random number generator, I recommend asking for the respondents&#8217; WorkerID on the same page as the random number generator output or directly before (if possible and ethical, require this output in your survey).<\/li>\n<\/ul>\n<\/li>\n<li>Use MTURK&#8217;s CSV file download\/upload method to &#8220;mass-approve&#8221; HITs.\n<ul>\n<li>Click the &#8220;Manage&#8221; tab, then &#8220;Manage HITs Individually,&#8221; then &#8220;Download Results,&#8221; and again &#8220;Download Results&#8221; as in the screenshot below.<img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-204 size-large\" src=\"https:\/\/i0.wp.com\/justinwiegand.com\/blog\/wp-content\/uploads\/2015\/01\/mturk-download.jpg?resize=720%2C191\" alt=\"mturk download CSV screenshot\" width=\"720\" height=\"191\" srcset=\"https:\/\/i0.wp.com\/justinwiegand.com\/blog\/wp-content\/uploads\/2015\/01\/mturk-download.jpg?resize=1024%2C272&amp;ssl=1 1024w, https:\/\/i0.wp.com\/justinwiegand.com\/blog\/wp-content\/uploads\/2015\/01\/mturk-download.jpg?resize=300%2C80&amp;ssl=1 300w, https:\/\/i0.wp.com\/justinwiegand.com\/blog\/wp-content\/uploads\/2015\/01\/mturk-download.jpg?w=1190&amp;ssl=1 1190w\" sizes=\"auto, (max-width: 720px) 100vw, 720px\" \/><\/li>\n<li>This provides you with a CSV file \u00a0you can open in Excel or any type of text editor.<\/li>\n<li>Amending\u00a0your spreadsheet created with Tyler&#8217;s guidance earlier in this step, you can now match up WorkerIDs lacking a legitimate verification code to\u00a0your downloaded CSV file and use the Excel&#8217;s fill function to\u00a0approve all HITs (with an &#8220;x&#8221; in the &#8220;approve&#8221; column). \u00a0Then, hand-select those without legitimate verification codes for rejection and re-upload the file. \u00a0Alternatively, you could create an Excel function\u00a0to do this for you.<\/li>\n<\/ul>\n<\/li>\n<li>That takes care of the externally hosted aspect of your surveys and the use of verification codes to ensure data integrity. \u00a0The next part (2)\u00a0explains how to\u00a0follow-up with workers for subsequent surveys and match data between your survey waves.<\/li>\n<\/ol>\n<p><strong>References<\/strong><\/p>\n<div id=\"zotpress-2c5519928c0295e3ab33edb472facb4a\" class=\"zp-Zotpress zp-Zotpress-Bib wp-block-group\">\n\n\t\t<span class=\"ZP_API_USER_ID ZP_ATTR\">1559346<\/span>\n\t\t<span class=\"ZP_ITEM_KEY ZP_ATTR\">{1559346:ZTWW9MKX},{1559346:G8WWE2R8}<\/span>\n\t\t<span class=\"ZP_COLLECTION_ID ZP_ATTR\"><\/span>\n\t\t<span class=\"ZP_TAG_ID ZP_ATTR\"><\/span>\n\t\t<span class=\"ZP_AUTHOR ZP_ATTR\"><\/span>\n\t\t<span class=\"ZP_YEAR ZP_ATTR\"><\/span>\n        <span class=\"ZP_ITEMTYPE ZP_ATTR\"><\/span>\n\t\t<span class=\"ZP_INCLUSIVE ZP_ATTR\">1<\/span>\n\t\t<span class=\"ZP_STYLE ZP_ATTR\">apa<\/span>\n\t\t<span class=\"ZP_LIMIT ZP_ATTR\">50<\/span>\n\t\t<span class=\"ZP_SORTBY ZP_ATTR\">creator<\/span>\n\t\t<span class=\"ZP_ORDER ZP_ATTR\">asc<\/span>\n\t\t<span class=\"ZP_TITLE ZP_ATTR\"><\/span>\n\t\t<span class=\"ZP_SHOWIMAGE ZP_ATTR\"><\/span>\n\t\t<span class=\"ZP_SHOWTAGS ZP_ATTR\"><\/span>\n\t\t<span class=\"ZP_DOWNLOADABLE ZP_ATTR\"><\/span>\n\t\t<span class=\"ZP_NOTES ZP_ATTR\"><\/span>\n\t\t<span class=\"ZP_ABSTRACT ZP_ATTR\"><\/span>\n\t\t<span class=\"ZP_CITEABLE ZP_ATTR\"><\/span>\n\t\t<span class=\"ZP_TARGET ZP_ATTR\"><\/span>\n\t\t<span class=\"ZP_URLWRAP ZP_ATTR\"><\/span>\n\t\t<span class=\"ZP_FORCENUM ZP_ATTR\"><\/span>\n        <span class=\"ZP_HIGHLIGHT ZP_ATTR\"><\/span>\n        <span class=\"ZP_POSTID ZP_ATTR\">195<\/span>\n\t\t<span class=\"ZOTPRESS_PLUGIN_URL ZP_ATTR\">https:\/\/justinwiegand.com\/blog\/wp-content\/plugins\/zotpress\/<\/span>\n\n\t\t<div class=\"zp-List loading\">\n\t\t\t<div class=\"zp-SEO-Content\">\n\t\t\t\t<span class=\"ZP_CACHETIMER ZP_ATTR\"><\/span><span class=\"ZP_JSON ZP_ATTR\">%7B%22status%22%3A%22success%22%2C%22updateneeded%22%3Afalse%2C%22instance%22%3Afalse%2C%22meta%22%3A%7B%22request_last%22%3A0%2C%22request_next%22%3A0%2C%22used_cache%22%3Atrue%7D%2C%22data%22%3A%5B%7B%22key%22%3A%22ZTWW9MKX%22%2C%22library%22%3A%7B%22id%22%3A1559346%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Landers%20and%20Behrend%22%2C%22parsedDate%22%3A%222014%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BLanders%2C%20R.%20N.%2C%20%26amp%3B%20Behrend%2C%20T.%20S.%20%282014%29.%20An%20inconvenient%20truth%3A%20Arbitrary%20distinctions%20between%20organizational%2C%20Mechanical%20Turk%2C%20and%20other%20convenience%20samples.%20%26lt%3Bi%26gt%3BIndustrial%20and%20Organizational%20Psychology%3A%26%23xA0%3B%20Perspectives%20on%20Science%20and%20Practice%26lt%3B%5C%2Fi%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttp%3A%5C%2F%5C%2Fwww.siop.org%5C%2Fjournal%5C%2FArticle_1.aspx%26%23039%3B%26gt%3Bhttp%3A%5C%2F%5C%2Fwww.siop.org%5C%2Fjournal%5C%2FArticle_1.aspx%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22An%20inconvenient%20truth%3A%20Arbitrary%20distinctions%20between%20organizational%2C%20Mechanical%20Turk%2C%20and%20other%20convenience%20samples%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Richard%20N.%22%2C%22lastName%22%3A%22Landers%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tara%20S.%22%2C%22lastName%22%3A%22Behrend%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222014%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Fwww.siop.org%5C%2Fjournal%5C%2FArticle_1.aspx%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%22D4RP8FVP%22%5D%2C%22dateModified%22%3A%222017-08-22T21%3A25%3A32Z%22%7D%7D%2C%7B%22key%22%3A%22G8WWE2R8%22%2C%22library%22%3A%7B%22id%22%3A1559346%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Podsakoff%20et%20al.%22%2C%22parsedDate%22%3A%222003%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BPodsakoff%2C%20P.%20M.%2C%20MacKenzie%2C%20S.%20B.%2C%20Lee%2C%20J.-Y.%2C%20%26amp%3B%20Podsakoff%2C%20N.%20P.%20%282003%29.%20Common%20method%20biases%20in%20behavioral%20research%3A%20A%20critical%20review%20of%20the%20literature%20and%20recommended%20remedies.%20%26lt%3Bi%26gt%3BJournal%20of%20Applied%20Psychology%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B88%26lt%3B%5C%2Fi%26gt%3B%285%29%2C%20879%26%23x2013%3B903.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1037%5C%2F0021-9010.88.5.879%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1037%5C%2F0021-9010.88.5.879%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Common%20method%20biases%20in%20behavioral%20research%3A%20A%20critical%20review%20of%20the%20literature%20and%20recommended%20remedies.%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Philip%20M.%22%2C%22lastName%22%3A%22Podsakoff%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Scott%20B.%22%2C%22lastName%22%3A%22MacKenzie%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jeong-Yeon%22%2C%22lastName%22%3A%22Lee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nathan%20P.%22%2C%22lastName%22%3A%22Podsakoff%22%7D%5D%2C%22abstractNote%22%3A%22Interest%20in%20the%20problem%20of%20method%20biases%20has%20a%20long%20history%20in%20the%20behavioral%20sciences.%20%20Despite%20this%2C%20a%20comprehensive%20summary%20of%20the%20potential%20sources%20of%20method%20biases%20and%20how%20to%20control%20for%20them%20does%20not%20exist.%20Therefore%2C%20the%20purpose%20of%20this%20article%20is%20to%20examine%20the%20extent%20to%20which%20method%20biases%20influence%20behavioral%20research%20results%2C%20identify%20potential%20sources%20of%20method%20biases%2C%20discuss%20the%20cognitive%20processes%20through%20which%20method%20biases%20influence%20responses%20to%20measures%2C%20evaluate%20the%20many%20different%20procedural%20and%20statistical%20techniques%20that%20can%20be%20used%20to%20control%20method%20biases%2C%20and%20provide%20recommendations%20for%20how%20to%20select%20appropriate%20procedural%20and%20statistical%20remedies%20for%20different%20types%20of%20research%20settings.%22%2C%22date%22%3A%222003%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1037%5C%2F0021-9010.88.5.879%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Fdoi.apa.org%5C%2Fgetdoi.cfm%3Fdoi%3D10.1037%5C%2F0021-9010.88.5.879%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%220021-9010%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%225GBBP77W%22%2C%22E9GMFV3Z%22%2C%22ECV458MC%22%2C%22IBDNFZPX%22%5D%2C%22dateModified%22%3A%222014-10-20T22%3A13%3A41Z%22%7D%7D%5D%7D<\/span>\n\n\t\t\t\t<div id=\"zp-ID-195-1559346-ZTWW9MKX\" data-zp-author-date='Landers-and-Behrend-2014' data-zp-date-author='2014-Landers-and-Behrend' data-zp-date='2014' data-zp-year='2014' data-zp-itemtype='journalArticle' class=\"zp-Entry zpSearchResultsItem\">\n<div class=\"csl-bib-body\" style=\"line-height: 2; padding-left: 1em; text-indent:-1em;\">\n  <div class=\"csl-entry\">Landers, R. N., & Behrend, T. S. (2014). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. <i>Industrial and Organizational Psychology:\u00a0 Perspectives on Science and Practice<\/i>. <a class='zp-ItemURL' href='http:\/\/www.siop.org\/journal\/Article_1.aspx'>http:\/\/www.siop.org\/journal\/Article_1.aspx<\/a><\/div>\n<\/div>\n\t\t\t\t<\/div><!-- .zp-Entry .zpSearchResultsItem -->\t\t\t\t<div id=\"zp-ID-195-1559346-G8WWE2R8\" data-zp-author-date='Podsakoff-et-al.-2003' data-zp-date-author='2003-Podsakoff-et-al.' data-zp-date='2003' data-zp-year='2003' data-zp-itemtype='journalArticle' class=\"zp-Entry zpSearchResultsItem\">\n<div class=\"csl-bib-body\" style=\"line-height: 2; padding-left: 1em; text-indent:-1em;\">\n  <div class=\"csl-entry\">Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. <i>Journal of Applied Psychology<\/i>, <i>88<\/i>(5), 879\u2013903. <a class='zp-DOIURL' href='https:\/\/doi.org\/10.1037\/0021-9010.88.5.879'>https:\/\/doi.org\/10.1037\/0021-9010.88.5.879<\/a><\/div>\n<\/div>\n\t\t\t\t<\/div><!-- .zp-Entry .zpSearchResultsItem -->\n\t\t\t<\/div><!-- .zp-zp-SEO-Content -->\n\t\t<\/div><!-- .zp-List -->\n\t<\/div><!--.zp-Zotpress-->\n\n\n","protected":false},"excerpt":{"rendered":"<p>Amazon\u2019s MTURK is a promising way to collect data in the social sciences . \u00a0The sample (as compared to other convenience samples), timeliness, ease, and cost are all strong positive attributes of the service (Landers &amp; Behrend, 2014). \u00a0However, there is not a convenient method to collect multiple waves of&#46;&#46;&#46;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Guide for using Amazon\u2019s MTURK for multiple waves of data collection http:\/\/wp.me\/p51adQ-39","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[32],"tags":[63,67,62,65,68,64,66],"class_list":["post-195","post","type-post","status-publish","format-standard","hentry","category-guides","tag-amazon","tag-follow-up","tag-mturk","tag-qualtrics","tag-random-number-generator","tag-survey-gizmo","tag-waves"],"jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/p51adQ-39","jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/195","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=195"}],"version-history":[{"count":21,"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/195\/revisions"}],"predecessor-version":[{"id":896,"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/195\/revisions\/896"}],"wp:attachment":[{"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=195"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=195"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=195"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}