{"id":211,"date":"2015-01-05T13:54:59","date_gmt":"2015-01-05T19:54:59","guid":{"rendered":"http:\/\/justinwiegand.com\/blog\/?p=211"},"modified":"2026-03-10T22:31:55","modified_gmt":"2026-03-11T05:31:55","slug":"using-amazons-mturk-for-multiple-waves-of-data-collection-part-2-follow-up-survey-invitations-and-tracking-responses-between-surveys","status":"publish","type":"post","link":"https:\/\/justinwiegand.com\/blog\/?p=211","title":{"rendered":"Using Amazon\u2019s MTURK for Multiple Waves of Data Collection: Part 2 (Qualifications, Follow-up Survey Invitations and Tracking Responses between Surveys)"},"content":{"rendered":"

Now for the good stuff–follow-ups with MTURK!<\/p>\n

After completing your first survey, you have your set of workers to follow-up with for one or more additional waves of data collection. \u00a0Here’s how to make it work:<\/p>\n

    \n
  1. First, you are going to need CRAN-R<\/a> (free, open-source software) and the MTurkR package<\/a> (also free and open-source; created by\u00a0Thomas Leeper<\/a>) to carry out mass mailings to MTURK workers (to send your follow-up) using the methods I recommend here. \u00a0So download both as necessary from the given links. \u00a0The MTurkR package provides access to MTURK’s Requester API tools. \u00a0Feel free to pay for Amazon’s own services, but Thomas’ R package works very well and he is quite helpful and accessible.<\/li>\n
  2. Once you’re all setup with MTurkR and Amazon, you are almost ready to go live with your follow-up HIT(s) and contact workers. \u00a0But first, you need to go into MTURK and setup your follow-up HIT(s) to only allow those who completed your first survey to participate in the follow-up. \u00a0To do this, you need to create a Qualification Type on MTURK. \u00a0Click the “Manage” tab and then “Qualification Types” and “Create New Qualification Type” as in the screenshot below:\"mturk<\/a><\/li>\n
  3. Create a qualification type called “Follow-up Survey(s)” and assign this to all the Workers from your first survey by downloading a Worker CSV file from Amazon and assigning your unique qualification type to all the workers in your first survey. \u00a0This can be done be clicking the “Manage” tab, then “Workers,” then “Download CSV.” \u00a0In the “UPDATE-Follow-up Survey(s)” column enter the number “99” for all workers (that you want to take follow-up surveys).<\/li>\n
  4. Now for each follow-up HIT, click the “Advanced” link on the HIT creation screen and select “Follow-up Survey(s),” “Equal to,” and “99” respectively under “Specify ALL the qualifications Workers must meet to work on your HITs:” \u00a0Also select “Yes” under “Only Workers who qualify to do my HITs can preview my HITs.” \u00a0This only allows those who completed your first survey to see and complete your follow-ups. \u00a0Using the qualification name “Survey Two” you can see my setup in the screenshot below:\"HIT<\/a><\/li>\n
  5. You can now go live with your follow-up HIT and move on to contact the Workers to complete the next survey on MTURK using R and the MTurkR package.<\/li>\n
  6. To prepare for using MTurkR,\u00a0access your initial set of Amazon WorkerIDs by downloading those who completed your first survey. \u00a0Click the “Manage” tab, then “Manage HITs Individually,” “Download Results,” and again “Download Results” as in the screenshot below.\"mturk<\/li>\n
  7. You will need to select the column of WorkerIDs from this\u00a0file and copy the entries in order to paste into the MTurkR program. \u00a0I recommend filtering by workers you approved in the first round so you do not invite problematic workers to participate in subsequent surveys.<\/li>\n
  8. Now it’s time to power up R and load Leeper’s package. \u00a0This will allow you to contact your workers for follow-up. \u00a0I provide some thoughts for what should be included in your communication after the syntax shown in\u00a0this step. \u00a0The R syntax is below with comments in red. \u00a0The actual selections you should enter are provided in blue. \u00a0R output is black. \u00a0Reference Leeper’s MTurkR-specific\u00a0instruction for further guidance if needed.<\/li>\n<\/ol>\n

    Example using MTurkR Version 0.5.52<\/span>
    \nTo get the latest version run the code below:<\/span>
    \nif(!require(“devtools”)){<\/span>
    \ninstall.packages(“devtools”)<\/span>
    \nlibrary(“devtools”)<\/span>}<\/span>
    \ninstall_github(“leeper\/MTurkR”)<\/span><\/p>\n

    Install MTurkR<\/span>
    \nlibrary(“MTurkR”, lib.loc=”~\/R\/win-library\/3.0″)<\/span>
    \n wizard.simple()<\/span>
    \nMTurkR Wizard loading…<\/p>\n

    Retrieve your AWS access keys from https:\/\/aws-portal.amazon.com\/gp\/aws\/securityCredentials<\/span><\/p>\n

    AWS\/MTurk Access Key ID: [enter yours]<\/span>
    \nAWS\/MTurk Secret Access Key: [enter yours]<\/span>
    \nUse Sandbox? (Y\/N): N<\/span>
    \nMTurkR Operations<\/p>\n

    1: Check Account Balance 2: Check Sufficient Funds 3: Create HIT
    \n4: Check HIT Status 5: Get Assignment(s) 6: Extend HIT
    \n7: Expire HIT 8: Approve Assignment(s) 9: Reject Assignment(s)
    \n10: Grant Bonus(es) 11: Contact Worker(s) 12: Block Worker(s)
    \n13: Unblock Worker(s) 14: Manage Qualifications 15: Requester Statistics
    \n16: Worker Statistics 17: Open MTurk RUI Pages 18: Load MTurkR Log File\/Entries
    \n19: Exit<\/p>\n

    Selection: 11<\/span><\/p>\n

    Contact one worker or multiple workers?<\/p>\n

    1: Single Worker
    \n2: Multiple Workers<\/p>\n

    Selection: 2<\/span><\/p>\n

    Email Subject Line: MTURK Survey\u2026[enter your subject]<\/span><\/p>\n

    Email body text: Hello,\u2026 [enter your body text]<\/span>
    \nNOTE: Use \\n to indicate new lines. You cannot enter new lines by pressing \u201center.\u201d Ensure your text is all on one line or you will get errors!<\/span><\/p>\n

    How many workers to notify: 1000<\/span> [or however many you plan to contact]
    \nNOTE: The line directly above may be obsolete in future versions.<\/span><\/p>\n

    Enter each WorkerID on its line
    \nNOTE: You should copy and paste all your Workers directly from the Excel spreadsheet you downloaded in steps 6 and 7 earlier.<\/span><\/p>\n

    1: WorkerID<\/span>
    \n2: WorkerID<\/span>
    \n3: WorkerID<\/span>
    \n[\u2026 for each WorkerID]<\/span><\/p>\n

    Your final output should read [after a bunch of repetitious statements]:
    \nValid
    \n1 TRUE
    \n2 TRUE
    \n3 TRUE
    \n[\u2026 for each WorkerID contacted]<\/span><\/p>\n

    Then it\u2019s just:<\/span>
    \nMTurkR Operations<\/p>\n

    1: Check Account Balance 2: Check Sufficient Funds 3: Create HIT
    \n4: Check HIT Status 5: Get Assignment(s) 6: Extend HIT
    \n7: Expire HIT 8: Approve Assignment(s) 9: Reject Assignment(s)
    \n10: Grant Bonus(es) 11: Contact Worker(s) 12: Block Worker(s)
    \n13: Unblock Worker(s) 14: Manage Qualifications 15: Requester Statistics
    \n16: Worker Statistics 17: Open MTurk RUI Pages 18: Load MTurkR Log File\/Entries
    \n19: Exit<\/p>\n

    Selection: 19<\/span><\/p>\n

      \n
    • I recommend sending out an initial invitation to complete the survey and then two more, one at the midpoint of the survey window and one within the final 14 hours of the survey (we had a one week window of completion). \u00a0You should provide a link for the users to copy and paste into their browser which directly searches for your HIT (I was unable to use a direct link to the HIT because of the way the MTURK site handles links). \u00a0Thus, your HIT should have a unique name that you can search for and find independent of all other HITs. \u00a0Once you’ve performed such a search, copy the address bar in your browser for this link and place that in you email so workers can easily find the follow up survey and complete it. \u00a0I recommend reminding them that they must be logged-in to MTURK with the same WorkerID as the first survey in order to qualify for the HIT.<\/li>\n
    • I recommend emphasizing the need for workers to keep their browser window open while completing the survey in your communication. \u00a0Something like “Open the survey in a new window to ensure the HIT stays open for you and you can\u00a0submit your random number to the HIT on survey completion.\u00a0 To do this, right-click on the survey link and select \u201copen link in a new window.\u201d”<\/li>\n
    • Once you’ve collected all the waves of your surveys, you’ll need to go through and match up the results based on WorkerIDs. \u00a0The WorkerIDs can be associated with you surveys by matching them from the random number generated at the end of surveys and reported to MTURK, or by explicitly collecting WorkerIDs in each of your surveys. \u00a0I use SAS data steps to carry\u00a0out the merging of surveys by WorkerID.<\/li>\n<\/ul>\n

      That should do it! \u00a0Through Part 1 and Part 2 of this series, you have learned how to host surveys independent of MTURK and carry out waves of data collection over time by setting up qualifications for Workers and batch contacting them with\u00a0Amazon’s API and the MTurkR package.<\/p>\n","protected":false},"excerpt":{"rendered":"

      Now for the good stuff–follow-ups with MTURK! After completing your first survey, you have your set of workers to follow-up with for one or more additional waves of data collection. \u00a0Here’s how to make it work: First, you are going to need CRAN-R (free, open-source software) and the MTurkR package...<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Using Amazon\u2019s MTURK for Multiple Waves of Data Collection: Part 2 (Follow-up Survey Invitations and Tracking Responses between Surveys) http:\/\/wp.me\/p51adQ-3p","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[32],"tags":[63,69,70,67,62,66],"class_list":["post-211","post","type-post","status-publish","format-standard","hentry","category-guides","tag-amazon","tag-api","tag-data-collection","tag-follow-up","tag-mturk","tag-waves"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p51adQ-3p","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/211","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=211"}],"version-history":[{"count":18,"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/211\/revisions"}],"predecessor-version":[{"id":895,"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/211\/revisions\/895"}],"wp:attachment":[{"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=211"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=211"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/justinwiegand.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=211"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}