Recurring Turking: Conducting Daily Task Studies on Mechanical Turk

Turner, Henry; Eberz, Simon; Martinovic, Ivan

In this paper, we present our system design for conducting recurring daily studies on Amazon Mechanical Turk. We implement this system to conduct a study into touch dynamics, and present our experiences, challenges and lessons learned from doing so. Study participants installed our application on their Apple iOS phones and completed two tasks daily for 31 days. Each task involves performing a series of scrolling or swiping gestures, from which behavioral information such as movement speed or pressure is extracted. Taking place over a time period of 31 days, our study utilized a self-contained app which workers used to complete daily tasks without requiring extra HITs. This differs somewhat from the typical rapid completion of one-off tasks on Amazon Mechanical Turk. This atypical use of the platform prompted us to study aspects related to long-term user retention over the study period: payment schedule (amount and structure over time), regular notifications, payment satisfaction and overall satisfaction. We also investigate the specific concern of reconciling informed consent with workers' desire to complete tasks quickly. We find that using the Mechanical Turk platform in this way leads to data of comparable quality to that of lab based studies, and that our study design choices show a statistically significant effect in keeping workers engaged.