What the Crowd Yields: Considerations when Crowdsourcing

Jeffrey Mark Scagnelli


In this age of emerging technologies and fragmented populations, the ability to obtain cost-effective data from a broad sample is more elusive than ever. Crowdsourcing is a potentially attractive solution to the challenge of recruiting hard to reach participants. Crowdsourcing is defined as the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call (Howe 2006). Previous research has shown that this method can be effective at gathering reliable data, while enjoying the benefits discussed above (Behrend et al. 2011). While the ability to acquire data through open call web sources such as Amazon Mechanical Turk has been demonstrated, the quality of the data is a key concern. Wais et al. 2010 attempted to address this issue with their work on filtering low-quality results to improve quality. We have built on that approach, while also including a training task as (Le et al. 2010) have to accelerate the learning process. Crowdsourcing allows you to quickly reach a wide array of potential respondents and there is a need to ensure that you include these quality controls to reduce data quality issues. In many cases the respondents to these services will be taking part in multiple studies at once, often driven by the advertised incentives. While you cannot ethically deny payment of an incentive when a respondent participates in good faith, it is justifiable to monitor the quality of returns for unusable submissions to protect the integrity of your process. In this research we will share some results of a pilot study conducted by Nielsen between September 7th 2012 and October 17th 2012 within Hyderabad India.


Crowdsourcing; Mobile; Enumeration

Full Text: PDF HTML

About Survey Practice Our Global Partners Disclaimer
The Survey Practice content may not be distributed, used, adapted, reproduced, translated or copied for any commercial purpose in any form without prior permission of the publisher. Any use of this e-journal in whole or in part, must include the customary bibliographic citation and its URL.