Microtask Crowdsourcing Has Been Applied In Many Fields

Microtask crowdsourcing has been applied in many fields in the past decades, however there are still important challenges not fully addressed, especially in task/workflow de- sign and quality assessment. We take a deeper look at crowdsourcing classification tasks and explore how task and workflow design can impact the quality of the classification result. Our research will use large online knowledge base and citizen science projects as examples and investigate the workflow design considerations and its impact on worker performance as well as overall quality outcome based on statistical, probabilistic, or machine learning models for true label prediction, such that design principles can be recommended and applied in other citizen science†¦show more content†¦While the wisdom of crowd (Howe, 2006) has shown great potentials in these areas, it also exhibits many challenges (Kittur et al., 2013; Buettner and Buettner, 2016) no matter it is via volunteered activities (Lease, 2011 ; Newman et al., 2012) or paid microtask crowdsourcing (Kittur et al., 2013; Demartini, 2015; Bernaschina et al., 2015). In a typical microtask crowdsourcing scenario, from the requester’s point of view, the most important challenges include: 1, task and workflow design to make sure that it is easy to be carried out by non-expert workers, and at the same time risk of spammers or low-quality workers can be mitigated to achieve good quality of result; 2, quality assessment approaches to evaluate the work from crowd workers. Task and workflow design (Little et al., 2010b; Kittur et al., 2011; Demartini et al., 2012) are crucial in ensuring the task to be outsourced to the crowd is properly understood, mitigating the chance of spam and keeping users engaged. They are essential to obtain high quality and quantity of input from crowd workers. Quality assessment techniques can be either on the fly (Ipeirotis et al., 2014) during the task running that can be used to optimize task assignment hence reduce cost, or post aggregation (Whitehill et al., 2009; Ipeirotis et al., 2010; Bachrach et al., 2012; Dif allah et al., 2015) which involves various algorithms and strategies that evaluate howShow MoreRelatedEvaluation And Workflow Design And Quality Assessment1919 Words   |  8 PagesAlthough crowdsourcing has been successfully applied in many fields in the past decades, challenges still exist especially in task/workflow design and quality assessment. We take a deeper look at crowdsourcing classification tasks, and explore how task and workflow design can impact the answer quality. Our research is intended to use large knowledge base and citizen science projects as examples and investigate the workflow design considerations and its impact on worker performance as well as overallRead MoreSamasource Give Work Not Aid8822 Words   |  36 Pagesopportunity in soliciting help from me than in earning their own money,† she commented. According to Janah, these letters were symptomatic of a larger issue: well-meaning outsiders, both individuals and organizations, had created a culture of handouts in many developing countries. â€Å"The sad thing,† she continued, â€Å"is that ‘asking,’ for them, was a rational response to severely limited economic opportunities.† The students she had in her classes weren’t lacking in human capital, but they were lacking in opportunities