A Game Theory Approach for Estimating Reliability of Crowdsourced Relevance Assessments

In this article, we propose an approach to improve quality in crowdsourcing (CS) tasks using Task Completion Time (TCT) as a source of information about the reliability of workers in a game-theoretical competitive scenario. Our approach is based on the hypothesis that some workers are more risk-inclined and tend to gamble with their use of time when put to compete with other workers. This hypothesis is supported by our previous simulation study. We test our approach with 35 topics from experiments on the TREC-8 collection being assessed as relevant or non-relevant by crowdsourced workers both in a competitive (referred to as “Game”) and non-competitive (referred to as “Base”) scenario. We find that competition changes the distributions of TCT, making them sensitive to the quality (i.e., wrong or right) and outcome (i.e., relevant or non-relevant) of the assessments. We also test an optimal function of TCT as weights in a weighted majority voting scheme. From probabilistic considerations, we derive a theoretical upper bound for the weighted majority performance of cohorts of 2, 3, 4, and 5 workers, which we use as a criterion to evaluate the performance of our weighting scheme. We find our approach achieves a remarkable performance, significantly closing the gap between the accuracy of the obtained relevance judgements and the upper bound. Since our approach takes advantage of TCT, which is an available quantity in any CS tasks, we believe it is cost-effective and, therefore, can be applied for quality assurance in crowdsourcing for micro-tasks.

An Adaptive Model of Demand Adjustment in Weighted Majority Games

This paper presents a simple adaptive model of demand adjustment in cooperative games and analyzes this model in weighted majority games. In the model, a randomly chosen player sets her demand to the highest possible value subject to the demands of other coalition members being satisfied. This basic process converges to the aspiration set. By introducing some perturbations into the process, we show that the set of separating aspirations, i.e., demand vectors in which no player is indispensable in order for other players to achieve their demands, is the one most resistant to mutations. We then apply the process to weighted majority games. We show that in symmetric majority games and in apex games, the unique separating aspiration is the unique stochastically stable one.

On Weights and Quotas for Weighted Majority Voting Games

In this paper, we analyze the frequency distributions of weights and quotas in weighted majority voting games (WMVG) up to eight players. We also show different procedures that allow us to obtain some minimum or minimum sum representations of WMVG, for any desired number of players, starting from a minimum or minimum sum representation. We also provide closed formulas for the number of WMVG with n players having a minimum representation with quota up to three, and some subclasses of this family of games. Finally, we complement these results with some upper bounds related to weights and quotas.

An analytical and experimental comparison of maximal lottery schemes

Maximal lottery ($$ ML $$ ML ) schemes constitute an interesting class of randomized voting rules that were proposed by Peter Fishburn in 1984 and have been repeatedly recommended for practical use. However, the subtle differences between different $$ ML $$ ML schemes are often overlooked. Two canonical subsets of $$ ML $$ ML schemes are "Image missing" schemes (which only depend on unweighted majority comparisons) and "Image missing" schemes (which only depend on weighted majority comparisons). We prove that "Image missing" schemes are the only homogeneous $$ ML $$ ML schemes that satisfy $$ SD $$ SD -efficiency and $$ SD $$ SD -participation, but are also among the most manipulable $$ ML $$ ML schemes. While all $$ ML $$ ML schemes are manipulable and even violate monotonicity, they are never manipulable when a Condorcet winner exists and satisfy a relative notion of monotonicity. We also evaluate the frequency of manipulable preference profiles and the degree of randomization of $$ ML $$ ML schemes via extensive computer simulations. In summary, $$ ML $$ ML schemes are rarely manipulable and often do not randomize at all, especially for few alternatives. The average degree of randomization of "Image missing" schemes is consistently lower than that of "Image missing" schemes.

An investigation of ensemble combination schemes for genetic programming based hyper-heuristic approaches to dynamic job shop scheduling

Genetic programming based hyper-heuristic (GP-HH) approaches that evolve ensembles of dispatching rules have been effectively applied to dynamic job shop scheduling (JSS) problems. Ensemble GP-HH approaches have been shown to be more robust than existing GP-HH approaches that evolve single dispatching rules for dynamic JSS problems. For ensemble learning in classification, the design of how the members of the ensembles interact with each other, e.g., through various combination schemes, is important for developing effective ensembles for specific problems. In this paper, we investigate and carry out systematic analysis for four popular combination schemes. They are majority voting, which has been applied to dynamic JSS, followed by linear combination, weighted majority voting and weighted linear combination, which have not been applied to dynamic JSS. In addition, we propose several mea-sures for analysing the decision making process in the ensembles evolved by GP. The results show that linear combination is generally better for the dynamic JSS problem than the other combination schemes investigated. In addition, the different combination schemes result in significantly different interactions between the members of the ensembles. Finally, the analysis based on the measures shows that the behaviours of the evolved ensembles are significantly affected by the combination schemes. Weighted majority voting has bias towards single members of the ensembles. © This manuscript version is made available under the CC-BY-NC-ND 4.0 license https://creativecommons.org/licenses/by-nc-nd/4.0/

An investigation of ensemble combination schemes for genetic programming based hyper-heuristic approaches to dynamic job shop scheduling

Genetic programming based hyper-heuristic (GP-HH) approaches that evolve ensembles of dispatching rules have been effectively applied to dynamic job shop scheduling (JSS) problems. Ensemble GP-HH approaches have been shown to be more robust than existing GP-HH approaches that evolve single dispatching rules for dynamic JSS problems. For ensemble learning in classification, the design of how the members of the ensembles interact with each other, e.g., through various combination schemes, is important for developing effective ensembles for specific problems. In this paper, we investigate and carry out systematic analysis for four popular combination schemes. They are majority voting, which has been applied to dynamic JSS, followed by linear combination, weighted majority voting and weighted linear combination, which have not been applied to dynamic JSS. In addition, we propose several mea-sures for analysing the decision making process in the ensembles evolved by GP. The results show that linear combination is generally better for the dynamic JSS problem than the other combination schemes investigated. In addition, the different combination schemes result in significantly different interactions between the members of the ensembles. Finally, the analysis based on the measures shows that the behaviours of the evolved ensembles are significantly affected by the combination schemes. Weighted majority voting has bias towards single members of the ensembles. © This manuscript version is made available under the CC-BY-NC-ND 4.0 license https://creativecommons.org/licenses/by-nc-nd/4.0/

Group decisions based on confidence weighted majority voting

Background It has repeatedly been reported that, when making decisions under uncertainty, groups outperform individuals. Real groups are often replaced by simulated groups: Instead of performing an actual group discussion, individual responses are aggregated by a numerical computation. While studies have typically used unweighted majority voting (MV) for this aggregation, the theoretically optimal method is confidence weighted majority voting (CWMV)—if independent and accurate confidence ratings from the individual group members are available. To determine which simulations (MV vs. CWMV) reflect real group processes better, we applied formal cognitive modeling and compared simulated group responses to real group responses. Results Simulated group decisions based on CWMV matched the accuracy of real group decisions, while simulated group decisions based on MV showed lower accuracy. CWMV predicted the confidence that groups put into their group decisions well. However, real groups treated individual votes to some extent more equally weighted than suggested by CWMV. Additionally, real groups tend to put lower confidence into their decisions compared to CWMV simulations. Conclusion Our results highlight the importance of taking individual confidences into account when simulating group decisions: We found that real groups can aggregate individual confidences in a way that matches statistical aggregations given by CWMV to some extent. This implies that research using simulated group decisions should use CWMV instead of MV as a benchmark to compare real groups to.