Competition results

Choice Prediction Competition for Decisions under Risk and Ambiguity

CPC 2015 - Supported by the Max Wertheimer Minerva Center
for Cognitive Processing and Human Performance

Organized by: Ido Erev, Eyal Ert, and Ori Plonsky

Fifty-three teams of researchers have registered to the competition, of which twenty-five teams (with members from five continents) have submitted a model before the deadline. The submissions can be classified into three main categories. The first class includes models that can be described as variants of BEAST (14 submissions). This class provided the best predictions. Thirteen of the 14 submissions were ranked in places 1-13, with MSDs ranging between 0.0089 and 0.0114. The second class involves models that do not try to approximate the underlying processes, and use machine learning and/or similar statistical approaches (4 submissions). Models of this class provided the best fit of the estimation studies. Yet they failed in predicting the competition data, and were ranked in the last places, with MSDs of 0.0211 or higher. The third class can be classified as "others". It includes models that some of them could be described as generalizations of expected utility theory and prospect theory (6 submissions). The performance of these models on predicting the competition data was mild. The best predictors of this class models received MSD of 0.0124 and 0.0127, and were ranked 14-15th in the competition.

Models ranking

The models are ranked based on their MSD in the competition set (the column marked bold)

Rank 100*MSD
Estimation
100*MSD
Prediction
Description
1 0.5866 0.8893 variant of BEAST
2 0.7000 0.9308 variant of BEAST
3 0.7798 0.9325 variant of BEAST
4 0.6804 0.9516 variant of BEAST
5 1.0350 0.9658 variant of BEAST
6 0.7768 0.9688 variant of BEAST
7 0.6984 0.9781 variant of BEAST
- 0.7080 0.985 BEAST (baseline model)
8 0.7005 0.9965 variant of BEAST
9 0.6700 1.0336 variant of BEAST
10 0.9850 1.0535 variant of BEAST
11 0.6458 1.0631 variant of BEAST
12 1.1366 1.0750 variant of BEAST
13 0.9984 1.1438 variant of BEAST
14 0.7355 1.2376 other
15 0.6097 1.2690 other
16 1.8817 1.7043 Variant of BEAST
17 2.6557 1.7794 other
18 1.0598 1.8342 other
19 0.4081 1.8427 Statistical methods
20 0.9507 1.8433 other
21 0.4859 2.1158 Statistical methods
22 0.0019 2.6010 Statistical methods
23 0.8155 5.0146 Statistical methods
24 0.1996 NA Statistical methods
25 3.6314 NA other

 

Models Description and Code (with author's approval): 

  • CPC2015 winner: Cohen, D., & Cohen, O. (2015).

Cohen's Beast Model  

Description618 KB
Code8 KB