Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets

Abstract The recent COVID-19 crisis highlighted the inadequacy of human forecasting. We aim to leverage human prediction markets with real-time machine weighting of likely higher accuracy trades to improve performance. The crowd sourced Almanis prediction market longitudinal platform (n=1822) and Next Generation Social Science (NGS2) platform (n=103) were utilised. A 43-feature model predicted top quintile relative Brier accuracy scores in two out-of-sample datasets (p both <1×10 −9 ). Trades graded as high machine accuracy quality vs. other trades had a greater AUC temporal gain from before to after trade. Hybrid human-machine forecasts had higher accuracy than human forecasts alone, particularly when the two systems disagreed by 5% or more for binary event prediction: the hybrid system demonstrating substantial AUC gains of 13.2%, p=1.35×10 −14 and 13.8%, p=0.003 in the out-of-sample Almanis B and NGS2 datasets respectively. When discordant, the hybrid model was correct for COVID-19 event occurrence 72.7% of the time vs 27.3% for human-only models, p=0.007. This net classification benefit was replicated in the separate Almanis B dataset, p=2.4×10 −7 . Real-time machine classification followed by weighting human trades according to likely accuracy improves collective forecasting performance. Implementation may allow improved anticipation of and response to emerging risks and improved human collective efforts generally. Significance Statement Human-machine hybrid approaches have been identified as a new frontier for event prediction and decision making in the artificial intelligence and collective human intelligence fields. For the first time, we present the successful development and validation of a human-machine hybrid prediction market approach and demonstrate its superior accuracy when compared to prediction markets based on human forecasting alone. The advantages of this new hybrid system are demonstrated in the context of COVID-19-related event prediction.