Abstract:
Machine learning (ML) advice complements human expertise by offering distinct strengths in judgment and prediction, thereby improving decision outcomes. While previous research has examined individual responses to human or ML advisors, the dynamics of human–ML ensemble advice remain unexplored. This study investigates how decision-makers integrate knowledge claims when receiving advice from humans and/or ML advisors, and how emerging interactions affect decision quality at individual and group levels within an organizational context. Based on a field experiment in Porsche AG’s after-sales department, we identified distinct activities of engaged and unengaged advice-taking that facilitate knowledge transfer and shape decision quality. Our findings reveal that while individuals integrate their own knowledge more often in single-advisor settings (with only human or ML advice), human–ML ensemble advisory settings yield better decision outcomes when decision-makers actively reconcile diverse inputs. Conversely, in single-advisor settings, aligning with the advisor improves decision quality. Notably, engagement dropped from individual to group settings. Participants who integrate all decision-makers and advice input also tend to perform better. These results underscore the importance of the advisory context and offer design implications for collaboration in complex organizational decision-making, highlighting the role of ML-supported platforms in shaping the future of digital knowledge work.
