Please use this identifier to cite or link to this item: https://hdl.handle.net/10419/319837 
Year of Publication: 
2025
Series/Report no.: 
I4R Discussion Paper Series No. 239
Publisher: 
Institute for Replication (I4R), s.l.
Abstract: 
Webb, Holyoak & Lu (2023) compared human reasoners and GPT-3 on several tasks involving analogy resolution, documenting human-level or superhuman performance in most conditions. In this direct replication, we tested a different, open-weights language model (Mixtral-8x7B) on the same materials (Experiments 1 and 2, "Digit Matrices" and "Letter String" problems) or on an augmented dataset to obtain the desired statistical power (Experiment 4, "Story Analogies"). Our replication confirmed the sign and statistical significance of the reported effects in the Digit Matrices and Story Analogies problems, whereas the model did not surpass the human baseline in the Letter String problems.
Document Type: 
Working Paper

Files in This Item:
File
Size





Items in EconStor are protected by copyright, with all rights reserved, unless otherwise indicated.