Please use this identifier to cite or link to this item: https://hdl.handle.net/10419/236200 
Year of Publication: 
2021
Citation: 
[Journal:] Behavior Research Methods [ISSN:] 1554-351X [Volume:] 53 [Issue:] 3 [Publisher:] Springer [Place:] Berlin, Heidelberg [Year:] 2021 [Pages:] 1202-1217
Publisher: 
Springer, Berlin, Heidelberg
Abstract: 
Educational large-scale studies typically adopt highly standardized settings to collect cognitive data on large samples of respondents. Increasing costs alongside dwindling response rates in these studies necessitate exploring alternative assessment strategies such as unsupervised web-based testing. Before respective assessment modes can be implemented on a broad scale, their impact on cognitive measurements needs to be quantified. Therefore, an experimental study on N = 17,473 university students from the German National Educational Panel Study has been conducted. Respondents were randomly assigned to a supervised paper-based, a supervised computerized, and an unsupervised web-based mode to work on a test of scientific literacy. Mode-specific effects on selection bias, measurement bias, and predictive bias were examined. The results showed a higher response rate in web-based testing as compared to the supervised modes, without introducing a pronounced mode-specific selection bias. Analyses of differential test functioning showed systematically larger test scores in paper-based testing, particularly among low to medium ability respondents. Prediction bias for web-based testing was observed for one out of four criteria on study-related success factors. Overall, the results indicate that unsupervised web-based testing is not strictly equivalent to other assessment modes. However, the respective bias introduced by web-based testing was generally small. Thus, unsupervised web-based assessments seem to be a feasible option in cognitive large-scale studies in higher education.
Subjects: 
Mode effect
Web-based testing
Computerized testing
Measurement invariance
Selection effect
Higher education
Persistent Identifier of the first edition: 
Additional Information: 
Open access funding enabled and organized by Projekt DEAL
Creative Commons License: 
cc-by Logo
Document Type: 
Article
Document Version: 
Published Version

Files in This Item:
File
Size
292.84 kB





Items in EconStor are protected by copyright, with all rights reserved, unless otherwise indicated.