文章基本信息

标题：Wie zuverlässig ist das Peer-Review-Verfahren? Eine Untersuchung der Interrater-Reliabilität von Gutachter*innen auf DGPuK-Tagungen
本地全文：下载
作者：Thomas Koch ; Stefan Geiß
期刊名称：Studies in Communication, Media
电子版ISSN：2192-4007
出版年度：2019
卷号：8
期号：2
页码：203-235
DOI：10.5771/2192-4007-2019-2-203
出版社：Nomos Verlagsgesellschaft mbH & Co. KG
摘要：Journal editors and conference organizers frequently rely on peer-reviewing to assess the quality of submissions. Peer-reviewing is a technique in which independent colleagues with expertise in the same area of research rate the submission. The present study investigates the reliability of review ratings by different reviewers. To that end we studied the reviews made for the general conference of the German Communication Association (DGPuK) and the annual conferences of its five largest divisions (Fachgruppen) in the past five conferences. Based on 3537 reviews from 23 conferences, we analyze inter-rater reliability (Krippendorff’s á und Brennan und Prediger’s ê) and ranges, regarding both criteriabased scores (fit with conference theme, innovativeness, relevance, theory, method, clarity of presentation) and overall scores. The study shows that there is substantial disagreement between reviewers. This applies to overall scores as well as criteria-based scores. Calculating mean or sum scores across criteria leads to higher agreement between reviewers. We discuss potential modifications to optimize review procedures.
关键词：Wissenschaftliche Qualitätssicherung; Peer-Review-Verfahren; Gutachter*innen; Übereinstimmung; Reliabilität; Reviewer*innen