文章基本信息

标题：Validation Methodology for Expert-Annotated Datasets: Event Annotation Case Study
本地全文：下载
作者：Oana Inel ; Lora Aroyo
期刊名称：OASIcs : OpenAccess Series in Informatics
电子版ISSN：2190-6807
出版年度：2019
卷号：70
页码：12:1-12:15
DOI：10.4230/OASIcs.LDK.2019.12
出版社：Schloss Dagstuhl -- Leibniz-Zentrum fuer Informatik
摘要：Event detection is still a difficult task due to the complexity and the ambiguity of such entities. On the one hand, we observe a low inter-annotator agreement among experts when annotating events, disregarding the multitude of existing annotation guidelines and their numerous revisions. On the other hand, event extraction systems have a lower measured performance in terms of F1-score compared to other types of entities such as people or locations. In this paper we study the consistency and completeness of expert-annotated datasets for events and time expressions. We propose a data-agnostic validation methodology of such datasets in terms of consistency and completeness. Furthermore, we combine the power of crowds and machines to correct and extend expert-annotated datasets of events. We show the benefit of using crowd-annotated events to train and evaluate a state-of-the-art event extraction system. Our results show that the crowd-annotated events increase the performance of the system by at least 5.3%.
关键词：Crowdsourcing; Human-in-the-Loop; Event Extraction; Time Extraction