publications

2022

1. ACL NLPPower

   Why only Micro-F1? Class Weighting of Measures for Relation Classification

   Harbecke, David, Chen, Yuxuan, Hennig, Leonhard, and Alt, Christoph

   In Proceedings of the 1st Workshop on Efficient Benchmarking in NLP 2022

   Abs Website

   Relation classification models are conventionally evaluated using only a single measure, e.g., micro-F1, macro-F1 or AUC. In this work, we analyze weighting schemes, such as micro and macro, for imbalanced datasets. We introduce a framework for weighting schemes, where existing schemes are extremes, and two new intermediate schemes. We show that reporting results of different weighting schemes better highlights strengths and weaknesses of a model.

2. ACL REPL4NLP

   A Comparative Study of Pre-trained Encoders for Low-Resource Named Entity Recognition

   Chen, Yuxuan, Mikkelsen, Jonas, Binder, Arne, Alt, Christoph, and Hennig, Leonhard

   In Proceedings of the 7th Workshop on Representation Learning for NLP 2022

   Abs Website

   Pre-trained language models (PLM) are effective components of few-shot named entity recognition (NER) approaches when augmented with continued pre-training on task-specific out-of-domain data or fine-tuning on in-domain data. However, their performance in low-resource scenarios, where such data is not available, remains an open question. We introduce an encoder evaluation framework, and use it to systematically compare the performance of state-of-the-art pre-trained representations on the task of low-resource NER. We analyze a wide range of encoders pre-trained with different strategies, model architectures, intermediate-task fine-tuning, and contrastive learning. Our experimental results across ten benchmark NER datasets in English and German show that encoder performance varies significantly, suggesting that the choice of encoder for a specific low-resource scenario needs to be carefully evaluated.

2021

1. Doctoral Thesis

   Neural sequential transfer learning for relation extraction

   Alt, Christoph

   2021

   Abs PDF Slides

   Relation extraction (RE) is concerned with developing methods and models that automatically detect and retrieve relational information from unstructured data. It is crucial to information extraction (IE) applications that aim to leverage the vast amount of knowledge contained in unstructured natural language text, for example, in web pages, online news, and social media; and simultaneously require the powerful and clean semantics of structured databases instead of searching, querying, and analyzing unstructured text directly. In practical applications, however, relation extraction is often characterized by limited availability of labeled data, due to the cost of annotation or scarcity of domain-specific resources. In such scenarios it is difficult to create models that perform well on the task. It therefore is desired to develop methods that learn more efficiently from limited labeled data and also exhibit better overall relation extraction performance, especially in domains with complex relational structure. In this thesis, I propose to use transfer learning to address this problem, i.e., to reuse knowledge from related tasks to improve models, in particular, their performance and efficiency to learn from limited labeled data. I show how sequential transfer learning, specifically unsupervised language model pre-training, can improve performance and sample efficiency in supervised and distantly supervised relation extraction. In the light of improved modeling abilities, I observe that better understanding neural network-based relation extraction methods is crucial to gain insights that further improve their performance. I therefore present an approach to uncover the linguistic features of the input that neural RE models encode and use for relation prediction. I further complement this with a semi-automated analysis approach focused on model errors, datasets, and annotations. It effectively highlights controversial examples in the data for manual evaluation and allows to specify error hypotheses that can be verified automatically. Together, the researched approaches allow us to build better performing, more sample efficient relation extraction models, and advance our understanding despite their complexity. Further, it facilitates more comprehensive analyses of model errors and datasets in the future.

2020

1. ACL

   TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task

   Alt, Christoph, Gabryszak, Aleksandra, and Hennig, Leonhard

   In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 2020

   Abs PDF Slides Website

   TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE). But, even with recent advances in unsupervised pre-training and knowledge enhanced neural RE, models still show a high error rate. In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? And how do crowd annotations, dataset, and models contribute to this error rate? To answer these questions, we first validate the most challenging 5K examples in the development and test sets using trained annotators. We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled. On the relabeled test set the average F1 score of a large baseline model set improves from 62.1 to 70.1. After validation, we analyze misclassifications on the challenging instances, categorize them into linguistically motivated error groups, and verify the resulting error hypotheses on three state-of-the-art RE models. We show that two groups of ambiguous relations are responsible for most of the remaining errors and that models may adopt shallow heuristics on the dataset when entities are not masked.

2. ACL

   Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extraction

   Alt, Christoph, Gabryszak, Aleksandra, and Hennig, Leonhard

   In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 2020

   Abs PDF Slides Website

   Despite the recent progress, little is known about the features captured by state-of-the-art neural relation extraction (RE) models. Common methods encode the source sentence, conditioned on the entity mentions, before classifying the relation. However, the complexity of the task makes it difficult to understand how encoder architecture and supporting linguistic knowledge affect the features learned by the encoder. We introduce 14 probing tasks targeting linguistic properties relevant to RE, and we use them to study representations learned by more than 40 different encoder architecture and linguistic feature combinations trained on two datasets, TACRED and SemEval 2010 Task 8. We find that the bias induced by the architecture and the inclusion of linguistic features are clearly expressed in the probing task performance. For example, adding contextualized word representations greatly increases performance on probing tasks with a focus on named entity and part-of-speech information, and yields better results in RE. In contrast, entity masking improves RE, but considerably lowers performance on entity type related probing tasks.

3. ACL SRW

   Considering Likelihood in NLP Classification Explanations with Occlusion and Language Modeling

   Harbecke, David, and Alt, Christoph

   In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop 2020

   Abs PDF Website

   Recently, state-of-the-art NLP models gained an increasing syntactic and semantic understanding of language, and explanation methods are crucial to understand their decisions. Occlusion is a well established method that provides explanations on discrete language data, e.g. by removing a language unit from an input and measuring the impact on a model’s decision. We argue that current occlusion-based methods often produce invalid or syntactically incorrect language data, neglecting the improved abilities of recent NLP models. Furthermore, gradient-based explanation methods disregard the discrete distribution of data in NLP. Thus, we propose OLM: a novel explanation method that combines occlusion and language models to sample valid and syntactically correct replacements with high likelihood, given the context of the original input. We lay out a theoretical foundation that alleviates these weaknesses of other explanation methods in NLP and provide results that underline the importance of considering data likelihood in occlusion-based explanation.

4. SemEval

   Defx at SemEval-2020 Task 6: Joint Extraction of Concepts and Relations for Definition Extraction

   Hübner, Marc, Alt, Christoph, Schwarzenberg, Robert, and Hennig, Leonhard

   In Proceedings of the Fourteenth Workshop on Semantic Evaluation 2020

   Abs PDF Website

   Definition Extraction systems are a valuable knowledge source for both humans and algorithms. In this paper we describe our submissions to the DeftEval shared task (SemEval-2020 Task 6), which is evaluated on an English textbook corpus. We provide a detailed explanation of our system for the joint extraction of definition concepts and the relations among them. Furthermore we provide an ablation study of our model variations and describe the results of an error analysis.

5. ECNLP

   Bootstrapping Named Entity Recognition in E-Commerce with Positive Unlabeled Learning

   Zhang, Hanchu, Hennig, Leonhard, Alt, Christoph, Hu, Changjian, Meng, Yao, and Wang, Chao

   In Proceedings of The 3rd Workshop on e-Commerce and NLP 2020

   Abs PDF

   In this work, we introduce a bootstrapped, iterative NER model that integrates a PU learning algorithm for recognizing named entities in a low-resource setting. Our approach combines dictionary-based labeling with syntactically-informed label expansion to efficiently enrich the seed dictionaries. Experimental results on a dataset of manually annotated e-commerce product descriptions demonstrate the effectiveness of the proposed framework.

2019

1. TextGraphs

   Layerwise Relevance Visualization in Convolutional Text Graph Classifiers

   Schwarzenberg, Robert, Hübner, Marc, Harbecke, David, Alt, Christoph, and Hennig, Leonhard

   In Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13) 2019

   Abs PDF

   Representations in the hidden layers of Deep Neural Networks (DNN) are often hard to interpret since it is difficult to project them into an interpretable domain. Graph Convolutional Networks (GCN) allow this projection, but existing explainability methods do not exploit this fact, i.e. do not focus their explanations on intermediate states. In this work, we present a novel method that traces and visualizes features that contribute to a classification decision in the visible and hidden layers of a GCN. Our method exposes hidden cross-layer dynamics in the input graph structure. We experimentally demonstrate that it yields meaningful layerwise explanations for a GCN sentence classifier.

2. ACL

   Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction

   Alt, Christoph, Hübner, Marc, and Hennig, Leonhard

   In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics 2019

   Abs PDF Poster Website

   Distantly supervised relation extraction is widely used to extract relational facts from text, but suffers from noisy labels. Current relation extraction methods try to alleviate the noise by multi-instance learning and by providing supporting linguistic and contextual information to more efficiently guide the relation classification. While achieving state-of-the-art results, we observed these models to be biased towards recognizing a limited set of relations with high precision, while ignoring those in the long tail. To address this gap, we utilize a pre-trained language model, the OpenAI Generative Pre-trained Transformer (GPT) (Radford et al., 2018). The GPT and similar models have been shown to capture semantic and syntactic features, and also a notable amount of “common-sense” knowledge, which we hypothesize are important features for recognizing a more diverse set of relations. By extending the GPT to the distantly supervised setting, and fine-tuning it on the NYT10 dataset, we show that it predicts a larger set of distinct relation types with high confidence. Manual and automated evaluation of our model shows that it achieves a state-of-the-art AUC score of 0.422 on the NYT10 dataset, and performs especially well at higher recall levels.

3. AKBC

   Improving Relation Extraction by Pre-trained Language Representations

   Alt, Christoph, Hübner, Marc, and Hennig, Leonhard

   In Proceedings of Automated Knowledge Base Construction 2019 2019

   PDF Poster Slides Website

2018

1. SWAT4HCLS

   mEx - An Information Extraction Platform for German Medical Text

   Roller, Roland, Alt, Christoph, Seiffe, Laura, and Wang, He

   In Proceedings of the 11th International Conference on Semantic Web Applications and Tools for Healthcare and Life Sciences (SWAT4HCLS’2018) 2018

   PDF
2. BlackBoxNLP

   Learning Explanations from Language Data

   Harbecke, David, Schwarzenberg, Robert, and Alt, Christoph

   In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP 2018

   Abs PDF Website

   PatternAttribution is a recent method, introduced in the vision domain, that explains classifications of deep neural networks. We demonstrate that it also generates meaningful interpretations in the language domain.