About

Bootleg is a self-supervised named entity disambiguation (NED) system that links mentions in text to entities in a knowledge base. Bootleg is built to improve disambiguation of tail entities using a simple Transformer-based architecture. Bootleg is still in active development, but is already in use at industry and research labs.

Code

Check out our open-source code here.

If you would like to collaborate, please fill out the brief questionnaire so that we can get in touch.

Results

Bootleg achieves state of the art on NED benchmarks and improves over a standard BERT NED baseline by over 50 F1 points on the tail. Bootleg can even correctly disambiguate entities which were never seen during train!

NED Benchmarks

We compare the newest version Bootleg as of Sep 2020 against the current reported SotA numbers on two standard sentence-level benchmarks (KORE50 and RSS500) and the standard document-level benchmark (AIDA CoNLL-YAGO).

Benchmark System Precision Recall F1
KORE501 Hu et al., 20194 80.0 79.8 79.9
Bootleg 89.8 85.4 87.5
RSS5002 Phan et al., 20195 82.3 82.3 82.3
Bootleg 88.5 79.9 84.0
AIDA CoNLL YAGO3 Fevry et al., 20206 - 96.7 -
Bootleg 96.9 96.6 96.8

Tail Performance

We also evaluate the performance of Bootleg on the tail compared to a standard BERT NED baseline. We create evaluation sets from Wikipedia by filtering by the frequency of the true entities in the training dataset and report micro F1 scores. The slight increase in performance over unseen entities compared to tail entities is due to the lower degree of ambiguity among the unseen entities compared to the tail entities.

Evaluation Set BERT NED Baseline Bootleg
All Entities 92.1 96.7
Torso Entities 57.0 82.1
Tail Entities 17.8 77.7
Unseen Entities 23.0 80.1

References

1Johannes Hoffart, Stephan Seufert, Dat Ba Nguyen, Martin Theobald, and Gerhard Weikum. "Kore: keyphrase overlap relatedness for entity disambiguation." In CIKM, 2012.

2 Daniel Gerber, Sebastian Hellmann, Lorenz Bühmann, Tommaso Soru, Ricardo Usbeck, and Axel-Cyrille Ngonga Ngomo."Real-time rdf extraction from un- structured data streams." In ISWC, 2013.

3Johannes Hoffart, Mohamed Amir Yosef,Ilaria Bordino,Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. "Robust disambiguation of named entities in text." In EMNLP, 2011.

4Shengze Hu, Zhen Tan, Weixin Zeng, Bin Ge, and Weidong Xiao. "Entity linking via symmetrical attention-based neural network and entity structural features." Symmetry, 2019.

5 Minh C. Phan, Aixin Sun, Yi Tay, Jialong Han, and Chenliang Li. "Pair-linking for collective entity disambiguation: Two could be better than all". In TKDE, 2019.

6 Thibault Févry, Nicholas FitzGerald, Livio Baldini Soares, and Tom Kwiatkowski. "Empirical evaluation of pretraining strategies for supervised entity linking." In AKBC, 2020.