Bootleg is a self-supervised named entity disambiguation (NED) system that links mentions in text to entities in a knowledge base. Bootleg is built to improve disambiguation of tail entities using a simple Transformer-based architecture. Bootleg is still in active development, but is already in use at industry and research labs.
Check out our open-source code here.
If you would like to collaborate, please fill out the brief questionnaire so that we can get in touch.
ResultsBootleg achieves state of the art on NED benchmarks and improves over a standard BERT NED baseline by over 50 F1 points on the tail. Bootleg can even correctly disambiguate entities which were never seen during train!
NED BenchmarksWe compare the newest version Bootleg as of Sep 2020 against the current reported SotA numbers on two standard sentence-level benchmarks (KORE50 and RSS500) and the standard document-level benchmark (AIDA CoNLL-YAGO).
|KORE501||Hu et al., 20194||80.0||79.8||79.9||Bootleg||89.8||85.4||87.5|
|RSS5002||Phan et al., 20195||82.3||82.3||82.3||Bootleg||88.5||79.9||84.0|
|AIDA CoNLL YAGO3||Fevry et al., 20206||-||96.7||-||Bootleg||96.9||96.6||96.8|
Tail PerformanceWe also evaluate the performance of Bootleg on the tail compared to a standard BERT NED baseline. We create evaluation sets from Wikipedia by filtering by the frequency of the true entities in the training dataset and report micro F1 scores. The slight increase in performance over unseen entities compared to tail entities is due to the lower degree of ambiguity among the unseen entities compared to the tail entities.
|Evaluation Set||BERT NED Baseline||Bootleg|
|All Entities||92.1||96.7||Torso Entities||57.0||82.1||Tail Entities||17.8||77.7|
1Johannes Hoffart, Stephan Seufert, Dat Ba Nguyen, Martin Theobald, and Gerhard Weikum. "Kore: keyphrase overlap relatedness for entity disambiguation." In CIKM, 2012.
2 Daniel Gerber, Sebastian Hellmann, Lorenz Bühmann, Tommaso Soru, Ricardo Usbeck, and Axel-Cyrille Ngonga Ngomo."Real-time rdf extraction from un- structured data streams." In ISWC, 2013.
3Johannes Hoffart, Mohamed Amir Yosef,Ilaria Bordino,Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. "Robust disambiguation of named entities in text." In EMNLP, 2011.
4Shengze Hu, Zhen Tan, Weixin Zeng, Bin Ge, and Weidong Xiao. "Entity linking via symmetrical attention-based neural network and entity structural features." Symmetry, 2019.
5 Minh C. Phan, Aixin Sun, Yi Tay, Jialong Han, and Chenliang Li. "Pair-linking for collective entity disambiguation: Two could be better than all". In TKDE, 2019.
6 Thibault Févry, Nicholas FitzGerald, Livio Baldini Soares, and Tom Kwiatkowski. "Empirical evaluation of pretraining strategies for supervised entity linking." In AKBC, 2020.