LinkBERT: Improving Language Model Training with Document Link

Language Model Pretraining Language models (LMs), like BERT 1 and the GPT series 2, achieve remarkable performance on many natural language processing (NLP) tasks. They are now the foundation of today’s NLP systems. 3 These models serve important roles in products and tools that we use every day, such as search engines like Google 4 and personal assistants like Alexa 5. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. 2019. ↩ Language Models are Few-Shot Learners. Tom B. Brown, et al. 2020. ↩ On the Opportunities and Risks of Foundation Models. Rishi Bommasani et al. 2021. ↩ Google uses BERT for its search engine: https://blog.google/products/search/search-language-understanding-bert/ ↩ Language Model is All You Need: Natural Language Understanding as Question Answering. Mahdi Namazifar et al. Alexa AI. 2020. ↩