arXiv (Cornell University)· 2019· Preprint
Exploring the Limits of Transfer Learning with a Unified Text-to-Text\n Transformer
- 8,343citations
- 2019year
Short summary
A unified text-to-text framework, combined with a new large dataset ('Colossal Clean Crawled Corpus') and scale, achieved state-of-the-art results across numerous NLP benchmarks.
AI-generated from the title and abstract; the full text is not read.
Key points
- Introduced a unified text-to-text framework for all NLP tasks.
- Systematically studied various transfer learning components (objectives, architectures, data).
- Achieved state-of-the-art results on multiple language understanding benchmarks.
- Released a new dataset ('Colossal Clean Crawled Corpus'), pre-trained models, and code.
AI-generated from the title and abstract; the full text is not read.
Abstract
Transfer learning, where a model is first pre-trained on a data-rich task\nbefore being fine-tuned on a downstream task, has emerged as a powerful\ntechnique in natural language processing (NLP). The effectiveness of transfer\nlearning has given rise to a diversity of approaches, methodology, and\npractice. In this paper, we explore the landscape of transfer learning\ntechniques for NLP by introducing a unified framework that converts all\ntext-based language problems into a text-to-text format. Our systematic study\ncompares pre-training objectives, architectures, unlabeled data sets, transfer\napproaches, and other factors on dozens of language understanding tasks. By\ncombining the insights from our exploration with scale and our new ``Colossal\nClean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks\ncovering summarization, question answering, text classification, and more. To\nfacilitate future work on transfer learning for NLP, we release our data set,\npre-trained models, and code.\n
The authors' abstract, as published at the source. arXiv (Cornell University), 2019 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Artificial Intelligence
Artificial IntelligenceComputer Science