arXiv (Cornell University)· 2013· Preprint
Distributed Representations of Words and Phrases and their Compositionality
- 18,083citations
- 2013year
Short summary
This paper introduces extensions to the Skip-gram model for learning word vector representations, significantly improving training speed and vector quality through subsampling frequent words and a new negative sampling method.
AI-generated from the title and abstract; the full text is not read.
Key points
- Introduced subsampling of frequent words for faster training and more regular word representations.
- Presented negative sampling as an efficient alternative to hierarchical softmax.
- Developed a method to learn vector representations for phrases, overcoming word order limitations.
AI-generated from the title and abstract; the full text is not read.
Abstract
The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.
The authors' abstract, as published at the source. arXiv (Cornell University), 2013 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Artificial Intelligence
Artificial IntelligenceComputer Science