Learning Word Embeddings for Natural Language Processing

April 21, 2015, 4:30 pm





I will describe research on learning word embeddings using neural networks. Recent work has shown that such embeddings can successfully capture semantic and syntactic regularities in language (Mikolov2013) and improve the performance of various Natural Language Processing systems (Collobert2011, Socher2013).

Most methods for learning word embeddings are based on the same premise of "distributional semantics," where words from similar contexts are mapped to nearby vectors. In this talk, I shall argue that "compositional semantics" and "relational semantics" also need to be incorporated into representation learning. First, I will present a neural language model that dynamically re-computes its word representations on-the-fly; this helps account for the compositional semantics phenomena of new meaning formation and meaning shift. Second, I will describe a representation learning algorithm that incorporates relational semantics from WordNet and other knowledge bases. This algorithm, based on the Alternating Direction Method of Multipliers (ADMM), provides a simple yet flexible approach to integrating multiple types of semantic premises into word representation learning.