arXiv (Cornell University)· 2014· Preprint
Learning with Pseudo-Ensembles
- 361citations
- 2014year
Short summary
A new regularization technique formalizes 'pseudo-ensembles'—collections of models derived from a parent model via noise—achieving state-of-the-art results in semi-supervised learning.
AI-generated from the title and abstract; the full text is not read.
Key points
- Formalizes 'pseudo-ensembles' as collections of perturbed child models.
- Introduces a novel regularizer for pseudo-ensembles, robust to noise processes.
- Matches dropout performance in fully-supervised settings.
- Achieves state-of-the-art results in semi-supervised learning.
- Demonstrates significant performance gains on a real-world sentiment analysis benchmark.
AI-generated from the title and abstract; the full text is not read.
Abstract
We formalize the notion of a pseudo-ensemble, a (possibly infinite) collection of child models spawned from a parent model by perturbing it according to some noise process. E.g., dropout (Hinton et. al, 2012) in a deep neural network trains a pseudo-ensemble of child subnetworks generated by randomly masking nodes in the parent network. We present a novel regularizer based on making the behavior of a pseudo-ensemble robust with respect to the noise process generating it. In the fully-supervised setting, our regularizer matches the performance of dropout. But, unlike dropout, our regularizer naturally extends to the semi-supervised setting, where it produces state-of-the-art results. We provide a case study in which we transform the Recursive Neural Tensor Network of (Socher et. al, 2013) into a pseudo-ensemble, which significantly improves its performance on a real-world sentiment analysis benchmark.
The authors' abstract, as published at the source. arXiv (Cornell University), 2014 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Computational Mathematics
Computational MathematicsMathematics