arXiv (Cornell University)· 2015· Preprint
Escaping From Saddle Points --- Online Stochastic Gradient for Tensor\n Decomposition
- 611citations
- 2015year
Short summary
This paper introduces a novel stochastic gradient descent method that guarantees convergence to a local minimum for non-convex functions, a significant improvement over existing methods prone to saddle points.
AI-generated from the title and abstract; the full text is not read.
Key points
- Identified a 'strict saddle property' for efficient optimization of non-convex functions.
- Stochastic gradient descent converges to a local minimum in polynomial iterations.
- First global convergence guarantee for SGD on non-convex functions with many saddle points.
- Developed the first online algorithm for orthogonal tensor decomposition with global convergence.
AI-generated from the title and abstract; the full text is not read.
Abstract
We analyze stochastic gradient descent for optimizing non-convex functions.\nIn many cases for non-convex functions the goal is to find a reasonable local\nminimum, and the main concern is that gradient updates are trapped in saddle\npoints. In this paper we identify strict saddle property for non-convex problem\nthat allows for efficient optimization. Using this property we show that\nstochastic gradient descent converges to a local minimum in a polynomial number\nof iterations. To the best of our knowledge this is the first work that gives\nglobal convergence guarantees for stochastic gradient descent on non-convex\nfunctions with exponentially many local minima and saddle points. Our analysis\ncan be applied to orthogonal tensor decomposition, which is widely used in\nlearning a rich class of latent variable models. We propose a new optimization\nformulation for the tensor decomposition problem that has strict saddle\nproperty. As a result we get the first online algorithm for orthogonal tensor\ndecomposition with global convergence guarantee.\n
The authors' abstract, as published at the source. arXiv (Cornell University), 2015 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Computational Mathematics
Computational MathematicsMathematics