arXiv (Cornell University)· 2015· Preprint
Tensorizing Neural Networks
- 499citations
- 2015year
Short summary
Researchers have developed a method to convert dense weight matrices in neural networks to the Tensor Train format, significantly reducing the number of parameters while preserving the network's expressive power.
AI-generated from the title and abstract; the full text is not read.
Key points
- Fully-connected layers in neural networks are a major bottleneck for memory and computation.
- Converting dense weight matrices to the Tensor Train format drastically reduces the number of parameters.
- For Very Deep VGG networks, this method achieved compression factors of up to 200,000 for a single fully-connected layer.
- The overall network compression reached up to 7 times without sacrificing expressive power.
AI-generated from the title and abstract; the full text is not read.
Abstract
Deep neural networks currently demonstrate state-of-the-art performance in\nseveral domains. At the same time, models of this class are very demanding in\nterms of computational resources. In particular, a large amount of memory is\nrequired by commonly used fully-connected layers, making it hard to use the\nmodels on low-end devices and stopping the further increase of the model size.\nIn this paper we convert the dense weight matrices of the fully-connected\nlayers to the Tensor Train format such that the number of parameters is reduced\nby a huge factor and at the same time the expressive power of the layer is\npreserved. In particular, for the Very Deep VGG networks we report the\ncompression factor of the dense weight matrix of a fully-connected layer up to\n200000 times leading to the compression factor of the whole network up to 7\ntimes.\n
The authors' abstract, as published at the source. arXiv (Cornell University), 2015 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Computational Mathematics
Computational MathematicsMathematics