arXiv (Cornell University)· 2022· Preprint
DreamFusion: Text-to-3D using 2D Diffusion
- 468citations
- 2022year
Short summary
DreamFusion generates 3D models from text prompts by using a pre-trained 2D text-to-image diffusion model as a prior, optimizing a Neural Radiance Field (NeRF) without requiring any 3D training data.
AI-generated from the title and abstract; the full text is not read.
Key points
- Generates 3D models from text prompts using only a pre-trained 2D diffusion model.
- Optimizes a Neural Radiance Field (NeRF) via gradient descent using a novel distillation loss.
- Requires no 3D training data or modifications to the existing 2D diffusion model.
- Resulting 3D models can be rendered from any angle and relit with arbitrary illumination.
AI-generated from the title and abstract; the full text is not read.
Abstract
Recent breakthroughs in text-to-image synthesis have been driven by diffusion models trained on billions of image-text pairs. Adapting this approach to 3D synthesis would require large-scale datasets of labeled 3D data and efficient architectures for denoising 3D data, neither of which currently exist. In this work, we circumvent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. Using this loss in a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a Neural Radiance Field, or NeRF) via gradient descent such that its 2D renderings from random angles achieve a low loss. The resulting 3D model of the given text can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment. Our approach requires no 3D training data and no modifications to the image diffusion model, demonstrating the effectiveness of pretrained image diffusion models as priors.
The authors' abstract, as published at the source. arXiv (Cornell University), 2022 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Computer Graphics and Computer-Aided Design
Computer Graphics and Computer-Aided DesignComputer Science