· 2024
PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
- 606citations
- 2024year
Short summary
PyTorch 2's torch.compile feature, powered by TorchDynamo and TorchInductor, achieves 2.27x faster inference and 1.41x faster training on GPUs by dynamically compiling Python bytecode into optimized graphs.
AI-generated from the title and abstract; the full text is not read.
Key points
- PyTorch 2's torch.compile uses TorchDynamo to extract PyTorch operations into FX graphs by modifying Python bytecode.
- TorchInductor, the default backend, compiles these graphs into Triton (GPUs) or C++ (CPUs).
- Achieved 2.27x inference and 1.41x training speedup on NVIDIA A100 GPUs across 180+ models.
- Outperforms six other compilers in speedup benchmarks.
AI-generated from the title and abstract; the full text is not read.
Abstract
This paper introduces two extensions to the popular PyTorch machine learning framework, TorchDynamo and TorchInductor, which implement the torch.compile feature released in PyTorch 2. TorchDynamo is a Python-level just-in-time (JIT) compiler that enables graph compilation in PyTorch programs without sacrificing the flexibility of Python. It achieves this by dynamically modifying Python bytecode before execution and extracting sequences of PyTorch operations into an FX graph, which is then JIT compiled using one of many extensible backends. TorchInductor is the default compiler backend for TorchDynamo, which translates PyTorch programs into OpenAI's Triton for GPUs and C++ for CPUs. Results show that TorchDynamo is able to capture graphs more robustly than prior approaches while adding minimal overhead, and TorchInductor is able to provide a 2.27× inference and 1.41× training geometric mean speedup on an NVIDIA A100 GPU across 180+ real-world models, which outperforms six other compilers. These extensions provide a new way to apply optimizations through compilers in eager mode frameworks like PyTorch.
The authors' abstract, as published at the source., 2024 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Hardware and Architecture
Hardware and ArchitectureComputer Science