Proceedings of the Python in Science Conferences· 2015
Dask: Parallel Computation with Blocked algorithms and Task Scheduling
- 811citations
- 2015year
Short summary
Dask enables parallel and out-of-core computation by coupling blocked algorithms with dynamic, memory-aware task scheduling, creating a NumPy clone that scales to larger datasets.
AI-generated from the title and abstract; the full text is not read.
Key points
- Dask facilitates parallel and out-of-core computation.
- It uses blocked algorithms coupled with dynamic, memory-aware task scheduling.
- This enables a parallel and out-of-core NumPy clone.
- The system scales computation to larger datasets than available hardware memory.
- The principles can be applied to other parallel collections.
AI-generated from the title and abstract; the full text is not read.
Abstract
Dask enables parallel and out-of-core computation. We couple blocked algorithms with dynamic and memory aware task scheduling to achieve a parallel and out-of-core NumPy clone. We show how this extends the effective scale of modern hardware to larger datasets and discuss how these ideas can be more broadly applied to other parallel collections.
The authors' abstract, as published at the source. Proceedings of the Python in Science Conferences, 2015 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Hardware and Architecture
Hardware and ArchitectureComputer Science