· 2012
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
- 3,577citations
- 2012year
Short summary
We introduce Resilient Distributed Datasets (RDDs), a new in-memory data abstraction designed for large-scale cluster computing that enables fault-tolerant, high-performance computations.
AI-generated from the title and abstract; the full text is not read.
Key points
- Introduces Resilient Distributed Datasets (RDDs) as a fault-tolerant in-memory abstraction.
- Addresses performance bottlenecks in iterative algorithms and interactive data mining.
- Achieves fault tolerance through coarse-grained transformations, not fine-grained updates.
- Implemented in the Spark system and evaluated with user applications and benchmarks.
AI-generated from the title and abstract; the full text is not read.
Abstract
We present Resilient Distributed Datasets (RDDs), a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner. RDDs are motivated by two types of applications that current computing frameworks handle inefficiently: iterative algorithms and interactive data mining tools. In both cases, keeping data in memory can improve performance by an order of magnitude. To achieve fault tolerance efficiently, RDDs provide a restricted form of shared memory, based on coarsegrained transformations rather than fine-grained updates to shared state. However, we show that RDDs are expressive enough to capture a wide class of computations, including recent specialized programming models for iterative jobs, such as Pregel, and new applications that these models do not capture. We have implemented RDDs in a system called Spark, which we evaluate through a variety of user applications and benchmarks. 1
The authors' abstract, as published at the source., 2012 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Information Systems
Information SystemsComputer Science