Spark caching is cheaper only when a dataset gets reused enough times that the memory or disk it occupies costs less than re-running the lineage that built it. For a single pass over data, letting Spark recompute from source is usually the cheaper path. The right answer depends on reuse count, storage level, and how often the cluster churns underneath the job.
How Spark Decides What to Recompute, and What It Costs to Redo
Spark builds a directed acyclic graph of transformations before it runs anything. Nothing executes until an action, a count(), collect(), write, forces evaluation. That deferral is lazy evaluation, and it is the reason caching exists at all: Spark does not know a DataFrame will be reused until you tell it to persist, and it does not actually materialize that persisted copy until the first action touches it.
Lineage is the record of how a partition was produced, and it is what Spark caching actually bypasses when a dataset is reused: the chain of transformations from the original source read to the current RDD or DataFrame. When Spark needs a partition it does not have, whether because nothing cached it or because a cached block got evicted, it walks that lineage and re-executes the chain. That is recomputation, and it is Spark’s default fault-tolerance mechanism rather than a failure mode. If a cached partition is lost, Spark automatically recomputes it from the transformations that produced it, with no special handling required, per the RDD Programming Guide.
What Happens Between a Transformation and an Action?
A transformation (map, filter, join) only extends the DAG. Nothing reads a byte of data or spends a CPU cycle until an action triggers the plan, which is why calling .cache() on a DataFrame does nothing by itself. It registers the DataFrame for caching; the actual scan, computation and storage happen on the first action that follows. A second .cache() call on an already-registered DataFrame is a no-op, and a .cache() with no subsequent action never materializes anything at all. Teams who suspect caching “isn’t working” have often just never triggered it.
Why Does a Cached DataFrame Get Recomputed Anyway?
A cached DataFrame can still get recomputed even when Spark caching was set up correctly, and Databricks’ own troubleshooting documentation lists four distinct causes for why that happens. First, the cache only materializes on the first action, so an upstream check before that point sees nothing cached yet. Second, the underlying source data changed, which invalidates the cached relation outright. Third, memory pressure evicted the cached blocks: under MEMORY_ONLY, an evicted partition is simply gone and the next read silently recomputes it with no error and no warning. Fourth, a non-idempotent transformation in the pipeline made the logical plan diverge from the one Spark originally cached, so the optimizer no longer recognizes the new query as a match for the cached relation (per Databricks’ recompute troubleshooting guide). Databricks’ broader recommendation for this failure mode, in complex or branching pipelines generally, is to materialize to a Delta table and read it back rather than lean on cache().
Recomputation itself is not free, and this is the cost side of the decision that often gets ignored. Weighing Spark caching against recomputation means accounting for a re-read from object storage, which incurs new GET requests against S3 or ABFS and the scan time that comes with them, not just CPU. A wide transformation that reshuffles data has to rebuild shuffle files, not just re-run a map. None of that is guaranteed expensive, though: if the shuffle files from a prior stage are still sitting on an executor’s local disk, Spark can reuse them and skip re-running the map-side of a stage entirely, so a “recompute” sometimes only re-executes the reduce side. The most expensive case is usually silent eviction under MEMORY_ONLY, where a job that assumed a dataset was cached pays full recompute cost on every subsequent action without anyone noticing until the bill or the runtime says so.
Spark Cache vs Persist: Choosing a Storage Level
cache() is shorthand for persist(StorageLevel.MEMORY_ONLY) on an RDD; on a DataFrame or Dataset, the default level persist() uses with no argument is MEMORY_AND_DISK instead. That distinction between an RDD default and the DataFrame default is one of the most common sources of surprise in a Spark cache vs persist comparison, because switching API styles silently changes the fallback behavior under memory pressure.
What Is the Difference Between cache() and persist()?
persist() accepts an explicit StorageLevel, where cache() does not, and that single argument is most of what separates the two Spark caching entry points in practice. Both only materialize on the first action that follows, and both can be removed with unpersist().
import org.apache.spark.storage.StorageLevel
val hot = spark.read.parquet("s3://bucket/dim_customer")
.filter($"active" === true) // narrow BEFORE caching
.persist(StorageLevel.MEMORY_AND_DISK_SER)
hot.count() // materializes cache (1st action)
hot.join(fact, "customer_id").show() // reuses cache (2nd action)
hot.unpersist(blocking = true) // free memory deterministically As of Spark 3.0, unpersist() and destroy() default to non-blocking across every API, Scala RDD, DataFrame, PySpark, Broadcast.destroy, which closed an inconsistency where Scala’s RDD unpersist() used to block by default while everything else did not, fixed per Spark’s SPARK-26771 ticket. Pass blocking=True if the next step needs the freed memory to actually be free before it runs, rather than scheduled to be freed. On the SQL side, spark.catalog.cacheTable("t") and the CACHE TABLE statement store the table in an in-memory columnar format that scans only the columns a query needs and auto-tunes compression per column, and UNCACHE TABLE removes it cluster-wide, since cached tables are shared across every session on the cluster.
Which of the Spark Storage Levels Should You Choose?
Choosing among Spark storage levels trades memory for CPU, and sometimes both for disk. The table below lists the levels actually documented for RDD persistence.
| Storage level | Where it lives | Memory cost | CPU cost | Typical use |
|---|---|---|---|---|
| MEMORY_ONLY | JVM heap, deserialized objects | Highest | Lowest (no serialize/deserialize) | Small-to-medium RDDs that fit entirely in memory |
| MEMORY_AND_DISK | Heap, then spills to local disk | High, degrades gracefully | Low | DataFrame/Dataset default; dataset may not fully fit |
| MEMORY_ONLY_SER | Heap, serialized byte arrays | Lower than MEMORY_ONLY | Higher (serialize on write, deserialize on read) | JVM object-heavy RDDs where GC pressure matters more than CPU |
| MEMORY_AND_DISK_SER | Heap then disk, serialized | Lower, graceful spill | Higher | Same as above when the dataset doesn’t fully fit |
| DISK_ONLY | Local disk only | None | Low CPU, high I/O wait | Datasets far larger than available memory |
| OFF_HEAP (experimental) | Off-heap memory pool | Avoids JVM heap entirely | Moderate | Reducing GC pause time on large heaps |
| *_2 variants | Any of the above, replicated to two executors | Double the base cost | Same as base | Fast recovery without recompute, at twice the storage spend |
The RDD Programming Guide’s advice is to stay on the default MEMORY_ONLY if an RDD fits, move to MEMORY_ONLY_SER before spilling to disk, and avoid DISK_ONLY unless the transformations that built the dataset are themselves expensive or filter out most of the input. Otherwise, recomputing a partition can be about as fast as reading its spilled copy back off disk, which makes the disk tier pay for itself far less often than intuition suggests. Serializing a cached partition also changes its shape in the JVM: a MEMORY_ONLY_SER partition is a single byte array instead of potentially millions of individual objects, which is the documented reason the serialized levels reduce garbage collection pressure even when they cost more CPU to produce.
Unified Memory Management, Eviction and Spill
Spark’s execution memory (shuffles, joins, sorts, aggregations) and its storage memory (cached blocks) share one unified region of the JVM heap rather than two fixed partitions. spark.memory.fraction defaults to 0.6 of the heap minus 300MiB, and within that region spark.memory.storageFraction defaults to 0.5, marking the share of the unified region that is immune to eviction by execution. Spark’s documentation recommends leaving both at their defaults for most jobs (per the Spark configuration reference).
# memory sizing for a cache-heavy job
spark.memory.fraction=0.6
spark.memory.storageFraction=0.5 The Unified Memory Region: Execution vs Storage
The asymmetry in this model matters more than the default numbers do. Execution memory can evict cached storage blocks down to the storageFraction floor when it needs room for a shuffle or a sort. Storage memory can never evict execution memory in the other direction, “due to complexities in implementation,” in Spark’s own words in the tuning guide. A job with a large cache and a sudden wide join does not get an OutOfMemory error from that eviction; it gets a cache that quietly shrinks and partitions that quietly recompute on their next read, the same silent failure mode the Databricks troubleshooting doc describes for the MEMORY_ONLY eviction case above.
Spill and Garbage Collection Pressure
Eviction under the unified model has a second cost beyond the recompute it triggers: garbage collection. Spark’s tuning guide recommends sizing spark.memory.fraction so the resulting region fits comfortably inside the JVM’s old generation, specifically to avoid full GC pauses. Databricks’ own engineering blog documents a case where this went wrong in practice: an iterative Bagel job (Spark’s early Pregel-style graph-processing API) let cached data accumulate unchecked across iterations, growing from 4.3GB to 202.1GB over three iterations, which drove severe GC pauses; the post does not specify the underlying cluster hardware or Spark version behind that measurement. Removing the unnecessary intermediate-iteration caching shortened total job runtime by 10% to 20% in that same measurement, a case where trimming the cache, rather than growing it, was the cheaper choice.
When Caching Wins and When It Loses Money
Caching only pays for itself when the materialization cost, the CPU to scan and serialize a partition once, is recovered by skipping that scan on later actions. Below a certain reuse count, it never breaks even.
When Does Caching Pay for Itself?
Iterative machine learning is a clear case: an algorithm that scans the same training set across dozens of epochs pays the materialization cost once and recovers it on every subsequent pass. Repeated joins against a slowly changing dimension table behave the same way, as does a BI or interactive workload hitting the same hot dataset from multiple queries in a session. The common thread across all three is more than one action against the same DataFrame; a cache that only ever gets read once never recovers what it cost to build.
When Does Caching Lose Money?
A single-use DataFrame is the simplest way to lose money on a cache: the serialization and memory pressure of materializing it bought nothing, because nothing read it a second time. Caching before a filter() or select() compounds that problem, because it pins the full unfiltered dataset in memory and makes the cached relation opaque to Catalyst, so predicate and column pushdown to the source can no longer happen on the narrowed query. Caching a dataset that does not fit in available memory forces a choice between the silent eviction behavior of MEMORY_ONLY and the disk-spill cost of MEMORY_AND_DISK, neither of which is free. Spinning up a larger instance purely to hold a cache is a cost that shows up on the cloud bill directly, independent of whatever the cache was supposed to save. Teams practicing disciplined DataFrame caching narrow the data first, confirm a second action actually exists, and only then persist.
Checkpointing, Cluster Churn and Verifying the Cache
Checkpointing writes an RDD’s data to reliable storage, HDFS or an equivalent, and truncates its lineage so Spark no longer needs the original transformation chain to recover it. That makes it a different tool from caching: caching speeds up reuse within a job’s current lineage, while Spark checkpointing breaks the lineage so a very deep DAG does not become a liability on failure or in iterative jobs where the plan grows every pass. localCheckpoint() does the lineage truncation without reliable storage, writing to executor local storage instead, which is cheaper but not fault-tolerant against executor loss.
When Is checkpoint() Cheaper Than Caching?
Calling checkpoint() without first caching the RDD used to compute it twice: once for whatever action triggered the checkpoint, and again when Spark wrote the checkpoint data to reliable storage, because checkpointing re-executed the lineage rather than reading an already-materialized copy. Spark’s SPARK-8582 ticket tracked that double-compute and shipped a fix in Spark 3.3.0, so recent versions no longer pay it by default. Calling .cache() or .persist() immediately before .checkpoint() is still a cheap, safe habit on any version: the write reads from the in-memory copy instead of walking the lineage again, which is the same effect the original fix targets. On Spark releases before 3.3, that ordering is what turns checkpointing from a double-compute operation into roughly a single compute plus a storage write.
sc.setCheckpointDir("hdfs:///checkpoints/job123")
val expensive = bigDf.persist(StorageLevel.MEMORY_AND_DISK)
expensive.count() // materialize cache first (SPARK-8582 workaround)
expensive.checkpoint() // now reads from cache, not full recompute
expensive.count() // triggers the actual checkpoint write
// vs. cheap lineage truncation with no reliable storage:
// expensive.localCheckpoint() spark.checkpoint.compress defaults to false; setting it to true compresses checkpoint data with the codec named in spark.io.compression.codec, which cuts the storage cost of the write at some CPU expense. Checkpointing earns its keep on jobs with iterative or recursive lineages that would otherwise grow the DAG every pass: graph algorithms and some streaming state are the usual examples, where recomputing from the true origin eventually costs more than one checkpoint write.
A cached block lives on a specific executor, and anything that removes that executor removes the cache with it. Autoscaling scale-down and spot or preemptible instance reclamation are the two most common ways that happens in production, and both trigger the same default response: Spark recomputes the lost partitions from lineage the next time something reads them. Running a job that fits on one machine in Yeedu mode, where the whole job runs on a single VM, removes the executor fleet that churn takes caches away from.
What Happens to a Spark Cache When a Spot Instance Is Reclaimed?
By default, Spark treats a reclaimed spot node the same as any other lost executor: the cached and shuffle blocks on it are gone, and the next read recomputes them. Graceful decommissioning, controlled by spark.decommission.enabled, changes that by trying to migrate RDD cache blocks and shuffle files off the executor before it disappears, rather than losing them outright. Spark’s SPARK-20624 proposal laid out the design and explicitly targets cloud spot and Kubernetes node-loss scenarios; the work split into separate block-manager and shuffle-service migration paths, which is why the config surface ended up as two independent toggles: spark.storage.decommission.rddBlocks.enabled for cache blocks and spark.storage.decommission.shuffleBlocks.enabled for shuffle files, both already true by default. What actually requires an explicit flip is the pair of parents, spark.decommission.enabled and spark.storage.decommission.enabled, which both default to false.
# Survive spot/preemptible reclaim without a full recompute
spark.decommission.enabled=true
spark.storage.decommission.enabled=true
spark.storage.decommission.rddBlocks.enabled=true
spark.storage.decommission.shuffleBlocks.enabled=true
spark.dynamicAllocation.enabled=true Decommissioning does not close every failure mode, though. Databricks documents a specific stage-abort error on spot clusters: org.apache.spark.SparkException: Job aborted due to stage failure: A shuffle map stage with indeterminate output was failed and retried. However, Spark cannot rollback the ResultStage.... It occurs when a spot node carrying shuffle or cache output is reclaimed mid-stage, the stage is marked for retry, but a downstream ResultStage has already consumed partial, indeterminate output and cannot safely be rolled back, so the whole job aborts instead of just recomputing the lost partitions. Databricks recommends an operational response to this case, shifting the affected stage to on-demand nodes, and its KB entry does not claim that spark.decommission.enabled prevents it (per Databricks’ spot-instance stage-failure guide). spark.cleaner.referenceTracking=true (on by default) is the separate mechanism that eventually garbage-collects unreferenced cached and shuffle data once nothing points to it anymore, independent of any node loss.
How Do You Confirm a DataFrame Is Actually Cached?
The Storage tab in the Spark UI lists every cached RDD or DataFrame with its fraction cached, size in memory and size on disk; a fraction below 100% means part of it has already spilled or been evicted. The second check is in the query plan itself: running EXPLAIN on a query against a cached table should show InMemoryRelation and InMemoryTableScan operators, which confirm Spark is reading the cached columnar data rather than the original source.
CACHE LAZY TABLE customers
OPTIONS ('storageLevel' 'MEMORY_AND_DISK');
-- confirm it's materialized
EXPLAIN SELECT * FROM customers WHERE region = 'EU';
-- look for InMemoryTableScan / InMemoryRelation in the plan
UNCACHE TABLE IF EXISTS customers; The table below collects the symptoms that show up when Spark recomputation or a bad cache decision is quietly costing more than expected, with the config keys involved in each fix.
| Symptom / error string | Likely cause | First fix |
|---|---|---|
| Job runtime creeps up over iterations with no code change | Cached data accumulating across loop iterations, as in Databricks’ documented 4.3GB-to-202.1GB case | Unpersist intermediate-iteration caches explicitly; do not rely on the cleaner to catch up |
| cache() called but Storage tab shows nothing | No action has run yet; caching only materializes on the first action | Trigger a count() or equivalent immediately after .persist() |
| Fraction cached drops below 100% over a run | Execution memory evicted storage blocks under memory pressure | Raise spark.memory.storageFraction above its 0.5 default, or move to MEMORY_AND_DISK_SER to spill instead of silently dropping |
| Same DataFrame recomputes despite an earlier .cache() call | Logical plan diverged from the cached relation, often from a non-idempotent transformation upstream | Materialize to a Delta/Parquet table and read it back instead of chaining cache() |
| SparkException: ... shuffle map stage with indeterminate output was failed and retried ... cannot rollback the ResultStage | Spot node carrying shuffle/cache output reclaimed mid-stage | Move the affected stage to on-demand capacity; decommissioning does not guarantee recovery here |
| GC time dominates the Spark UI’s executor summary | Too much heap pinned by cached objects, especially under MEMORY_ONLY | Switch to a _SER storage level to shrink per-partition object count, or unpersist unused caches |
| checkpoint() call doubles the runtime of a stage | RDD was checkpointed without first being cached, forcing two full computations | .persist() or .cache() immediately before .checkpoint() |
A Worked Cost Model for Cache vs Recompute
No vendor-published formula ties a dollar figure to this trade-off, so the following is an illustrative framework to plug your own numbers into, not a benchmarked result.
Recompute cost scales with reuse count: recompute_cost ≈ compute_hours_per_pass × hourly_rate × reuse_count. Every additional action against an uncached DataFrame pays the full compute_hours_per_pass again. Cache cost is mostly fixed: cache_cost ≈ materialization_time × hourly_rate + memory_or_disk_premium, paid once regardless of how many times the cached copy gets read afterward.
Set illustrative numbers: a pass over a dataset costs 0.2 compute-hours at $4/hour, so one pass is $0.80. Caching it costs roughly the same $0.80 to materialize, plus a memory premium for holding it, say a bigger instance tier that adds $0.50/hour for the job’s duration. Three reuses of the uncached path cost 3 × $0.80 = $2.40. The cached path costs $0.80 to materialize plus whatever the memory premium adds over the time it stays resident, and after that the second and third reads cost close to nothing in compute. The break-even is the reuse count at which compute_hours_per_pass × hourly_rate × (reuse_count - 1) exceeds the memory premium; below that count, recompute wins, above it, caching does. The exact numbers belong to your workload, your instance pricing and your measured pass time, not to a generic formula. On a fixed platform licence such as Yeedu’s, the memory premium is cloud cost alone, with no per-hour platform fee added to it.
How Yeedu Makes Spark Caching Cheaper
Yeedu lowers the cost of Spark caching by changing where cached blocks live and how many jobs get to read them. Yeedu mode runs an entire Spark job on a single VM instead of spreading it across a driver and a fleet of workers. Because nothing is partitioned across machines, every cached block stays on that one VM’s memory and local disk rather than on executors that autoscaling or spot reclamation can take away mid-job: there is one machine to lose a cache from, not a fleet of executors scattered across it, which changes the exposure to the cluster-churn scenario described above. It suits jobs that fit on one VM; jobs that genuinely need a cluster keep one.
Spark jobs submitted through Yeedu’s spark-submit multiplexing run inside one shared, long-lived Spark session rather than each getting its own. A dataset cached by one job in that session is served to every later job from memory, with no re-read from object storage and no I/O wait, so queries that repeatedly hit the same data return faster across jobs, not just within one job’s own lineage.
Yeedu’s Turbo engine layers a C++, vectorized execution runtime on top, with CPU-aware caching placed in processor L2/L3 cache rather than the JVM heap, alongside a cache-optimized columnar runtime and a local Parquet disk cache that serves repeat reads of the same files from disk instead of object storage. According to Yeedu’s performance benchmarks, Turbo delivers 4-10x faster execution and 60-80% lower compute cost, a range the vendor scopes specifically to the 30-40% of workloads that are CPU-bound: joins, aggregations and multi-stage transforms. Yeedu does not publish the underlying hardware, dataset size, or Spark version behind that comparison, so the range is best read as a vendor-reported ceiling for CPU-bound stages rather than a guaranteed result on a specific job. It does not apply to shuffle- or I/O-bound stages, which is the same workload-shape caveat that applies to the storage-level choices discussed earlier in this piece.
Yeedu’s pricing is a fixed annual licence rather than a per-core or per-hour charge, so moving a cache-heavy job to a larger memory-optimized VM to hold more in memory adds cloud infrastructure cost only, with no additional platform fee stacked on top. The practical starting point is job by job: point the jobs that re-read the same cached data most often at Yeedu first, rather than attempting to move an entire estate at once.
Matching the Caching Decision to the Job
The caching-versus-recompute decision is a function of reuse count, storage level and how stable the executor holding the data actually is, not a single rule that applies to every job in a pipeline. A single-use ingestion job should usually run uncached. A dimension table joined forty times a day belongs in memory, deserialized, with a plan to unpersist it explicitly rather than trust the cleaner to catch it eventually. A job running on spot capacity needs decommissioning configured before it needs a bigger cache, because losing the executor loses the cache anyway. None of these are permanent choices: a job’s reuse pattern changes as the pipeline around it changes, and the storage level that was right at launch is worth revisiting once the Storage tab shows how the job actually behaves in production.
