Faster Auto Loader: How Yeedu Turbo Speeds Up Streaming File Ingestion

Most Auto Loader pipelines don’t slow down because Spark is doing too much work. They slow down because the discovery phase of each micro-batch has to look at far more files than actually arrived since the last run. A bigger cluster rarely helps. Speeding one up usually means matching the discovery mode, the batch size and the schema handling to the shape of the landing zone, and reading the streaming query progress log closely enough to see where each second actually goes.

How the cloudFiles Checkpoint Tracks File Discovery Between Micro-Batches 

Auto Loader is a Spark Structured Streaming source, invoked as spark.readStream.format("cloudFiles"), that incrementally and automatically processes new files landing in cloud storage. It can optionally also ingest files that existed before the stream started, and according to Databricks it scales to millions of files per hour. The cloudFiles.format option tells it what’s inside those files: json, csv, parquet, and so on.

The cloudFiles Source and the Checkpoint Location 

The checkpoint location is where the stream remembers what it has already done. Every stream needs one, set through the standard Structured Streaming checkpointLocation option, and the source uses part of that directory to persist file-level metadata rather than just read offsets. 

That metadata store is a RocksDB key-value store. It tracks which files have already been ingested, so a restart after a driver failure or a job cancellation resumes from where the stream left off, instead of reprocessing files it already wrote to the sink.

Schema Inference, Schema Hints and the Rescued Data Column 

Schema handling in Auto Loader works on a simple principle: infer once, remember forever, and give every ambiguous record somewhere to land instead of failing the stream. The source can infer a schema on first read instead of requiring one upfront, and it stores whatever it infers so later runs don’t repeat the work. To do this, it samples the first 50 GB sampling limit or the first 1,000 files it discovers, whichever threshold it hits first, and writes the inferred schema into a _schemas subdirectory under whatever path you set as cloudFiles.schemaLocation. 

Inference is a best guess, and sometimes it guesses wrong. cloudFiles.schemaHints lets you pin the type of a specific column that inference would otherwise get wrong, without silently casting values that don’t fit. A record that violates the hint isn’t dropped. It’s rescued. 

That’s what the _rescued_data column is for: added automatically, it stores, as a JSON string, any field that failed to parse because of a type mismatch or a schema mismatch, including a casing difference between the file and the inferred schema. Databricks’ own recommended practice is to set a pipeline expectation asserting _rescued_data is NULL under normal conditions, so schema drift shows up as a monitoring alert rather than a silent data-quality problem three dashboards downstream. 

Schema evolution has its own default, and it changes depending on whether you supplied a schema at all. cloudFiles.schemaEvolutionMode defaults to addNewColumns when no schema is given, and to none when one is. The other documented modes are rescue, failOnNewColumns and addNewColumnsWithTypeWidening. 

(spark.readStream.format("cloudFiles") 
  .option("cloudFiles.format", "json") 
  .option("cloudFiles.schemaLocation", "/mnt/schemas/orders") 
  .option("cloudFiles.schemaHints", "event_ts TIMESTAMP, amount DECIMAL(18,2)") 
  .option("cloudFiles.inferColumnTypes", "true") 
  .option("rescuedDataColumn", "_rescued_data") 
  .load("/Volumes/catalog/schema/vol/orders/") 
) 

Auto Loader Directory Listing vs File Notification 

Discovery happens one of two ways, and the choice between them is the single biggest lever on cost for a landing zone that keeps growing. 

Directory listing is the default, and it needs no IAM roles or cloud queue resources to set up. On every micro-batch, Auto Loader re-lists the input directory and compares the result against the RocksDB metadata store to work out which files are new. Databricks Auto Loader performance at this layer comes down almost entirely to how discovery is configured, because the listing step itself is not free. Databricks’ listing implementation is smarter than a naive recursive walk: it works from flattened storage-provider responses rather than listing every subdirectory in parallel, and the docs give the example of a YYYY/MM/DD/HH tree that would otherwise need roughly 8,761 naive listing calls to cover a year, with actual call counts depending on the provider’s page size per call (S3 returns 1,000 results per call, ADLS Gen2 returns 5,000, GCS returns 1,024). It still has to touch the whole tree. Databricks’ own directory-listing documentation states plainly that this mode “continuously lists the entire directory and can significantly increase LIST API costs,” particularly under a continuous trigger that keeps re-listing on a fixed interval. 

File notification mode replaces that re-listing with a cloud event pipeline. Setting cloudFiles.useNotifications to true, alongside cloudFiles.region, makes it provision an SNS topic, an SQS queue and an S3 bucket event notification, all named with a databricks-auto-ingest- prefix, up to a documented limit of 100 concurrent notification pipelines per S3 bucket. Each SQS message is deleted only once the corresponding file has actually been read, so queue depth reflects the real ingestion backlog rather than the total object count sitting in the bucket. 

(spark.readStream.format("cloudFiles") 
  .option("cloudFiles.format", "json") 
  .option("cloudFiles.useNotifications", "true") 
  .option("cloudFiles.region", "us-west-2") 
  .option("cloudFiles.schemaLocation", "/mnt/schemas/events") 
  .load("s3://landing-bucket/events/") 
) 

Common Questions About Discovery, Schema and Notification Cost 

The four questions below come up constantly once a stream has been running long enough for its landing zone to grow, so it’s worth answering them together rather than scattering each one under its own heading. 

How Does Exactly-Once File Ingestion Actually Work? 

It works by writing discovered-file metadata into that checkpoint-backed RocksDB store before marking a file done, per the Databricks Auto Loader documentation. A file that shows up in the store is never re-read on a later micro-batch, even if the directory listing or the notification queue surfaces it again. This is why the checkpoint directory is load-bearing. Delete it, and the stream loses its entire memory of what it has already ingested, not just its read position. 

What Happens When Schema Inference Fails? 

It raises CF_FAILED_TO_INFER_SCHEMA, with the literal error text “Failed to infer schema for format <fileFormatInput> from existing files in input path <path>.” One documented sub-condition, CANNOT_MERGE, fires specifically when the sampled files contain incompatible data types for the same column, and Databricks’ error-reference page for this condition recommends supplying cloudFiles.schemaHints so the mismatched records route to the rescued-data column instead of failing the stream outright. For the generic form of the error, the recommended remedy is to stop relying on inference and supply an explicit schema, or to check that the format and read options actually match the files on disk. 

When Is File Notification Mode Cheaper Than Directory Listing? 

It’s cheaper once the landing zone holds enough total objects that a full listing pass costs more, in LIST requests, than the trickle of new-file events actually arriving. Directory listing cost scales with everything under the path. Notification cost scales with new arrivals. A prefix with a few thousand files behaves similarly either way; a prefix with tens of millions of historical objects does not.

Why Does a Second Micro-Batch Take Longer Than the First? 

Because the directory it lists has grown since the first run, and directory listing mode re-lists the entire tree on every trigger regardless of how many files actually arrived. A community-reported run shows exactly this shape: a stream ingesting roughly 250 GB across about six million files on Azure, read with the text format rather than JSON parsing, finished its first batch in under two hours, then spent roughly 69 minutes just on the discovery phase of the second batch, with measured throughput falling from around 1,300 records per second to under one record per second. Nothing about the data changed; the listing got more expensive because the tree got bigger. Two remedies get discussed for this pattern in practice: enabling the now-deprecated cloudFiles.useIncrementalListing together with cloudFiles.backfillInterval to avoid a full scan on every single trigger, or moving to cloudFiles.useNotifications=true so discovery stops depending on total object count at all. The second option is the more durable one, since it removes the dependency on total object count entirely rather than just slowing its growth.

Date-Partitioned Landing Zones and the Cost of Discovery 

Plenty of upstream systems write under a date-partitioned prefix: YYYY/MM/DD, sometimes with an hour folder underneath. That layout is convenient for humans browsing a bucket and genuinely costly for a stream that lists the whole thing on every trigger. 

The underlying mechanism is one Apache Spark itself had to address at the engine level, not just something Databricks papers over. Spark 3.0.0’s regression made loading 300,000 files from S3 take roughly an hour on the driver before any actual processing began, because DataSource#checkAndGlobPathIfNecessary performed glob matching and existence checks one network round trip at a time, single-threaded, and redundantly re-checked paths that globStatus() had already confirmed existed. Spark 3.1.0 fixed this by skipping existence checks for paths globStatus() already validated and parallelizing the remaining glob and existence operations instead of running them sequentially on the driver. An engine version from before that fix makes large-prefix discovery slow independent of anything the streaming source does on top of it. 

Scoping the read helps, but only if it’s done at the right layer. The generic Spark file-source option pathGlobFilter only includes files whose names match a glob pattern, and per the Spark documentation it explicitly “does not change the behavior of partition discovery”: it filters after the listing, not before it. recursiveFileLookup has a sharper trade-off still: it loads nested directories recursively but disables partition inference entirely, and it throws if you also supply an explicit partition spec.

Why Glob Patterns Don’t Shrink the Initial Scan 

A glob pattern filters results after the scan, not before it. Databricks’ own knowledge base describes exactly this symptom: a stream scoped at the table root with a pattern like /mnt/my_table/{year=2025/month=1/day=2,year=2025/month=1/day=3} stays slow even though only two days of data are wanted, because, per that article, the pattern only filters results after the full scan finishes rather than limiting what gets scanned in the first place. Auto Loader directory listing still walks every partition under the root first.

# Slow: glob at table root forces a full tree scan before filtering 
/mnt/my_table/{year=2025/month=1/day=2,year=2025/month=1/day=3} 
 
# Fast: push the glob down so unwanted branches are skipped during the scan 
/mnt/my_table/year=2025/month=1/{day=2,day=3}

The remedy documented for this specific case is to push the glob as deep into the path as the data layout allows, so unwanted branches are skipped during traversal rather than discarded afterward. It’s a one-line change to the input path, and it’s the first thing worth checking when a stream scoped to “just the last couple of days” is still slow. 

Scoping a path down to “today’s folder” carries its own trade-offs, and they’re easy to miss until something breaks at month-end. A job that hardcodes today’s date has to roll over at midnight, which means a scheduling or timezone bug quietly reads the wrong folder, or no folder, right at the boundary. Late-arriving files that land under yesterday’s prefix after the job has already moved on need a separate backfill pass, scoped explicitly to the older path, rather than relying on the regular run to pick them up. Neither problem shows up in testing; both show up in production a few weeks later.

Where Micro-Batch Time, CPU and Cost Actually Go 

In streaming file ingestion, a single micro-batch spends time in four distinct places: discovering which files are new, reading and parsing their contents, running whatever transformations the query defines, and writing the result to the Delta sink. Figuring out which one is slow is mostly a matter of reading the numbers Structured Streaming already publishes, rather than guessing.

How Do You See Where Time Goes in the Spark UI? 

You look at the Structured Streaming tab, and at the matching entries in the History Server for completed runs. The StreamingQueryProgress object backing that UI exposes numInputRows, inputRowsPerSecond, processedRowsPerSecond and the overall batch duration for every trigger. A batch where inputRowsPerSecond collapses while the file count barely changed is a strong signal that the problem sits in discovery, not in parsing; a batch with healthy discovery numbers and a long wall-clock time points at the transformation chain instead. There isn’t a single current Databricks page that enumerates every per-phase duration field with worked examples, so the practical approach is to pair the StreamingQueryProgress fields with the batch-over-batch pattern described above.

maxFilesPerTrigger, maxBytesPerTrigger and Trigger.AvailableNow 

Two options control how much a single micro-batch takes on, and they trade latency against per-batch overhead in opposite directions. cloudFiles.maxFilesPerTrigger defaults to 1,000 and is a hard per-micro-batch ceiling. cloudFiles.maxBytesPerTrigger has no default and is a soft ceiling: the source won’t split a file mid-stream, so actual bytes consumed can run over the limit by up to one file’s worth. When both are set, the stream stops at whichever limit it hits first. 

For scheduled, batch-style runs rather than always-on streaming, Trigger.AvailableNow is the trigger Databricks recommends. It snapshots every file available at query start, processes that snapshot across one or more micro-batches, and then stops. That’s a materially cheaper discovery pattern than a continuous ProcessingTime trigger, which Databricks’ production configuration guidance calls out as especially expensive under directory listing specifically because it keeps re-listing the whole directory on a timer whether or not anything new has landed.

Why Is JSON Parsing CPU-Bound Rather Than I/O-Bound? 

Because the cost lives in tokenizing and type-coercing every field, not in waiting on a disk or a network socket. Nested objects and arrays multiply that cost: each level of nesting is another pass through the parser before from_json or an inferred schema can hand Spark a flat set of typed columns. An explicit schema skips the inference sampling pass described earlier and avoids the type-widening casts inference sometimes applies, but it doesn’t change how much CPU the parse itself costs once the schema is known. A long chain of transformations stacked on top, several nested from_json calls, multiple withColumn casts, a handful of UDFs, tends to dominate total micro-batch time on workloads with small or moderate file counts, where discovery and write are both already cheap. Spark Structured Streaming JSON parsing on wide, deeply nested payloads is a CPU problem before it’s anything else.

Meeting a Per-File SLA and a Worked Cost-Per-File Formula 

A per-file latency SLA is usually a cluster-sizing and isolation question more than a tuning one. Multiple cloudFiles streams sharing a cluster each keep their own checkpoint and their own RocksDB state, and they compete for the same executor cores. Slow one down and the others inherit some of the delay. 

Parallel Streams and Cluster Sizing 

The standard pattern for an SLA that one stream can’t hit is to split the input across parallel streams, each scoped to its own input path, each with its own checkpoint location, sized so the cluster’s available cores roughly match the arrival rate rather than the historical daily average. Watch two overhead sources as streams scale up. Small files turn into proportionally more LIST and GET calls and task-scheduling overhead per useful byte processed. Checkpoint growth is governed in part by cloudFiles.maxFileAge, which controls how much file-tracking state the stream retains; Databricks documents a 14-day minimum for it and warns that a value set too low can lead to duplicate ingestion or missing files. A checkpoint location should also never sit under a cloud storage lifecycle or TTL policy, because a rule that deletes files under that path corrupts the RocksDB-backed state it’s meant to protect. 

Running several stateful streams on one cluster has a well-documented failure mode worth configuring around before it happens rather than after:

# spark.conf / cluster Spark config - fix for 
# org.rocksdb.RocksDBException: ... Too many open files 
spark.sql.streaming.stateStore.rocksdb.compactOnCommit true 
spark.sql.streaming.stateStore.rocksdb.maxOpenFiles 20000 
spark.databricks.rocksDB.fileManager.compactSmallFilesThreshold 8 

A Worked Cost-Per-File Formula 

A rough per-file cost model is useful for comparing configurations even without a vendor-published formula tying it to actual billing, and no such formula is published for the connector’s DBU consumption. Treat every number below as illustrative, not measured:

cost_per_day ≈ (T_discovery_per_batch + T_process_per_file × files_per_batch) 
               × batches_per_day × compute_rate_per_hour / 3600 
 
# illustrative inputs, not measured: 
# T_discovery_per_batch   = 45s   (directory listing against a large tree) 
# T_process_per_file      = 0.4s  (JSON parse + transform chain) 
# files_per_batch         = 1000  (cloudFiles.maxFilesPerTrigger default) 
# batches_per_day         = 96    (15-minute trigger interval) 
# compute_rate_per_hour   = $2.10 (illustrative cluster rate)

The shape of the formula matters more than any number plugged into it. Discovery time is paid once per batch regardless of file count, while processing time scales with files per batch. A landing zone with a bloated discovery phase and a thin file count per trigger is paying mostly for the first term. A landing zone with cheap discovery and heavy per-file transformation logic is paying mostly for the second. A pipeline that mixes both problems needs both remedies, not just one.

Troubleshooting Table: Symptoms, Causes and Fixes 

The table below maps the error strings and symptoms covered above to their documented cause and the first thing to try.

How Yeedu Turbo Makes Auto Loader Faster 

Streaming pipelines built on the cloudFiles source run on Yeedu using that same source and the same incremental file discovery; Yeedu states that this code moves over without a rewrite. 

Running on Yeedu means running on its Turbo engine, and Yeedu describes three effects. On a date layout such as YYYY/MM/DD, Yeedu’s implementation can scan just the current day’s folder instead of recursively searching the whole tree, so discovery work covers the new day’s files rather than the accumulated history of the landing zone. Yeedu describes this as the larger effect, ahead of Turbo’s acceleration of the JSON transformation step covered next. 

The second effect sits on the CPU side. Turbo accelerates the JSON transformation step itself, and the vendor scopes that acceleration to CPU-bound work generally, joins, aggregations, multi-stage transforms, with JSON parsing falling into that same category rather than into the I/O-bound share of a workload. 

The third effect is implementation tuning for recursive folder depth and file-split distribution, which Yeedu says matters on deep or unevenly distributed landing zones. No speedup or cost figure is published for any of the three, so none is attached here. 

The pricing and adoption shape follows the same pattern as any other job run on Yeedu. A fixed annual licence covers unlimited usage rather than a per-core, per-hour or per-job charge, under transparent annual licensing tiers, so adding parallel streams or a larger VM adds cloud cost only, with no per-DBU fee on top. Because Yeedu runs as an execution layer inside a customer’s own cloud account, adopting it for a stream like this happens job by job, with the rest of the pipeline estate untouched. 

Matching the Fix to the Bottleneck 

None of the remedies above substitute for reading the numbers a running stream already produces. A batch with a shrinking inputRowsPerSecond and a flat file count points at discovery. A batch with healthy discovery and a long wall-clock time per batch points at the transformation chain. A batch that fails outright with a RocksDB or schema-inference error points straight at its error string, almost always with a documented, specific remedy attached. 

Treat the landing zone’s shape, flat versus deeply date-partitioned, high file count versus large average file size, as the input to every decision in this piece. The same maxFilesPerTrigger value that’s generous for one pipeline is a bottleneck for another. For a stream already diagnosed as spending its CPU budget on JSON transformation, or losing time to directory listing on a date-partitioned prefix, the sections above cover the tuning options available inside the connector itself, and Yeedu Turbo is a separate, product-level option addressing those same bottlenecks.