
Apache Spark usage in AI companies concentrates almost entirely in the data stage of the machine learning lifecycle, not in the training loop itself. Deciding where Spark belongs in an AI pipeline comes down to which jobs need distributed, fault-tolerant shuffle and which need tight GPU-to-GPU collective communication, and the available options split along that line: Apache Spark itself, Ray, managed platforms such as Databricks, Amazon EMR and Google’s Managed Service for Apache Spark, and execution layers such as Yeedu that run existing Spark code inside a company’s own cloud account. On Yeedu’s own TPC-DS benchmark run, a synthetic decision-support workload that completed 99 of 99 queries on a configuration the benchmark page presents as production-grade rather than a stripped-down demo, the vendor reports compute cost of $0.52 for 1 TB and $2.33 for 3 TB, a figure worth citing because it carries a stated dataset and configuration, not because it predicts what any particular pipeline will cost.
That split holds across the industry. Wherever apache spark usage in AI companies shows up in engineering blogs and open-source repositories, it sits behind the training set: cleaning a corpus, computing features, generating embeddings, scoring a batch overnight. The training loop, where gradients pass between GPUs on every step, needs coordination a shuffle-and-lineage batch engine was never built to provide.
This guide works through that boundary section by section, with a source attached to each claim.
Where Apache Spark Usage in AI Companies Starts: the Data Stage, Not the Training Loop
Spark’s role in an AI pipeline begins where raw data lands and ends where a training job reads its input files: ingestion, cleaning, joining and feature computation, not numerical optimization. Its execution model recovers lost work by recomputing a partition’s lineage and moves data between stages through disk-backed shuffle, built for a cluster where any machine can fail independently.
This is where apache spark usage in AI companies meets a hard boundary: gradient descent across GPUs needs the opposite property, low-latency, synchronous communication so every worker’s gradients combine before the next step. Ray and PyTorch’s distributed data parallel provide that; a shuffle-oriented batch engine was not built to. Read what an engineering team writes about Spark and the subject is almost always a corpus becoming a feature table, never the forward and backward pass itself.
.png)
Where Apache Spark Usage in AI Companies Starts: the Data Stage, Not the Training Loop
Training Data Curation and Deduplication at Scale
Training data curation turns a web-scale crawl into a corpus a model can train on, and at these volumes it is a shuffle-heavy batch computation before anything else. Pipelines in this style, at what is loosely called Common Crawl scale, run language identification, quality filtering and tokenization as wide transformations over partitions running into the billions of rows.
Near-duplicate detection is usually implemented with Spark’s own MinHash LSH, since exact pairwise comparison does not scale past a small fraction of a web corpus:
from pyspark.ml.feature import MinHashLSH
mh = MinHashLSH(inputCol="features", outputCol="hashes", numHashTables=5)
model = mh.fit(rescaledData)
# Approximate near-duplicate pairs within Jaccard distance 0.81
dupes = model.approxSimilarityJoin(dfA, dfB, 0.81, distCol="JaccardDistance") That call is documented in PySpark’s own MinHashLSH API reference, with a signature that has stayed stable since the 2.x line and remains current through Spark 4.2.0. The approxSimilarityJoin step is itself a bucketed shuffle join, which is why every near-duplicate pipeline of this shape is, underneath the model call, the same kind of wide operation. This is a recurring pattern in apache spark usage in AI companies: it turns dedup into a shuffle problem rather than a GPU problem, and no amount of GPU throughput changes how many bytes must cross the network to compare every candidate pair.
Related Read: Top 10 Innovations in the Apache Spark Ecosystem
Feature Engineering and Feature Stores for AI
Feature engineering for a model under active retraining carries a requirement most reporting pipelines do not: the join has to be point-in-time correct, so a feature only uses data that existed at that example’s timestamp. Getting it wrong leaks future information into training, and the model then looks better offline than it performs once deployed.
Backfilling those features over years of history is routine, which is why feature pipelines stay on a batch engine rather than move to an interactive one: the job has to replay a long partition history deterministically. Teams iterating on feature definitions often compute features on demand with Spark transforms instead of materializing every intermediate dataset, so a changed definition can be evaluated without waiting on a full historical backfill first.
A feature table used for training has to be regenerable, auditable and traceable to the exact code and inputs that produced it, a bar most reporting pipelines never have to clear. Reproducibility of exactly that particular kind is where apache spark usage in AI companies matters most: a deterministic batch job with stable lineage is easier to reason about than a pipeline with hidden mutable state. Apache Spark for AI workloads earns its keep here, in the unglamorous work of making a feature table reconstructable months later.
Batch Inference and Embedding Generation on Spark
Batch inference is one of the clearest everyday uses of Spark for AI data engineering: it usually means orchestrating partitions of data so a GPU-resident model sees a steady stream of batches, not Spark doing the inference math itself. It dispatches batches of rows to a model loaded once per worker process, work suited to Pandas UDFs and mapInPandas rather than row-at-a-time Python UDFs.
def predict(inputs: np.ndarray) -> np.ndarray:
model = load_model_once() # cached per worker
return model.predict(inputs)
embed_udf = predict_batch_udf(
make_predict_fn=lambda: predict,
return_type=ArrayType(FloatType()),
batch_size=1024,
)
df.withColumn("embedding", embed_udf("flattened_features"))predict_batch_udf, introduced in Spark 3.4, caches the loaded model per worker rather than reloading it per batch or row, per the PySpark API reference: the expensive part, loading weights onto a GPU, happens once per worker for the job’s life. This is Spark as coordinator rather than compute engine: the embedding math runs on the GPU, Spark decides which rows go where.
Streaming Feature Pipelines with Structured Streaming
Streaming feature pipelines exist because a model retrained weekly or daily still needs features from events that happened minutes ago, a gap a purely batch pipeline cannot close on its own. Structured Streaming treats a stream as an unbounded table and runs the same DataFrame operations incrementally against new micro-batches, checkpointing so a restart resumes rather than recomputes.
CREATE OR REFRESH STREAMING TABLE cleaned_features
(CONSTRAINT valid_ts EXPECT (event_ts IS NOT NULL) ON VIOLATION DROP ROW)
AS SELECT *
FROM STREAM read_files('/raw/events', format => 'json')
WHERE status = 'valid'; LinkedIn’s engineering organization has written about adopting Apache Beam to close the gap between its batch and real-time feature computation, reporting a drop from a 24 to 48 hour offline delay down to millisecond or second-level updates; Spark continues to run the batch backfill path underneath that architecture, by LinkedIn’s own account. Spark for large-scale AI data pipelines still does the historical backfill while the low-latency path keeps the feature store current between retrains.
.png)
AI Companies Using Apache Spark
AI companies using Apache Spark describe it in the same place in their stack: behind a recommendation, ranking or scoring system, computing the features and historical joins that feed it. Each case below comes from that company’s own engineering publication with the year attached, because the distance between a 2014 cluster and a 2022 one is most of the story.
.png)
AI Companies Using Apache Spark
Related Read: Data Engineering Communities to follow
Mobility: Spark carries most of Uber’s analytics compute
Uber’s engineering blog reported in 2022, in a post on its distributed shuffle as a service, that Spark jobs account for more than 95% of compute resources on its analytics clusters, running on more than 10,000 nodes and processing hundreds of petabytes of data every day. The shuffle load underneath that, roughly 220,000 applications and 8 to 10 petabytes of shuffled data daily, is why Uber built a remote shuffle service to move shuffle writes off its compute machines rather than tune the built-in one further.
Social and advertising: Meta folded hundreds of Hive jobs into one Spark job
Meta’s engineering team wrote in 2016, in a post titled “Apache Spark @Scale: A 60 TB+ production use case”, about moving feature preparation for one of its entity ranking systems off Hive, where the pipeline had decomposed into hundreds of smaller jobs, onto a single Spark job that reads 60 TB of compressed data, performs a 90 TB shuffle and sort, and runs 250,000 tasks. Against a Hive pipeline that took roughly three days, the post reports gains of 4.5 to 6 times CPU, 3 to 4 times resource reservation and about 5 times latency, while stating directly that those figures “aren’t a direct comparison of Spark to Hive at the query or job level”. Reaching that scale needed fixes to Spark itself, filed as SPARK-13793, SPARK-13369 and SPARK-13958 and contributed back.
.png)
Social and advertising: Meta folded hundreds of Hive jobs into one Spark job
E-commerce: eBay clustered sellers with Spark MLlib
eBay’s engineering blog described in 2014, in a post titled “Using Spark to Ignite Data Analytics”, running Spark through Hadoop YARN on clusters approaching the range of 2000 nodes, 100 TB of RAM and 20,000 cores, with MLlib’s KMeans clustering sellers from attribute data held in HDFS. The numbers describe a 2014 stack, but the shape is the one that recurs: the cluster does the preparation and the clustering, and the model call itself is a single line.
Genomics: population-scale queries through Hail
Hail, the open-source genomics library from the Broad Institute, runs its distributed MatrixTable on Spark and describes that structure as supporting scalable queries even on petabyte-size datasets. Its install documentation states Hail should work with any Spark 4.1.x cluster built with Scala 2.13. Hail is the analysis layer in work on UK Biobank, gnomAD, TopMed, FinnGen and Biobank Japan, which puts Spark under statistical genetics rather than under a recommender.
Professional networking: LinkedIn rebuilt shuffle, then job startup
LinkedIn’s engineering team built the Magnet shuffle service, writing in 2020 that shuffle reliability, not raw compute, was the bottleneck once a pipeline grows large, and contributed the project back to Apache Spark. A separate 2023 optimization cut a dependency-upload step in job startup by 99%, bringing median job runtime from 9.4 minutes to 5.5 minutes, a 40% reduction measured on production jobs. Neither account claims Spark is the fastest engine available.
Open Table Formats, Catalogs and Lineage in the AI Stack
Open table formats give an AI pipeline a consistent, versioned view of a dataset a retraining job can pin to, something a plain file listing cannot provide. Parquet is the columnar storage layer; Delta Lake and Iceberg add transactional metadata, schema evolution and time travel; Arrow is the in-memory columnar format engines use to exchange batches without a serialization round trip. Versioning matters more here than in a reporting pipeline, because a retrained model is only as reproducible as the exact snapshot it trained on, and catalogs such as Hive Metastore, AWS Glue or Unity Catalog keep that snapshot addressable months afterward.
GPU Acceleration: the RAPIDS Accelerator for Apache Spark
The RAPIDS Accelerator lets Spark push suitable SQL and DataFrame operations onto a GPU through a query-planner plugin, without rewriting the job’s code. NVIDIA’s best-practices documentation names the ideal candidates plainly.
Operations it accelerates
Large joins, hash aggregates, window functions and sort-heavy stages are what NVIDIA documents as benefiting, since they parallelize well across a GPU’s cores.
Operations that fall back to CPU
Operations without GPU support in a given spark-rapids release fall back to CPU silently; NVIDIA directs users to check Spark’s logs for “not-supported” messages.

Operations that fall back to CPU
spark.rapids.sql.concurrentGpuTasks=1
spark.rapids.memory.gpu.pool=ASYNC
spark.sql.shuffle.partitions=400
spark.executor.resource.gpu.amount=1
spark.task.resource.gpu.amount=0.125 These settings, from NVIDIA’s spark-rapids 24.02 tuning guide, lower concurrent GPU tasks to fight out-of-memory errors, set the CUDA async memory pool, and double shuffle partitions to ease memory pressure during large joins. The accelerator is documented as supported on Dataproc, Databricks for AWS and Azure, and Amazon EMR. Spark has also treated GPUs as a schedulable resource since the 3.x line: spark.executor.resource.gpu.amount=1 claims one GPU per executor, and the fractional spark.task.resource.gpu.amount=0.125 shown above packs eight tasks onto it.
Spark Versus Ray, Daft and Dask for AI Data Pipelines
No single engine covers every shape an AI data pipeline takes; the comparison depends on data scale, failure tolerance and how interactive the work needs to be. Spark’s shuffle-and-lineage model suits petabyte-scale, fault-tolerant batch jobs; Ray’s actor and task model suits tighter coordination; single-node engines suit anything that fits in one machine’s memory.
| Engine | Workload shape it fits | Data scale | Failure and recovery model | Documented fit |
|---|---|---|---|---|
| Apache Spark | Shuffle-heavy batch: joins, aggregations, corpus curation | Multi-terabyte to petabyte, cluster-distributed | Lineage-based recomputation; automatic stage retry on executor loss | Databricks, EMR, GCP Managed Spark |
| Ray | Actor and task graphs: training orchestration, online serving | Scales with cluster, including GPU-dense | Lineage reconstruction per object by re-executing the task; actor restarts off by default via max_restarts | Anyscale’s hosted and BYOC platform |
| Daft | DataFrame operations with native multimodal and embedding operators | Single machine to distributed cluster | Not the subject of this comparison’s sourced claims | Connects to S3, GCS, Iceberg, Delta, Hugging Face per its own docs |
| Dask | Small-to-medium task graphs, centrally scheduled dependencies | Efficient where per-task overhead, about 1ms per Dask’s docs, stays below JVM startup cost | Central scheduler tracks a dynamic DAG, reschedules failed tasks | Python-native workloads that skip the JVM |
Ray’s documentation is explicit this is a different failure model: it assumes tasks are deterministic and idempotent, and actor results are not reconstructable unless max_task_retries is changed from its default of 0, with max_task_retries=-1 retrying indefinitely. Spark’s own design sets its limits: a dataset small enough for one machine’s RAM gains nothing from a JVM cluster, and a GPU-resident pipeline belongs on a training framework.
Apache Spark Usage in AI Companies and Where an Execution Layer Like Yeedu Fits
AI data pipelines are some of the most repeated Spark workloads a company runs, because the same curation and feature jobs re-execute on every retrain rather than once. That repetition is why execution efficiency compounds with retraining cadence in a way it does not for a quarterly report.
Yeedu is positioned as exactly that: a re-engineered Spark execution layer that the vendor states runs existing PySpark jobs with no code changes, inside a company’s own AWS, Azure, GCP, Oracle Cloud or on-premises environment, rather than a platform to migrate onto. Its Turbo engine, a vectorized, SIMD-accelerated runtime, is reported by the vendor at 4 to 10 times faster on the CPU-bound share of a workload, joins, aggregations and multi-stage transforms, which the vendor puts at 30 to 40% of a typical mix. Jobs dominated by ingestion, ELT or streaming are not excluded from the gains. They pick them up through Smart Scheduling, reported at 2 to 4 times higher cluster efficiency from packing tasks into idle CPU windows during read and write waits, plus multi-threaded execution across every core, spark-submit multiplexing that lets many jobs share one long-lived session and its cache, and Parquet disk caching that serves repeat reads locally. The vendor publishes no speedup figure for those last three.
Pricing is the other structural difference worth naming: Yeedu licenses at a fixed annual rate for unlimited usage, where Databricks’ DBU model and EMR’s consumption-based pricing, billed per vCPU-hour and per GB-hour for Serverless and on EKS (which adds a per-cluster hourly fee), or as an uplift on raw EC2 and EBS costs for EMR on EC2, both bill by usage. Which is cheaper depends on a team’s usage curve, not on either model being better in the abstract.
| Team situation | Reasonable next step |
|---|---|
| CPU-bound ETL or feature jobs dominate the Spark bill | Point those jobs at an execution layer such as Yeedu’s Turbo engine, leaving the rest of the estate where it runs |
| Ingestion, ELT or streaming jobs mostly wait on I/O | Look at task-packing or scheduling optimizations first |
| Bursty batch with no existing Spark footprint | EMR Serverless, billed per vCPU-hour and GB-hour, or Google’s Managed Service for Apache Spark, billed per job execution in serverless mode |
| Training loop or anything needing GPU collective communication | Ray or a training framework, not a Spark execution layer |
The catalog, the orchestration DAGs and the jobs that are not moved stay as they are: the unit of adoption is a job pointed at a different execution layer, not a platform swap.
Conclusion
Apache Spark’s place in an AI company’s stack is specific: the data stage, where shuffle-heavy, fault-tolerant batch computation turns raw corpora into curated, versioned datasets a model can train on repeatedly. It stops being the right tool once the problem becomes tight GPU coordination, or data small enough for one machine.
- Spark’s shuffle-and-lineage model fits distributed batch jobs, not the collective communication gradient descent needs.
- Training data curation, MinHash LSH dedup and multi-year feature backfills stay on a batch engine.
- Batch inference on Spark usually means orchestrating GPU-resident models through Pandas UDFs, not performing the math itself.
- The RAPIDS Accelerator speeds up computation-heavy joins and sorts, and falls back to CPU silently for anything unsupported.
- Ray, Daft and Dask fit shapes Spark does not: tighter task coordination, multimodal operators, lightweight task graphs.
- Execution layers such as Yeedu let a team point specific expensive jobs at faster execution without migrating the pipeline around them.
- Uber, Meta, eBay, LinkedIn and the Broad Institute’s Hail all put Spark at the data stage, in mobility, advertising, e-commerce and genomics respectively.
- The right standard is a set of criteria, not a vendor: data scale, failure tolerance, interactivity and retraining cadence point toward different tools.
FAQs
Do AI companies still use Apache Spark?
Yes. Companies running large-scale recommendation and ranking systems, including Uber, Meta and LinkedIn, describe Spark in their own engineering writing as the engine behind feature computation and corpus curation, not the training loop.
Where does Yeedu fit into an Apache Spark AI data pipeline?
Yeedu sits underneath an existing PySpark job as an execution layer, inside a company’s own cloud account, rather than replacing Spark, Ray or a training framework. Its Turbo engine is reported by the vendor at 4 to 10 times faster on CPU-bound work such as joins, and it is licensed at a fixed annual rate rather than billed per core or hour.
What is Apache Spark used for in AI workloads?
Spark handles the data stage: cleaning corpora, deduplicating documents, computing point-in-time correct features, generating embeddings through batched inference, and running the backfills a feature store depends on, not the training step itself.
Is Apache Spark used for training machine learning models?
Not for the gradient-descent loop itself. Training needs synchronous communication between GPUs on every step, which a shuffle-oriented batch engine does not provide. Spark’s role resumes once a model needs batch scoring.
Can Apache Spark run on GPUs for AI workloads?
Yes, two ways. Spark has treated GPUs as a schedulable resource since the 3.x line via spark.executor.resource.gpu.amount, and the RAPIDS Accelerator plugin pushes computation-heavy joins onto a GPU, falling back to CPU for anything unsupported.
How do AI companies deduplicate training data with Spark?
Most commonly through MinHash LSH and an approximate similarity join, estimating Jaccard distance without comparing every pair directly, implemented as a bucketed shuffle operation rather than a GPU one.
What does a Spark execution layer change for an AI data pipeline?
It changes how a job runs, not what the pipeline’s code does or outputs. An execution layer such as Yeedu replaces the runtime underneath existing PySpark code with a vectorized engine, speeding up CPU-bound stages, which the vendor states needs no pipeline rewrite.



