Vectorized Execution in Apache Spark: What It Is and Why It Matters

Vectorized execution in Apache Spark means processing data in batches of column values instead of one row at a time, and it matters because it lets the CPU use its cache, its SIMD instruction set and its branch predictor the way they were designed to be used, which on CPU-bound stages can cut per-row overhead by an order of magnitude. Whether you ever touch a vectorization flag depends on what your jobs actually spend their time doing, and the honest answer for most Spark estates is that some jobs benefit a lot and most barely notice. 

This guide covers where stock Spark already vectorizes, where it falls back to the slow path, how to tell the difference in an explain plan, and the native engines (Photon, Gluten, Comet and RAPIDS) that push the idea further. Most of it needs no platform change: a few configuration flags, a careful read of explain('formatted'), and a judgment about which jobs are CPU-bound enough to be worth the effort.

What Is Vectorized Execution in Apache Spark? 

Vectorized execution in Apache Spark is the practice of having each operator process a batch of column values, typically a few thousand at a time, instead of pulling one row through the whole operator tree before starting the next. The alternative is the Volcano iterator model: every operator implements a next() call that returns a single tuple to its parent, and the parent calls next() again for the row after that. Spark 1.x ran this way, and Databricks’ 2016 engineering writeup on whole-stage code generation describes why it was slow: each next() call is a virtual function dispatch, intermediate results live in heap-allocated objects rather than CPU registers, and the JVM’s JIT compiler loses the chance to unroll loops or emit SIMD instructions because control flow jumps through an interface on every row.

Row-at-a-time versus vectorized execution

 The foundational case for batching over row-at-a-time processing predates Spark. The MonetDB/X100 paper presented at CIDR 2005 described an engine that processes “vectors” of roughly 1,000 cache-resident column values per operator call, keeping execution inside CPU cache rather than spilling intermediates to main memory, with decompression happening right at the RAM-to-cache boundary. Because each primitive operates on a tight array rather than being re-invoked per tuple, the compiler can emit genuinely data-parallel, SIMD-friendly machine code. On a 100GB TPC-H run, the paper reported raw execution throughput one to two orders of magnitude above DB2 UDB 8.1, with the caveat that its own numbers ran in memory while DB2’s included I/O. Every vectorized reader, Photon operator and native Spark accelerator inherits that idea. 

Three things change when vectorized execution in Apache Spark replaces row-at-a-time processing. CPU cache locality improves, because a batch of same-typed values sits contiguously in memory instead of being scattered across row objects. SIMD becomes usable, because the CPU can apply one instruction to several values in a register at once instead of one value per instruction. Branch prediction improves too, since a loop over a uniform batch has a stable, predictable control flow compared to an interpreter dispatching through a different virtual call on every tuple. These gains only show up once the CPU is the bottleneck, so an I/O-bound scan waiting on object storage sees little of them.

How Stock Apache Spark Executes Queries 

Vectorized execution in Apache Spark happens inside the Tungsten engine, once Catalyst has turned a DataFrame or SQL query into a physical plan; the two optimizers do different jobs, and the row-versus-batch distinction only plays out in the second one. Catalyst handles logical and physical planning: predicate pushdown, join reordering, constant folding. Tungsten handles the runtime mechanics: binary in-memory row format, manual memory management off the JVM heap, and, since Spark 2.0, whole-stage code generation. 

Whole-stage codegen is not the same thing as vectorization. Codegen collapses an entire stage (filter, project and the arithmetic between them) into one generated Java method, so the JIT compiler can optimize it as a unit. Databricks’ 2016 post measured a filter dropping from 15ns per row under Spark 1.6 to 1.1ns with Spark 2.0 codegen, a hash join from 115ns to 4.0ns, and a billion-row join finishing in under a second on a laptop. Codegen gets you out of the virtual-call trap. Vectorization additionally gets you SIMD and cache-friendly batch layout, and Spark’s engine does both, at different points in the pipeline. 

The columnar side of vectorized execution in Apache Spark is built on two classes inside Tungsten: ColumnarBatch, which holds a batch of rows laid out column-by-column, and ColumnVector, the per-column storage inside it, backed by on-heap or off-heap arrays. Historically these classes only existed inside Spark’s own file-format readers. SPARK-27396, resolved for Spark 3.0.0, opened that machinery up: it shipped executeColumnar()/doExecuteColumnar() on SparkPlan and a ColumnarRule extension point reachable through SparkSessionExtensions, so a plugin can swap in columnar operators while Spark inserts RowToColumnarExec/ColumnarToRowExec nodes at the boundary. Gluten, Comet and the RAPIDS Accelerator build on this API. 

You can spot whole-stage codegen directly in an explain plan: look for a WholeStageCodegen (id) wrapper around a group of operators, where the id groups everything the JIT compiled into one function. A ColumnarToRow node breaking that chain tells you the data arrived as column batches and had to be converted to rows before the next operator could consume it, which is the hinge point the next section covers.

Where a vectorized scan stays columnar or falls back to rows 

Where Spark Already Vectorizes Data Access 

Spark ships several vectorized code paths that are on by default, mostly where data enters the engine from a file format or crosses the JVM-to-Python boundary, and together they are a large part of what people mean by vectorized processing in Spark today. Each has its own toggle, batch size and set of exceptions.

1. Vectorized Parquet reader 

spark.sql.parquet.enableVectorizedReader has defaulted to true since Spark 2.0.0, per the Spark SQL Parquet documentation, decoding Parquet pages directly into ColumnarBatch objects rather than one row object per record. It engages directly on atomic-type columns; struct, array and map columns need the separate nested-reader flag covered below, which has defaulted to on since Spark 3.4.0. spark.sql.parquet.columnarReaderBatchSize, default 4096 since Spark 2.4.0, sets rows per batch; the docs warn to size it carefully since the batch is bounded by row count, not bytes. spark.sql.parquet.recordLevelFilter.enabled only takes effect when the vectorized reader is off and spark.sql.parquet.filterPushdown is on. 

2. Vectorized ORC reader 

spark.sql.orc.enableVectorizedReader has defaulted to true since Spark 2.3.0, but it only applies under the native ORC implementation, which became the default (spark.sql.orc.impl=native) in Spark 2.4.0. Under the legacy hive implementation the flag is ignored. spark.sql.orc.columnarReaderBatchSize mirrors the Parquet setting, defaulting to 4096 rows since Spark 2.4.0. 

3. Nested-column vectorization 

Flat columns were vectorized first; structs, arrays and maps came later. spark.sql.parquet.enableNestedColumnVectorizedReader was introduced in Spark 3.3.0, defaulting to false at first, and has defaulted to true since Spark 3.4.0, but only if the base reader is enabled too. ORC’s equivalent, spark.sql.orc.enableNestedColumnVectorizedReader, arrived in Spark 3.2.0 and also flipped to true in 3.4.0. Either nested flag is ignored when its base reader is off. 

4. Apache Arrow for PySpark and pandas UDFs 

Apache Arrow gives PySpark a columnar wire format for moving data between the JVM and Python, avoiding per-row pickling. spark.sql.execution.arrow.pyspark.enabled governs Arrow transfer for toPandas() and createDataFrame(). It was false throughout Spark 3.x, and the Spark 4.2.0 docs show it defaulting to true; check your own point release, since the default has moved between 4.x branches. spark.sql.execution.arrow.pyspark.fallback.enabled, default true, silently drops to row-based conversion if an Arrow-path error occurs before computation starts. spark.sql.execution.arrow.maxRecordsPerBatch, default 10,000, caps rows per Arrow batch and is worth lowering for wide schemas. Pandas UDFs always use Arrow for the JVM-to-Python handoff regardless of that setting, and Spark 4.2 extends the same default to plain Python UDFs through spark.sql.execution.pythonUDF.arrow.enabled.

Where Stock Spark Falls Back to Row-at-a-Time Execution 

Vectorization in Spark is a path some operators take, not a property of the whole engine, so a plan can slip back into row-at-a-time processing at specific points. Knowing where to look is most of the troubleshooting work. 

1. The ColumnarToRow boundary 

Every vectorized scan has to hand its data to operators that only understand rows, and Spark marks that handoff with an explicit ColumnarToRow node. Anything downstream, most UDFs and legacy operators, runs one row at a time again regardless of how efficiently the scan produced its batches. SPARK-27396 made this boundary visible and pluggable, letting a ColumnarRule’s preColumnarTransitions insert or remove it. 

2. Whole-stage codegen opting out 

Codegen has size limits, exposed through spark.sql.codegen.hugeMethodLimit and spark.sql.codegen.maxFields; a stage with too many expressions or an unusually deep generated method isn’t code-generated and falls back to the interpreted path instead. Spark logs a fallback rather than failing the query, so the first sign is usually a regression on a query that looks the same as last month’s. 

3. Row-based Python UDFs 

A plain Python UDF, unlike a pandas UDF, still serializes one row at a time to a Python worker process and back, even on a Spark version where Arrow accelerates toPandas() and pandas UDFs elsewhere in the same job. That round trip per row is exactly the overhead vectorization removes, which is why a hot row-based UDF in an otherwise vectorized pipeline often shows disappointing wall-clock gains. 

4. Limited JVM auto-vectorization 

The JVM’s JIT compiler can auto-vectorize simple, tight loops, but not across a chain of virtual calls, heap-allocated intermediates and runtime type dispatch, the shape of Volcano-style iterator code. Databricks’ 2016 post showed a hand-written loop beating Spark’s interpreted path by roughly an order of magnitude on the same operation.

How to Check Whether a Query Is Vectorized 

Checking whether a query is vectorized means reading the physical plan Spark generated with df.explain('formatted'), a habit worth keeping for all Spark query performance work. Look for three things: Batched: true on a scan node, a ColumnarToRow node and where it sits, and the WholeStageCodegen (id) groupings around everything else.

== Physical Plan == 
* Filter (1) 
+- * ColumnarToRow (2) 
   +- Scan parquet default.orders (3) 
 
(1) Filter [codegen id : 1] 
Condition : (isnotnull(amount#5) AND (amount#5 > 100.0)) 
 
(2) ColumnarToRow [codegen id : 1] 
Input [4]: [order_id#3L, customer_id#4L, amount#5, order_date#6] 
 
(3) Scan parquet default.orders 
Batched: true 
PushedFilters: [IsNotNull(amount), GreaterThan(amount,100.0)] 
ReadSchema: struct<order_id:bigint,customer_id:bigint,amount:double,order_date:date>

Batched: true on the scan confirms the vectorized Parquet reader produced ColumnarBatch output rather than row objects. The ColumnarToRow node above it is the conversion back to rows, needed because the Filter above is a row-oriented codegen operator. That’s a healthy plan: the conversion costs a little CPU, far less than row-at-a-time Parquet decoding would. A plan with no Batched: true anywhere, or a ColumnarToRow sitting right after the scan before any filter has reduced row count, points at vectorization not doing the work you might assume.

Native Vectorized Engines That Extend Spark Further 

Several engines replace or augment Spark’s own execution with a native, vectorized runtime underneath the same Catalyst-planned query, most of them through the columnar plugin surface SPARK-27396 opened up. They differ in what they run on and how they fall back, and picking between them is one more lever within Apache Spark performance optimization work, alongside partitioning and shuffle tuning. 

1. Yeedu Turbo 

Yeedu’s Turbo engine is a C++ execution layer that runs underneath existing Spark code: a vectorized, SIMD-accelerated, cache-optimized columnar runtime that keeps working data in CPU L2/L3 cache instead of the JVM heap and rewrites the query plan. Jobs run with zero code changes, and Yeedu adds scheduling, multiplexing and caching for the I/O-bound share, covered below. 

2. Databricks Photon 

Photon is Databricks’ native C++ vectorized engine, documented as replacing JVM-based Spark execution at the operator level for SQL, DataFrame, ETL and stateless streaming workloads, processing data in columnar batches with SIMD. Databricks states Photon delivers up to 5x better price-performance on TPC-DS-based measurements against other cloud data warehouses. UDFs, the RDD and Dataset APIs and stateful streaming fall back to JVM Spark transparently, and Query Profile reports the share of task time spent in Photon. 

3. Apache Gluten with Velox 

Gluten converts Spark’s physical plan to Substrait and hands it to the native Velox engine, falling back operator-by-operator to vanilla JVM Spark for anything Velox doesn’t support, with GlutenRowToArrowColumnar and VeloxColumnarToRowExec marking the boundary. Per the Gluten v1.3.0 documentation, ANSI mode (spark.sql.ansi.enabled=true) falls the whole plan back rather than partially, case-sensitive mode can silently return wrong results since Velox only implements Spark’s default case-insensitive semantics, and listed gaps include byte-type Parquet columns, nested arrays of structs and regex lookaround. 

4. Apache DataFusion Comet 

Comet intercepts Spark’s physical plan, translates supported operators and expressions for execution on the native Rust DataFusion engine via an internal protobuf representation, keeping data in Arrow columnar format across scan, operators, expressions, shuffle and broadcast, per the project’s README. Enabling it means adding the Comet JAR to the driver and executor classpath and setting spark.plugins and spark.shuffle.manager: 

spark-submit \ 
  --jars $COMET_JAR \ 
  --conf spark.driver.extraClassPath=$COMET_JAR \ 
  --conf spark.executor.extraClassPath=$COMET_JAR \ 
  --conf spark.plugins=org.apache.spark.CometPlugin \ 
  --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \ 
  --conf spark.comet.explain.fallback.enabled=true \ 
  --conf spark.memory.offHeap.enabled=true \ 
  --conf spark.memory.offHeap.size=4g 

The README lists Spark 3.4 through 4.2, with 3.4 deprecated, so check it for your target release. spark.comet.explain.fallback.enabled=true prints why an operator didn’t qualify for native execution.

5. NVIDIA RAPIDS Accelerator 

RAPIDS offloads eligible Spark SQL operators to GPU execution via spark.plugins=com.nvidia.spark.SQLPlugin and spark.rapids.sql.enabled=true. NVIDIA renames and re-versions the project often, so confirm current flags in its latest docs before scripting against them. spark.rapids.sql.explain=all prints per-operator GPU eligibility to the driver log, lines prefixed ! marking operators that fall back to CPU:

spark-submit \ 
  --conf spark.plugins=com.nvidia.spark.SQLPlugin \ 
  --conf spark.rapids.sql.enabled=true \ 
  --conf spark.rapids.sql.explain=all 
# driver log lines starting with '!' mark operators that fall back to CPU, e.g.: 
# ! <RowDataSourceScanExec> can't run on GPU because GPU doesn't currently support the operator class... 

Which Workloads Benefit from Vectorized Processing in Spark 

Vectorized processing in Spark pays off on CPU-bound work: scans, filters, aggregations, hash joins and expression-heavy projections over wide columnar tables, where the bottleneck is instructions executed per row rather than bytes moved or time spent waiting. It does close to nothing for stages where the CPU is idle waiting on something else. 

I/O-bound reads against object storage, shuffle-bound stages blocked on network fetches, small datasets where per-query overhead swamps per-row savings, and jobs dominated by row-based UDFs all see little from vectorization, because the CPU was never the constraint. Running a vectorized decoder faster than a disk can deliver bytes does not make the job finish sooner. 

The Spark UI already holds the Spark query performance metrics that tell the two apart. The Executors tab reports CPU time against wall-clock task time: a task that spends most of its time on CPU is a candidate for vectorization, while one whose CPU time is a small fraction is waiting on I/O or shuffle. Stage metrics break out Shuffle Read Blocked Time and Fetch Wait Time separately, which is the clearest sign a stage’s slowness has nothing to do with per-row speed. 

Is a slow stage worth vectorizing

A Configuration Reference for Spark’s Vectorized Readers and Arrow Transfer 

The settings below decide whether a scan or transfer runs vectorized; tuning them is routine Apache Spark performance optimization work. Values come from the Spark SQL Parquet, ORC and Arrow docs.

Modeling the Cost Impact with Amdahl’s Law 

Amdahl’s law gives a way to estimate what a vectorization win is worth once part of a job’s runtime speeds up and part doesn’t. If a fraction p of total runtime is CPU-bound and that fraction gets a speedup of s, the new overall runtime is (1 - p) + p/s times the original. The numbers below are illustrative, not measurements of any engine. 

Take a 100-minute job where 35% of the time (p = 0.35) is CPU-bound joins and aggregations, and the other 65% is shuffle and I/O wait.

p = 0.35   (CPU-bound share of runtime) 
s = 5      (illustrative speedup applied to that share only) 
 
new_runtime_factor = (1 - p) + p / s 
                    = 0.65 + 0.35 / 5 
                    = 0.65 + 0.07 
                    = 0.72 
 
100 minutes -> 72 minutes (28% faster overall) 

Push the speedup to s = 10 and the factor only falls to 0.685, about 68.5 minutes. Once the untouched 65% dominates, a bigger speedup on the smaller share buys little, so p matters more than s. Measure p with the CPU-time-versus-wall-clock check before budgeting for any vectorization effort.

Troubleshooting Vectorization Failures 

Most vectorization problems show up as a specific exception, a silently absent Batched: true, or a job slower than the config would suggest, and each has a narrow, checkable cause.  

How Yeedu Turbo Brings Vectorized Execution to Spark 

Turbo, listed first among the native engines above, gives existing Spark jobs vectorized execution without a rewrite, and it runs inside the customer’s own cloud account or on-premises servers. 

Vectorization covers the CPU-bound share of a job. On that share, the joins, aggregations, multi-stage transforms and ML feature prep that the Amdahl example treats as p, Yeedu reports 4-10x faster execution and 60-80% lower compute cost from Turbo. Yeedu’s TPC-DS run, a synthetic decision-support benchmark rather than a stand-in for any particular workload, completed 99 of 99 queries at a compute cost the vendor states as $0.52 for 1TB and $2.33 for 3TB.

The shuffle and I/O wait that vectorization leaves untouched get their own mechanisms: 

• Multi-threaded execution keeps every available core working in parallel. 
• Smart Scheduling packs tasks into idle CPU windows during read and write waits. Yeedu claims 2-4x higher cluster efficiency for ingestion, ELT and streaming jobs. 
• spark-submit multiplexing runs many jobs in one shared Spark session, so data one job caches serves later jobs from memory, with no re-read from object storage. 
• The Turbo Parquet disk cache serves repeat reads of the same files from local disk.

 So a Yeedu job’s gain isn’t capped by its CPU-bound share the way a vectorized engine alone is. 

Yeedu bills an annual licence, while most Spark platforms bill consumption (DBUs or instance-hours). Adoption happens job by job: individual Spark jobs point at the Turbo engine while the catalog, orchestration and the rest of the job estate stay exactly where they are.

Conclusion 

Vectorized execution already runs inside most Spark clusters, quietly, through the vectorized Parquet and ORC readers and through Arrow-backed pandas UDF transfer, well before anyone reaches for a native engine. Native engines extend that baseline to the operators a plugin can take over. Shuffle-bound and I/O-bound stages gain little either way, so check CPU time against wall-clock time before changing any flag. 

• Vectorization batches column values instead of processing rows one at a time, improving CPU cache use, enabling SIMD and stabilizing branch prediction. 
• Whole-stage codegen and vectorization are related but distinct: codegen removes virtual-call overhead, vectorization adds batch-and-SIMD processing on top. 
• Spark’s vectorized Parquet and ORC readers, and Arrow-backed pandas UDF transfer, are on by default in current releases, each with its own version history and exceptions. 
• ColumnarToRow nodes, row-based UDFs and codegen size limits are the three places stock Spark falls back to row-at-a-time execution. 
• explain('formatted') with Batched: true, ColumnarToRow placement and WholeStageCodegen ids is the fastest way to confirm what actually ran. 
• Only CPU-bound stages benefit; shuffle-bound and I/O-bound stages need a different kind of fix entirely. 
• Native engines extend the same idea further, each with its own fallback behavior, and Yeedu pairs Turbo’s vectorization with scheduling, multiplexing and caching for I/O-bound work. 

 FAQs 

What batch size does Spark’s vectorized Parquet reader use? 

The default is 4,096 rows per batch, set by spark.sql.parquet.columnarReaderBatchSize since Spark 2.4.0. The batch is bounded by row count, not bytes, so very wide or array-heavy rows can overflow a column vector; lowering the value to somewhere between 32 and 100 is the documented workaround. 

Is whole-stage code generation the same as vectorization? 

No. Whole-stage code generation collapses a stage’s operators into one generated Java method to remove virtual-call overhead between operators, while vectorization batches data so operators process many values per call using SIMD. Spark’s engine uses both, and they address different costs in the same execution path. 

How do I know if my Parquet reads are vectorized? 

Run df.explain('formatted') and look for Batched: true on the scan node. If it’s missing, check that spark.sql.parquet.enableVectorizedReader is true (the default since Spark 2.0.0) and that the read schema contains only atomic types, since nested schemas need the separate nested-reader flag. 

Why doesn’t my job speed up with a vectorized engine? 

A job with little CPU-bound work, one dominated by shuffle, object-storage I/O or row-based Python UDFs, has little for vectorization to accelerate, because the bottleneck was never CPU instructions per row. Check CPU time against wall-clock task time in the Spark UI’s Executors tab before expecting a gain. 

What causes SchemaColumnConvertNotSupportedException? 

It’s thrown by Spark’s vectorized Parquet reader when a timestamp column uses legacy INT96 storage, lacks timezone information, or carries nanosecond precision the reader can’t decode. The fix is to re-encode the column to a supported type or set spark.sql.parquet.enableVectorizedReader=false for that specific read. 

Do pandas UDFs always use Apache Arrow? 

Yes. Pandas UDFs, including the Series-to-Series, iterator and Series-to-scalar variants, always use Arrow for JVM-to-Python transfer regardless of the spark.sql.execution.arrow.pyspark.enabled setting, which only governs toPandas() and createDataFrame(). 

What’s the difference between Gluten, Comet and RAPIDS? 

Gluten offloads Spark’s plan to the native Velox engine via Substrait, Comet runs it on the Rust-based DataFusion engine keeping data in Arrow format end to end, and RAPIDS offloads eligible operators to GPU execution. All three fall back to standard JVM Spark, operator by operator, for anything they don’t support. 

Can I check GPU eligibility without provisioning GPUs? 

Yes, with the NVIDIA RAPIDS Accelerator. Setting spark.rapids.sql.mode=explainOnly, available since v22.02, or using the qualification tool’s explainPotentialGpuPlan function, available since v21.12, runs the same per-operator eligibility analysis on a CPU-only cluster.