DuckDB popularity is best explained by a mismatch between the data most teams actually query and the infrastructure built to query it. Jordan Tigani, BigQuery’s former founding engineer, argued in his 2023 essay “Big Data is Dead” that among customers spending over $1,000 a year on BigQuery, 90% of queries scanned less than 100MB, a dataset size that fits in a laptop’s memory, let alone a modern server.
Single-node engines such as DuckDB and Polars now handle a slice of that work directly on a laptop or inside a single cloud instance. Managed services such as MotherDuck extend the same engine into the cloud, and distributed platforms built on Apache Spark, including Databricks and Amazon EMR, stay the default once a job genuinely outgrows one machine. A fourth category re-engineers Spark’s own execution layer: products such as Yeedu run inside a customer’s existing cloud account, and a team points a specific Spark job at it while the catalog, notebooks and orchestration around every other job stay exactly where they are. DuckDB popularity among the first group comes down to three things happening at once: the data turned out to be smaller than assumed, the hardware got much larger, and the engine removed the steps that used to sit between a question and an answer.
In this guide, we walk through the forces in roughly the order they appeared: the data-size argument, the hardware behind it, the architecture that took advantage of both, the developer experience that followed, and the ecosystem that formed around it. It closes with what the popularity does not mean, and where distributed engines and Spark execution layers still do work that a single node cannot.
Related Read: Where does DuckDB help
The Data and Hardware Arguments Behind DuckDB Popularity
We see two separate trends converge to make single-node analytics plausible again: the data most teams query turned out to be smaller than the systems built to hold it, and the hardware available to query it got dramatically bigger.
Query sizes smaller than the infrastructure assumes
Tigani’s essay is the clearest public statement of the data-size argument behind DuckDB popularity, and it is worth reading as an argument rather than a settled fact. Beyond the 90%-under-100MB figure, he points out that many thousands of BigQuery customers paid under $10 a month for storage, which at BigQuery’s rates is roughly half a terabyte, meaning the total data many organizations keep is itself small. He frames the big data systems of the 2010s as built for a minority of workloads and sold to everyone, a claim about the industry’s defaults rather than a measurement of any single company’s data.
Single-VM capacity today
A single cloud VM today carries more memory and more cores than the clusters distributed analytics was designed around. AWS lists the r5d.24xlarge and r5d.metal instances at 96 vCPUs, 768 GiB of RAM and four 900GB local NVMe drives, and its High Memory u-6tb1.112xlarge offers 448 vCPUs and 6,144 GiB of RAM on a single box. Tigani’s essay makes two related but separate points: that a modern node with 64 cores and 256GB of RAM is a common configuration today, and that a single modern node, he argues, can match the throughput that Google’s original Dremel paper needed roughly 3,000 parallel nodes to achieve. A job that needed a cluster in 2014 can plausibly fit on one instance in 2026, and that gap is most of the hardware argument behind DuckDB popularity.
The Architecture That Makes Single-Node Analytics Work
DuckDB’s design turns the hardware headroom described above into throughput, not a bigger, slower machine. It runs embedded, in-process, the way SQLite does, rather than as a server a client connects to over a socket. Data is stored in columnar form, reading only the columns a query touches.
In-process execution and columnar storage
That embedded model is the starting point: no socket, no port, no separate process to supervise, just a library linked into whatever application needs it. Columnar storage reads only the fields a query touches, which matters more as a table gets wider, and it is the layout OLAP engines have favored over row storage for exactly that reason.
Vectorized, morsel-driven parallelism
Query execution is vectorized and push-based: operators process batches of values at a time, and work is divided into small units called morsels, scheduled across available cores as they free up. That morsel-driven model lets one query use every core on a 96-vCPU box without the coordination overhead a distributed shuffle carries.
Out-of-core spilling and zero-copy interchange
DuckDB’s own troubleshooting docs describe out-of-core support for grouping, joining, sorting and windowing: an oversized hash table is radix-partitioned and spilled to a configurable temp_directory instead of raising an out-of-memory error, though several blocking operators stacked in one query can still exhaust memory. DuckDB also interchanges data zero-copy with Apache Arrow and dataframe libraries, moving a result into Pandas or Polars with no serialization step. We leave the rest of that story to a deeper architectural guide.
.png)
Out-of-core spilling fallback path
The Developer Experience Argument
The argument practitioners actually cite for DuckDB adoption centers on what the engine removes from a day of work more than on benchmark numbers, and that is also where DuckDB popularity is easiest to see without reading a single chart.
Embedding instead of a server
Installing DuckDB is one dependency, no server, no port, no credentials to arrange before the first query. The same binary and SQL run on a laptop, in CI and in production, collapsing a class of works-on-my-machine bugs. DuckDB ships as a library as comfortable inside a unit test as inside a notebook.
Dialect ergonomics
DuckDB’s SQL dialect trims boilerplate standard SQL has carried for decades: GROUP BY ALL groups by every selected column that is not an aggregate, SELECT * EXCLUDE avoids a long column list, and trailing commas and column aliases usable in WHERE and GROUP BY remove constant friction, per DuckDB’s 2022 “Friendlier SQL” post.
-- GROUP BY ALL: group by every non-aggregated SELECT column
SELECT category, region, SUM(sales)
FROM data
GROUP BY ALL;
-- SELECT * EXCLUDE: all columns except named ones
SELECT * EXCLUDE (internal_id, debug_flag)
FROM events; Reading Data Where It Already Lives
DuckDB queries Parquet, CSV, JSON, Arrow buffers and dataframes in place, no load step, removing one of the slowest parts of a typical workflow. The httpfs extension extends that to object storage, with partial reads of remote metadata.
INSTALL httpfs;
LOAD httpfs;
SET s3_region='us-east-1';
SELECT * FROM read_parquet('s3://your-bucket/file.parquet'); Per DuckDB’s core-extensions docs, httpfs sits at “Primary tier,” community-supported, while delta and iceberg sit at “Secondary tier,” best-effort. Delta installs with INSTALL delta; LOAD delta; and autoloads on first use. The absent ingest step is a large part of the DuckDB adoption story: a format already written for other tools becomes one DuckDB can query directly.
The DuckDB Ecosystem and DuckDB Adoption Flywheel
DuckDB’s extension mechanism, a plugin interface adding functions, formats and storage backends without bloating the core binary, let a surrounding DuckDB ecosystem grow faster than the project alone.
dbt-duckdb and the transformation layer
dbt-duckdb lets a team run dbt models against a local or embedded DuckDB database instead of a warehouse, part of why DuckDB shows up in CI pipelines testing transformations before anything touches production.
DuckDB-Wasm and embedded engines
DuckDB-Wasm runs the same engine compiled to WebAssembly inside a browser tab, and the core project is embedded inside other tools and products as a query layer rather than sold as the destination itself.
MotherDuck and cloud scale-out
MotherDuck is the managed cloud service built around DuckDB; its Dual Execution model routes query operators between the local DuckDB process and its cloud engine, able to join local and cloud data in one query, with data transfer between the two typically the main bottleneck per MotherDuck’s own docs. Pricing as of November 2026: a free Lite tier covers 3 users, 10GB storage and 10 hours of Pulse compute a month, Business runs $250 per organization monthly plus metered compute from $0.60 to $24 an hour by size.
.png)
MotherDuck dual execution routing
DuckLake, Governance and Release Stability
DuckDB’s popularity rests on what backs the project as much as on the engine itself: a metadata format, a licence and a release policy.
DuckLake and the metadata argument
DuckLake stores lakehouse metadata, schema, snapshots, file lists and statistics in a SQL catalog database such as a DuckDB file or Postgres, instead of scattered JSON and Avro files, while data stays in Parquet. DuckLake 1.0 shipped April 13, 2026 as a production-ready specification with a backward-compatibility guarantee.
INSTALL ducklake;
ATTACH 'ducklake:metadata.ducklake' AS my_lake
(DATA_PATH 's3://bucket/data/');
USE my_lake;
CREATE TABLE t AS SELECT 1; Licence, origin and the DuckDB Foundation
DuckDB and DuckLake are MIT-licensed; intellectual property sits with the non-profit DuckDB Foundation, kept separate from the commercial company DuckLabs. The project began as research at CWI, started by Mark Raasveldt and Hannes Mühleisen in 2018, with the Foundation committed to keeping DuckDB open-source under MIT in perpetuity.
Release cadence and LTS support
DuckDB 1.0.0 shipped June 2024 with a guarantee that files written in that format stay readable by future versions. Starting with version 1.4, every other minor release is Long-Term Support; as read on 2026-11-02, the release calendar shows 2.0.0 LTS shipped October 21, 2026, with roughly a year of support per line.
Cost Pressure and Performance Benchmarks
Part of why DuckDB is popular is a FinOps story, and part of it is a benchmarking story, and the two get confused more often than they should.
The FinOps case for single-node analytics
A cluster that scales out bills for idle compute between jobs and pays a startup cost on every spin-up. Databricks bills a normalized DBU per second with rates varying by cloud and workload, and MotherDuck bills per second from $0.60 to $24 an hour by size. A process running inside compute already paid for is the FinOps appeal; it is also the structural difference behind Yeedu’s pricing, a fixed annual licence with unlimited usage rather than a metered rate, worth noting without a head-to-head figure.
Public benchmarks and their scope
DuckDB’s own blog documents DuckDB performance on the h2oai database-like-ops benchmark, run on an AWS m4.10xlarge with 40 cores and 160GB of RAM, covering 10 groupby and 5 join queries run twice each. The post frames results like these as single-machine numbers, not a verdict on distributed workloads, and the suite has been rerun on newer hardware since without changing that scope.
What DuckDB Popularity Does Not Mean
Single-node analytics does not fit every job, and we think the honest version of this guide has to spend real space on what the architecture does not do.
DuckDB’s own concurrency docs are explicit: one process reads and writes a database, or many processes read it concurrently in read-only mode; concurrent writers from separate processes error. That single-writer model trades write concurrency for fast reads from memory, ruling out DuckDB as a shared, many-writer database the way a warehouse is used. Large joins and sorts still spill to disk, but several blocking operators stacked in one query, per DuckDB’s docs, can still exhaust a single machine’s memory.
There is no cluster-level retry if the one machine a job runs on fails partway through. DuckDB’s release line keeps moving: 1.5.0 shipped March 9, 2026 and 2.0.0 followed October 21, 2026, both within the same calendar year, per DuckDB’s own release calendar. A project moving that fast rewards users who track it closely, but it leaves a team embedding DuckDB to requalify behavior at each upgrade, why pinning to an LTS line matters more than chasing the newest release. None of this erases why DuckDB is popular for most jobs; popular for most jobs and right for the job at hand are different claims.
Where Spark and Distributed Engines Still Win
In our view, single-node analytics moves the line between one machine and a cluster; it does not erase the cluster side. Large, shuffle-heavy joins across terabytes, streaming with exactly-once guarantees, and hundreds of concurrent analysts writing to shared tables, in our view, stay on a cluster, because that is what fault-tolerant, horizontally scaled execution is for.
The same single-machine wager DuckDB makes for new SQL is made separately for Spark code that already exists. Yeedu is a re-engineered Spark execution layer running inside a customer’s own cloud account or on-premises servers, adopted one job at a time: a team points a specific Spark job at it while the catalog, notebooks, dashboards and orchestration DAGs around every other job stay put. Its Yeedu mode runs an entire Spark job on a single VM instead of a driver plus a fleet of workers, so no distributed shuffle happens at all, the same argument scoped to jobs that fit on one VM. Some workspaces list DuckDB alongside Python, PySpark, SQL and Pandas as a supported notebook language, so both sit in one workspace.
What a data team can do about this next week is narrower than adopting a new engine wholesale, and, we think, more useful for it. Audit the scheduled jobs small enough for one node. Keep data in open formats such as Parquet, Iceberg or Delta so the engine choice stays a choice, and decide per job, not per estate.
.png)
Choosing where a workload runs
| Workload shape | Fits best on |
|---|---|
| Ad hoc analysis on a file or two | DuckDB or Polars, no server |
| Dashboards needing cloud scale-out | MotherDuck, routing between local DuckDB and its cloud engine |
| Existing Spark job fitting one large VM | A single-VM Spark layer such as Yeedu mode, no shuffle |
| Multi-terabyte shuffles, streaming, high concurrency | Spark or Databricks on a cluster |
| Many concurrent writers to shared tables | A warehouse or cluster; DuckDB’s single-writer model does not fit this |
Conclusion
DuckDB’s rise is a hardware-and-data-size story more than a maturity story: cloud machines grew large enough to hold what most jobs touch, and an in-process engine removed the server and the load step between a question and an answer. We don’t think it erases the distributed end of the spectrum; it makes the dividing line a per-job decision.
- Most analytical queries touch far less data than the systems built for them assume, per Jordan Tigani’s 2023 “Big Data is Dead” essay.
- A single cloud VM now carries the cores and memory a cluster needed a decade ago, per AWS’s own instance specs.
- DuckDB’s in-process, columnar, vectorized design turns that headroom into throughput, with out-of-core spilling as a safety net, not a scaling strategy.
- One dependency, no server, the same SQL on a laptop and in production: the developer experience practitioners cite most often.
- DuckLake, the MIT licence and the LTS release line are the governance story behind the project.
- A single-writer model and memory limits on stacked operators are real constraints, not footnotes.
- Shuffle-heavy joins, streaming and high concurrency, plus Spark execution layers applying the same single-machine logic to existing jobs, stay a separate decision.
FAQs
Why is DuckDB getting so much attention?
DuckDB gets attention because it answers most analytical queries on object storage or a local file without a server, a cluster or a load step, at a time when a single cloud VM has enough RAM and cores for datasets that once needed a distributed system.
Where does a Spark execution layer fit if DuckDB handles the small jobs?
A Spark execution layer fits Spark jobs that already exist and need more than one machine’s worth of data, which DuckDB was never built to run. Yeedu's TPC-DS benchmark, a synthetic decision-support benchmark, completing 99 of 99 queries at a compute cost of $0.52 for 1TB and $2.33 for 3TB on what the vendor describes as a production-grade configuration; that is a statement about Spark execution cost, a different question from whether a given job fits one node.
Is DuckDB production ready?
Yes, with the scope that implies: DuckDB 1.0.0 shipped June 2024 with a guarantee that its storage format stays readable by future versions, and the project ships alternating Long-Term Support releases with roughly a year of support each. Production-ready describes stability, not fitness for every workload.
Is DuckDB replacing data warehouses?
No single source here supports that claim outright: DuckDB replaces the need for a warehouse on jobs that fit one machine, while shared, many-writer, high-concurrency workloads stay on warehouse-style or distributed systems built for that. The two tend to coexist inside one organization.
What is DuckLake?
DuckLake is a lakehouse format storing table metadata, schema, snapshots and file statistics in a SQL catalog database such as a DuckDB file or Postgres instead of scattered JSON and Avro files, while data stays in Parquet. It shipped as DuckLake 1.0 in April 2026.
Is DuckDB free and open source?
Yes. DuckDB and DuckLake are MIT-licensed, and intellectual property is held by the non-profit DuckDB Foundation, kept separate from the commercial company DuckLabs, whose statutes commit to keeping the project open-source under MIT in perpetuity.
Does DuckDB scale to big data?
DuckDB scales up on one machine rather than out across many, spilling oversized joins and aggregations to disk instead of raising an out-of-memory error, which covers most jobs but not all. Workloads needing many machines stay on Spark, Databricks or a Spark execution layer such as Yeedu, each covering a different part of the spectrum.
