Why the industry’s oldest measuring stick still matters, and where it doesn’t
Every platform evaluation runs into the same wall: there’s no common basis for comparison. Take a typical scenario. You’re an engineering leader evaluating five different Spark platforms. Each vendor hands you a whitepaper. Each whitepaper tells a different story: different workloads, different cluster configurations, different metrics. Some report query latency, some report throughput, some report cost. None of the numbers are comparable.
You have a budget decision to make, a migration to plan, and no reliable way to tell which platform will perform better for your team.
That scenario hasn’t gotten easier in 2026; it’s gotten harder. Five years ago you were comparing a handful of mature platforms. Today the market has fragmented: cloud-native warehouses, lakehouse engines, serverless compute, managed Spark, proprietary execution layers. Every vendor has a performance story. Almost none of those stories are told using the same measuring stick.
And the question buyers are asking has shifted. In 2016 it was: “which platform is fastest?” Today it’s: “which platform gives me the best outcome per dollar spent?” That’s a harder question to answer from a whitepaper, and an easier one to obscure with carefully chosen benchmarks.
This is why TPC benchmarks still matter. They’re not perfect, but without them, the only evidence you have is whatever each vendor decided to put in their marketing deck.
Platform A shows it completes an ETL pipeline in 4 minutes. Platform B processes 10 TB in under an hour. Platform C claims 40% cost savings over competitors. Three different measurements, three different workloads, three different hardware setups. Comparing them is like comparing race times from different tracks.
It gets worse. Every vendor picks the workload that makes their platform look best. A columnar engine picks a query-heavy analytical benchmark. A streaming platform picks a latency test. A cost-competitive platform picks a large batch job where fixed overhead amortizes well. Each benchmark is real and each result is accurate, but the comparison across them is still useless.
A standard benchmark is only meaningful when everyone follows the same rules. The same queries run against the same data using the same methodology. Without that consistency, you're just comparing sales pitches.
The Transaction Processing Performance Council is an independent, non-profit body that publishes detailed benchmark specifications: the exact dataset, schema, queries, measurement methodology, and reporting requirements. When two platforms both publish TPC results under the same specification, you can actually compare them. That’s rarer than it sounds.
For analytical workloads (Spark platforms, data warehouses, cloud-native analytics engines), TPC-DS is the benchmark that matters. It models a multi-channel retail enterprise across 24 tables, 99 queries, and scale factors from 1 GB to 100 TB. That description makes it sound manageable. It isn’t.
TPC-DS was specifically designed to expose weaknesses in query planners, execution engines, and storage layers.
Engineers who haven't worked with it often assume it's easy to tune for until they run it.
This isn’t a set of straightforward GROUP BY queries. TPC-DS includes:
Any one of these is manageable. All of them together, across 99 queries, at multi-terabyte scale, with no cherry-picking which queries to run, that’s a real test.
A lot of platforms publish TPC-DS results against a subset of queries (sometimes 80, sometimes fewer) without saying so explicitly. The dropped queries are almost always the correlated subqueries and complex window functions, the ones that break things.
If a platform completed all 99 queries, at scale, without modifications, that’s a meaningful signal. The query planner handles the full range of analytical SQL, not just the tractable half. That matters in production, where your workloads don’t come pre-screened for what the engine can handle.
Run TPC-DS at 1 TB and again at 10 TB. The ratio between those two results tells you more about the engine than either number alone. An engine with poor intermediate result materialization shows superlinear cost growth as data scales. A well-architected one doesn’t. Most vendors publish one scale factor. Ask for all three.
When two platforms both publish complete TPC-DS results (same scale factor, same query set, same reporting methodology), you are comparing actual performance on the same problem, not marketing claims or a demo built for your evaluation. In enterprise data infrastructure, that almost never happens any other way
TPC-DS has always reported price-performance as a co-primary metric - but it was defined in terms of the 3-year total cost of ownership of the priced hardware configuration. In a consumption-priced cloud world, that ownership model no longer matches how you actually pay.
A platform that completes TPC-DS at 1 TB in 20 minutes on a $4/hour instance costs $1.33 for that run. One that finishes in 45 minutes on a $6/hour instance costs $4.50. Same benchmark, same result, 3.4x the cost. That difference compounds at the scale most enterprises actually operate. Publishing cost-per-run alongside execution time is the number that should drive procurement decisions, and it’s still surprisingly rare.
None of the above means you should evaluate a platform on TPC-DS alone. There are real gaps.
TPC-DS generates data with a controlled distribution. Your production environment has skewed join keys, inconsistently formatted strings, schema drift between pipeline versions, columns that are 30% NULL because of an upstream bug nobody’s fixed yet. A platform can ace TPC-DS and still fall apart on your actual data.
That’s a reason to keep going, not to ignore TPC results.
It’s a batch benchmark
TPC-DS has nothing to say about:
Not all TPC benchmarks are relevant to your evaluation. Here’s what each one covers:
For most data engineering teams, TPC-DS is the one that matters. TPC-H is simpler and less representative of modern analytical workloads, useful as a sanity check, not as a primary evaluation criterion.
Most teams treat TPC benchmarks as either the whole answer or completely irrelevant. Both are wrong. The right approach is treating them as one layer in a staged evaluation.
Before investing real time in any platform, verify it can complete the full TPC-DS suite at a scale factor representative of your data volumes. Without query modifications. A platform that can’t clear this bar isn’t ready for your production workloads. This stage is fast and cuts your shortlist quickly.
For platforms that clear the filter, run TPC-DS at your target scale on each one. Record total execution time and total infrastructure cost (compute + storage + network). Use equivalent instance configurations across platforms. Now you have a normalized cost number you can actually bring to a budget conversation.
TPC-DS shows a platform is capable; only your own workloads show whether it’s right for you.
Pull a representative sample of production queries (your most expensive jobs, your most latency-sensitive pipelines) and run them on your shortlist. This is the only evaluation that actually predicts production performance.
Performance and cost are measurable. These aren’t:
A platform that scores slightly lower on TPC-DS but cuts operational overhead significantly can easily be the better choice, especially for teams without dedicated infrastructure engineers.
TPC benchmarks are over thirty years old. The workloads they model predate cloud computing. Critics who call them outdated aren’t wrong about their age.
But the alternative isn’t better information. It’s five vendor whitepapers, each optimized to make that vendor look best, with no common basis for comparison. TPC benchmarks exist to create one: a floor of comparability in a market where every other signal is strategically crafted.
Use them as what they are: a starting filter and the only neutral baseline the industry has. Then go deeper with your own workloads and cost model. The platforms that hold up through all of that are the ones worth running in production.
