A warehouse whose tables are just files you own.
A traditional warehouse takes your data and puts it somewhere only it can read. The lakehouse inverts that: your tables stay as Delta Lake or Apache Iceberg files in your own cloud storage, and Databricks supplies a query engine, an optimiser that rewrites the layout for you, serverless compute that starts in seconds and governance over the top. Another engine can read the same files tomorrow, which is the part that changes the negotiation.
Six words you will meet in week one.
This vocabulary does more work than any diagram. Once these six are clear, most of the architecture arguments resolve themselves.
An open table format over Parquet files, now a Linux Foundation project. It adds a transaction log, which is what turns a folder of files into a table: atomic commits, consistent reads while a write is in progress, and a history you can query.
The other open table format, supported as a managed format here too. If the rest of your estate has standardised on Iceberg, write Iceberg — Databricks will read and write both and let other engines read what it produced.
Because every change is a version in the log, you can audit what happened, query the table as it stood on a date, or roll it back. The first time a bad load lands at 3am this stops being a feature and becomes the reason you sleep.
Row-level changes between two versions of a table, readable as a stream. It is how a downstream table updates incrementally instead of recomputing, and how Lakeflow and Lakebase move change around without anybody writing a comparison query.
A table whose storage and layout Databricks looks after, with the optimisations it reserves for them. It doubles as a batch table and as a streaming source and sink, which is why one pipeline can serve both.
A table that lives in somebody else’s catalogue — AWS Glue, Hive Metastore, Snowflake Horizon — exposed here with Unity Catalog’s fine-grained access controls on top. The way you govern what you have not moved yet.
Nobody should be designing partitions in 2026.
For twenty years the craft of warehouse performance was choosing a partition key and living with the consequences — skewed partitions, small-file problems, a rewrite every time the query pattern changed. Databricks replaced that with two things that run on their own.
Liquid clustering is a self-tuning data layout that needs no partitions and adapts as the table grows; Databricks recommends it in place of partitioning outright. Predictive optimisation then watches the data and the query patterns and decides when to rewrite a table’s layout, without a maintenance job on anybody’s calendar.
The honest framing: this is not magic, it is amortised. The work still happens, it just happens when the platform judges it worth paying for rather than when a human remembered to schedule it. What you get back is the engineer who used to own that calendar.
Illustrative only, and deliberately unlabelled on the axis. The shape is what we see in practice — most of the gain comes from stopping a human guessing the layout — but the size depends entirely on your tables, so the only number worth quoting is one measured on them.
What Databricks claims, and who said it
Shell reports up to 10x faster time-series queries using liquid clustering. Anker Innovations reports saving 50% of annual storage cost and more than doubling query speed. Databricks itself claims up to 6x better price-performance for SQL against traditional cloud warehouses, around 9x for ETL, and that customer queries run roughly 5x faster than three years ago.
These are vendor figures with no published method. They are a reason to run a two-week measurement on your own queries, not a reason to skip one.
The one choice that moves your bill the most.
A warehouse here is not a server you own, it is a size and a type. Get this decision right and most cost conversations never happen; get it wrong and you will pay for idle machinery every night for a year.
| Type | What it is | When it is right |
|---|---|---|
| Serverless SQL warehouse | Compute Databricks provisions and manages, starting in seconds and idling away when nothing is running. Databricks recommends it wherever it is available. | Almost always, and particularly for BI, where load is bursty and the gaps between bursts are where the money goes. |
| Pro and classic warehouses | Compute that runs in your own cloud account, which you start, size and stop. Classic warehouses do not support AI functions. | When a network or residency requirement means the compute itself has to live in your account. |
| Job compute | Created by the scheduler for a single run and terminated afterwards. It cannot be restarted, by design. | Every scheduled pipeline. Running nightly work on all-purpose compute is the single most common waste we find. |
| All-purpose compute | A cluster someone starts to work interactively in a notebook, shared across a team, restartable. | Exploration and development. It should not be running at 4am, and an auto-termination setting is not optional. |
| Lakehouse//RT | A real-time tier on the Reyden engine, chosen from the warehouse selector once enabled, with automatic baseline sizing and node-by-node autoscaling. | When you have a genuine millisecond requirement at high concurrency — and can live with a beta. |
Photon is the vectorised engine underneath the SQL and ETL work; you do not choose it per query, it is what makes the price-performance claims possible. The lever you do control is the type above, plus whether your dashboards read a materialized view or three years of raw events. See pricing and cost for the rest.
The copy you used to make for speed.
Until recently, anything that had to answer in milliseconds at high concurrency needed its own serving database: data copied out of your open tables into a proprietary store, security policies redefined by hand and then drifting, another pipeline to watch, another cluster to pay for.
Lakehouse//RT, announced in June 2026, runs those queries directly on governed Delta and Iceberg tables with no data movement. It sizes its own baseline compute instead of asking you to pick a t-shirt size, and autoscales node by node rather than doubling a cluster.
Databricks reports response times around 10ms on smaller datasets and under 100ms on larger ones, up to 16x better than separate serving layers in preview, and sub-100ms latency at 12,000 queries per second on standard benchmarks. Customers quoted include Meta at tens of milliseconds, SES at 20x faster than its previous setup and Magnite at sub-200ms dashboard queries.
Read the status before you design around it
It is in beta, limited to selected read-only workloads, and reached through your Databricks account team. Independent reviewers note that the headline 12,000 queries per second came from a controlled test on a small batch dataset and that no P99 figure has been published. There is an introductory 30% discount on usage through January 2027.
Our rule: a beta can be in a roadmap, never in a critical path, and every design that mentions one carries a written fallback.
Flat is the claim worth testing: Databricks says latency held steady while throughput rose from a few queries per second into the thousands.
Query what you have not moved yet.
This is the feature that turns a terrifying migration into a sequence you control. You register a connection to a system you already run, and its tables appear in Unity Catalog to be queried in place — with row filters, column masks and attribute-based policies applied here, on the way through.
Proving before moving
- Joining a new domain to an old one without a migration project in between.
- Governing an inherited catalogue under one permission model from day one.
- Reconciling the old warehouse against the new model, side by side, in one query.
- Retiring extracts one at a time, with the consumers moved before the source is switched off.
Make the old system fast
- A federated query runs on the source’s compute, at the source’s speed, and competes with its users.
- It does not make Databricks the enforcement point inside that system — a direct connection is still governed by the source.
- It is a bridge, not a destination. Federated tables left in place for two years become the new legacy.
The most underrated thing on this platform.
You can call a model from a SQL statement. Not through a pipeline, not through a notebook, not through an application someone has to build — as a function, in the query, over a column. For a great many problems that are currently a six-week project, this is a morning.
| Function | What it does | What it replaces |
|---|---|---|
ai_parse_document | Extracts text, tables, figure descriptions and layout from a document. | A document-processing vendor and an integration to it. |
ai_extract | Pulls structured fields out of text or a document against a schema you define. | Hand-written parsers that break on the next template change. |
ai_classify | Assigns text to labels you supply. | A keyword rulebook nobody has updated since 2019. |
ai_query | Runs your own prompt against a model serving endpoint you name. | A bespoke batch inference service. |
ai_forecast | Projects a time series forward to a horizon, as a table-valued function. | A spreadsheet with a trendline in it. |
ai_detect_anomalies | Flags anomalous points in history with expected values, bounds and scores. In beta. | Threshold alerts that fire on every seasonal peak. |
ai_top_drivers | Ranks which dimension values are behind a change in a metric. In beta. | An analyst’s afternoon of slicing a pivot table. |
vector_search | Queries an AI Search index from SQL. | A separate vector database and the sync into it. |
ai_mask, ai_translate, ai_summarize | Masks named entities, translates, summarises — alongside sentiment, similarity and grammar functions. | Three more vendors and three more contracts. |
The part to read carefully: cost. The compute running the query is always billed, and task-specific functions and Databricks-hosted endpoints add inference charges on top. ai_query bills according to the endpoint you name, and a custom or provisioned-throughput endpoint bills its own dedicated serving compute whether it is busy or not. Forecasting and prediction functions carry no separate inference cost. Spend is attributable in the system billing tables, and which functions can be used at all is controlled by Unity Catalog permissions. AI functions do not run on classic SQL warehouses.
Nobody has to learn a new chart tool.
Power BI, Tableau, Looker, Sigma, Qlik and ThoughtSpot are all named as supported, with connections tuned for high concurrency and low latency, and datasets publishable to them without anyone hand-managing an ODBC or JDBC connection.
This matters more than it sounds. The expensive half of a data platform project is rarely the platform — it is retraining four hundred people and rebuilding three hundred reports. Keeping the tool your people already know and changing only what sits underneath it is the cheapest version of this project, and usually the one that finishes.
What does change underneath is worth doing deliberately: materialized views so a dashboard reads a maintained summary that refreshes incrementally rather than scanning raw history on every open, and metric views so the definition of margin lives in the catalogue instead of in a dozen tools. Those two changes account for most of the speed and most of the trust.
The tool and the muscle memory
- Existing reports and their audiences
- The refresh schedules people plan around
- Your report authors’ skills
What the tool is pointed at
- One governed source instead of four extracts
- Materialized views instead of raw scans
- Metric definitions held once, centrally
The scaffolding nobody liked
- Scheduled extracts into spreadsheets
- The “one true report” nobody trusts
- Per-tool copies of the same data
Whether it actually got better
- Dashboard load time, before and after
- Reports retired, not just added
- Questions answered without a ticket
The three pages this one leans on.
A warehouse is only as good as what feeds it, what governs it and what people do with it.
Lakeflow
Managed connectors that capture change from your source systems, declarative pipelines that run the same code for batch and streaming, and one scheduler over the lot.
Read it →GovernanceUnity Catalog
Tables, files, models, agents and dashboards in one permission model, with column-level lineage, attribute-based policies and a gateway that governs what AI is allowed to touch.
Read it →BI and the AI assistantGenie
The place business users live: governed dashboards, natural-language questions answered from curated data, and the shared metric definitions that keep the answers consistent.
Read it →Where this leads next.
© 2026 Mawkish Technologies. This guide, its diagrams and illustrations are protected by copyright. You’re welcome to learn from it and to quote it with credit and a link. Reproducing it, by hand or through AI, and publishing it as your own is not allowed. Read our copyright notice
What people ask about the warehouse.
If the data is in open files, what am I paying Databricks for?
The engine, the optimiser and the plumbing. The files are Delta Lake or Iceberg in your bucket and another tool can read them, but what you buy is Photon executing queries over them, serverless compute that starts in seconds, predictive optimisation deciding when to rewrite a table's layout, liquid clustering replacing the partitions you used to design by hand, and the catalogue that makes all of it safe to expose.
The openness is not the product. It is the exit: it means your data is not held hostage if you change your mind.
Do we have to pick Delta Lake or Iceberg?
Databricks supports both as managed table formats and will let other engines read what it writes. It also supports foreign tables over catalogues it does not own — AWS Glue, Hive Metastore, Snowflake Horizon — so existing tables can be governed through Unity Catalog without being moved first.
In practice the choice follows what else reads your data. If the rest of your estate has standardised on Iceberg, write Iceberg.
What is Lakehouse//RT, and should we wait for it?
It is a real-time tier announced in June 2026, running on a new engine Databricks calls Reyden, aimed at dashboards and applications that need an answer in milliseconds at high concurrency. Databricks reports around 10ms responses on smaller datasets, under 100ms on larger ones, and sub-100ms at 12,000 queries per second on standard benchmarks, with up to 16x gains over separate serving layers in preview.
It is in beta, limited to selected read-only workloads, and access goes through your Databricks account team. Design for it if you have a genuine millisecond requirement; do not build a plan that only works if a beta ships on time.
Can our BI tool connect, or do people have to learn a new one?
Power BI, Tableau, Looker, Sigma, Qlik and ThoughtSpot are all named as supported, and datasets can be published to them without anyone hand-managing ODBC or JDBC connections. Materialized views refresh incrementally behind them so a dashboard is not recomputing the world on every open.
Most clients keep the BI tool their people already know and change what sits underneath it. That is the cheapest version of this project.
Databricks quotes 5x and 9x improvements. Should we believe them?
Treat them as what they are: vendor figures without a published method. The page says customer queries run about 5x faster than three years ago, claims 9x better price-performance for ETL, 4x throughput on batch inference and up to 6x against traditional cloud warehouses for SQL.
The only number that should influence a decision is the one from your own queries on your own data. That is a two-week exercise, not a leap of faith, and it is how we would start.
Let us measure your slowest ten queries.
Give us the ten queries your business actually waits on and the dashboards behind them. We will tell you what they would cost and how fast they would run here, with the method written down so you can check it — and if the answer is that your current warehouse is fine, we will say that too.
- Your queries, your data, your numbers
- Compute type chosen on evidence
- Table layout left to the optimiser
- Materialized views where they pay
- A written method you can audit
Databricks, Lakehouse, Lakeflow, Lakebase, Unity Catalog, Genie, Agent Bricks, OpenSharing, Delta Sharing and Lakewatch are trademarks of Databricks, Inc. Apache, Apache Spark, Apache Iceberg and Spark are trademarks of the Apache Software Foundation. Delta Lake and MLflow are projects of the Linux Foundation. SAP, SAP Business Data Cloud and SAP Datasphere are trademarks of SAP SE. PostgreSQL is a trademark of the PostgreSQL Community Association. All other names belong to their owners. Mawkish Technologies is not affiliated with, endorsed by or a reseller for Databricks, Inc. Everything described here is taken from material Databricks publishes; product names, availability and prices change, so check the current documentation before you commit to anything. Screens shown are our own illustrations with sample data, not screenshots.
We call our approach to Databricks Lakehouse the Tevton Lens: advice that stays on your side of the table.