One copy of your data. Every engine that needs it.
Most companies keep their data twice: a warehouse for the people who report on it and a lake for the people who model it, with a nightly job copying between them and two sets of permissions nobody can fully explain. Databricks is the argument that you only need it once — as open files in your own cloud storage, with a warehouse engine, a pipeline engine, a Postgres database, a BI layer and an agent runtime all reading the same tables under one catalogue.
Everything on these pages is taken from what Databricks publishes — its product pages, its documentation, its engineering blog and its newsroom. Where its own pages contradict each other, or where a price or a release date looks likely to move, we say so rather than choose.
The two-copy problem, and what it costs you.
Nobody set out to build it this way. A warehouse arrived first, for reporting. Then the data science team needed raw history the warehouse could not hold economically, so a lake appeared. Then both needed to be kept in step.
Two copies, two sets of rules
- A warehouse holding curated tables, priced to discourage keeping history.
- A lake holding the raw history, with permissions managed separately.
- Pipelines copying between them, which are the first thing to break and the last thing anyone documents.
- Two definitions of revenue, because each side modelled it independently.
- A separate serving database for anything that needed an answer in milliseconds, and a third copy to feed it.
- Machine-learning features computed one way for training and another way for production.
One copy, one catalogue
- Tables are open files — Delta Lake or Apache Iceberg — in storage you own, in your region.
- ACID transactions, table history and time travel sit on top of those files, so the lake behaves like a warehouse.
- One catalogue governs tables, files, models, dashboards and agents with the same privileges.
- Metric definitions are held once and used by every dashboard, question and agent.
- A real-time tier and a Postgres database read the same governed data instead of a copy of it.
- Another engine can read your files tomorrow, which is what makes leaving possible.
The honest caveat: consolidating onto one platform trades several small lock-ins for one large commercial relationship. The open file formats are the hedge against that, and they are a real one — but the engine, the catalogue, the pipelines and the agent runtime are not portable. Go in knowing which part is open and which part is not.
Eight parts, as Databricks itself lists them.
Databricks groups its own product menu by job rather than by technology: data engineering, data warehousing, governance, BI, AI, database, application development, sharing. Each of the eight has a page here, written to be read on its own.
Lakehouse
SQL analytics on Delta Lake and Apache Iceberg tables in your own cloud storage, with serverless compute, self-tuning layouts and a real-time tier for dashboards and apps.
Read it →Data engineeringLakeflow
Managed connectors that capture change from your source systems, declarative pipelines that run the same code for batch and streaming, and one scheduler over the lot.
Read it →GovernanceUnity Catalog
Tables, files, models, agents and dashboards in one permission model, with column-level lineage, attribute-based policies and a gateway that governs what AI is allowed to touch.
Read it →BI and the AI assistantGenie
The place business users live: governed dashboards, natural-language questions answered from curated data, and the shared metric definitions that keep the answers consistent.
Read it →Artificial intelligenceAgent Bricks
A runtime for agents that survive a crash, memory and sessions, tools and MCP servers, search over your documents, traced runs and evaluation by judges and by your experts.
Read it →DatabaseLakebase
Open-source Postgres with storage and compute pulled apart: branch it like code, scale it to zero, and read the same governed data your analytics read.
Read it →Application developmentDatabricks Apps
Streamlit, Dash and Gradio apps on serverless compute, with sign-in, permissions and Unity Catalog around them instead of a server you have to ask IT for.
Read it →Sharing and collaborationOpenSharing
An open protocol for handing live tables, models and files to another company on another platform, with clean rooms for the work neither side may see, and a marketplace.
Read it →A company that gave away its foundations.
Databricks was founded in 2013 by the people who created Apache Spark at Berkeley, and it has kept open-sourcing the layers underneath its product: Spark itself, then Delta Lake, then MLflow, then Unity Catalog. The pattern is deliberate — the formats and the engine core are open, and the managed platform around them is what you pay for.
That history is the reason the openness claim is more than marketing. Delta Lake and MLflow are Linux Foundation projects; Delta Sharing was developed with the Linux Foundation as a vendor-neutral protocol. It is also why so much of the vocabulary — Spark, notebooks, Delta tables — is already in your team’s hands even if Databricks is not.
Customer and adoption numbers on this page are Databricks’ own, published on its about page and its product pages. We have not independently verified them, and we label them so you can weigh them accordingly. The founding date, the open-source projects and the cloud support are matters of record.
The files are open so you can leave. The platform is good enough that you probably won’t. Both halves of that sentence are the strategy.
Our reading of it, not a Databricks statementIt does not replace the systems that run your business.
This is the question we are asked first, usually with some suspicion, and the answer is simple. Your ERP runs your processes. Your CRM runs your pipeline. Your HR system runs your payroll. Databricks is where what came out of them gets combined, measured and built on.
A connector built by both vendors
SAP and Databricks have published a connector that shares data between SAP Business Data Cloud and Databricks over the open sharing protocol, in both directions, with no copy made. SAP data products mount as catalogues in Unity Catalog, and the table comments, keys and governance tags come with them.
A managed connector, change captured
Salesforce is one of the managed sources in Lakeflow Connect: the connector handles incremental ingestion using a cursor column and keeps the destination streaming tables current, which is a different thing from the scheduled export most teams have today.
Change data capture, not a nightly dump
Anything on PostgreSQL, MySQL or SQL Server — which includes an Odoo database — can be ingested through a database connector with an ingestion gateway reading the change stream, rather than a full extract that competes with your users for the server.
Models behind one governed gateway
Unity Gateway puts frontier and open models behind a single API with guardrails, rate limits, budgets and usage tracking — Anthropic and OpenAI among the named providers, with your own key. Databricks’ own security product uses Claude models inside it.
What arrived in the last year, and how finished it is.
This platform is moving quickly enough that advice eighteen months old is misleading. Several things on it are genuinely new, and a few are not finished. Knowing which is which is the difference between a design that ships and a design that waits.
| What | What it is | How finished |
|---|---|---|
| Lakehouse//RT | A real-time tier on a new engine called Reyden, answering queries on governed Delta and Iceberg tables in milliseconds at high concurrency, so dashboards and apps stop needing a separate serving database. | Beta, selected read-only workloads. Announced June 2026. |
| The new Genie | The business-user experience formerly called Databricks One, renamed and widened: questions answered across spaces rather than inside one, reaching connected sources such as SharePoint and Google Drive, with account-level access and identity sync. | Generally available on web. Mobile apps on request. |
| Agent Bricks | An agent development platform rather than a library: durable runtime, memory and sessions, sandboxed code execution, MCP tools, tracing and evaluation, governed by Unity Gateway. | Shipping, with parts arriving continuously. |
| Lakebase | Serverless Postgres with storage and compute separated, instant branching, scale to zero, and a change feed into the lakehouse with no pipeline between. | Available, with a published uptime SLA. |
| Unity Gateway | Governance applied to AI at runtime: which models, agents, tools and MCP servers may be used, by whom, with rate limits, budgets, guardrails and usage attribution. | Available. Naming still settling against “AI Gateway” in places. |
| Lakewatch | Databricks’ own security monitoring product, built as a security lakehouse rather than a tool you copy logs into, with triage agents and an assistant that writes and tests detections. | Private preview. Announced March 2026. |
We keep this table current because it is the one most people need and nobody publishes. If a design of yours depends on something in beta, we will say so in writing, with what the fallback is.
Begin with the thing that is actually costing you.
Nobody needs a platform. People need a specific problem to stop happening. Find the sentence below that sounds like your week, and the page behind it is the one to read first.
“Three teams bring three different revenue numbers to the same meeting.”
This is a definition problem wearing a data problem’s clothes. Both numbers are usually correctly calculated from different rules. Start with shared business semantics in the catalogue, so the metric is defined once and every dashboard, question and agent computes it the same way. Unity Catalog →
“Our SAP data is the data we need, and we cannot get at it.”
The connector SAP and Databricks built shares data products into Unity Catalog with no copy and with the business meaning attached, and shares Databricks tables back. Start there rather than by writing another extract. SAP Business Data Cloud and Databricks →
“Every dashboard is slow and the warehouse bill keeps climbing.”
Usually a handful of settings rather than a rewrite: serverless compute that idles to nothing, materialized views behind the dashboards, self-tuning table layouts instead of hand-designed partitions. Then look at the rates. Lakehouse → or Pricing and cost →
“Our AI pilot demoed beautifully and never reached production.”
Almost always one of two causes: the agent had nothing governed to retrieve, or there was no definition of good enough and no way to measure it. Both are fixable, and both are fixed before the model is chosen. Agent Bricks →
“We are on a warehouse we want to leave, and the estimate frightens us.”
Federate before you migrate: query the old system in place under one catalogue, prove the numbers reconcile, then move domain by domain. The tooling helps with the SQL rewrite and not much else, and it is better to know that at the start. Migration →
Four things that decide whether it works.
The eight products are what Databricks sells. These four are what determines whether what you build on them survives a procurement review, an audit, a migration and a budget cycle.
SAP Business Data Cloud and Databricks
The connector SAP and Databricks built: SAP data products mounted as catalogues in Unity Catalog, Databricks tables exposed back to SAP, and the business meaning carried across.
Read it →FoundationsSecurity and Architecture
Control plane, compute plane and your own cloud account: what runs where, which certifications Databricks holds, and the network and key controls you can turn on.
Read it →Getting thereMigration
What Lakebridge and the agentic code converter actually do to legacy SQL, where they stop, and how to sequence a move off Snowflake, Teradata, Oracle or a Hadoop estate.
Read it →The billPricing and Cost
How consumption pricing works, what a DBU is, which products bill separately, and the handful of settings that decide whether your platform costs double what it should.
Read it →We would rather talk you out of the wrong build.
A data platform is the easiest project in enterprise technology to start and one of the hardest to finish, because nothing fails loudly. Reports keep being produced. They are just wrong, or late, or nobody trusts them, and that state can persist for years.
So we begin with the inventory and the definitions, not the architecture. What decisions are being made badly today, what data they need, who owns each domain, and what number has to reconcile before anything is switched off. If the answer at the end of that is that you need better processes rather than a lakehouse, we will tell you, and that conversation is free.
It is the same discipline we bring to the other platforms we build on, and it is why the order of the work rarely changes: understand the decisions, agree the definitions, prove one domain, then scale what worked.
Decision inventory
The decisions that are slow or wrong today, the data each one needs, and who owns it.
Definitions before dashboards
The ten metrics that matter, defined once in the catalogue and agreed by the people who argue about them.
One domain, end to end
Ingest, model, govern, publish and measure a single domain, with the old numbers reconciled.
Cost controls on day one
Tags, budgets, the system billing tables, and a monthly review each team can see itself in.
AI on governed data only
Agents pointed at described, certified tables, with a written bar for accuracy before anything ships.
In writing, including the doubts
What is generally available, what is in beta, what we have assumed, and what would change our advice.
© 2026 Mawkish Technologies. This guide, its diagrams and illustrations are protected by copyright. You’re welcome to learn from it and to quote it with credit and a link. Reproducing it, by hand or through AI, and publishing it as your own is not allowed. Read our copyright notice
What people ask before they start.
Is Databricks a data warehouse, a data lake, or something else?
It is one platform that does the work of both, which is what the word lakehouse means. Your tables live as open files — Delta Lake or Apache Iceberg — in your own cloud storage account, and Databricks puts a warehouse engine, a pipeline engine, a notebook environment, an AI stack and a governance layer over the same copy.
The practical consequence is that you do not keep a lake for the data scientists and a warehouse for the finance team, with a nightly job copying between them and two sets of permissions to keep in step.
We already run SAP. Does Databricks replace anything there?
No. SAP runs your processes; Databricks is where you analyse what came out of them and where you build AI on top. SAP and Databricks have built a connector for exactly this: SAP Business Data Cloud data products can be mounted as catalogues in Unity Catalog over the open sharing protocol, with no copy of the data and with the table and column meanings carried across.
We have a page on that connector, what it shares in each direction and what SAP is told about your usage.
How much of this do we have to buy at once?
None of it is sold as a seat count or a suite. Databricks bills consumption: you pay for the compute and storage a workload uses, measured per second, and products such as Unity Catalog come with the tier rather than as a separate line. That means you can start with one warehouse, one pipeline and one catalogue, and the bill follows what you actually run.
It also means an unwatched platform can get expensive quietly, which is why we treat cost controls as part of the build rather than something to look at later.
Does our data have to leave our cloud account?
Your tables and files stay in storage you own, in the region you choose. Databricks splits itself into a control plane it operates — the web interface, job scheduling, metadata — and a compute plane that runs the work. With classic compute the compute plane runs inside your own cloud account; with serverless compute it runs in a Databricks-managed account in the same region, with its own network and key controls.
Which of the two you use changes what you can lock down, so it is worth deciding deliberately rather than by default.
Is it available near Sri Lanka?
Databricks publishes a region list per cloud. On AWS, Asia Pacific (Mumbai) and Asia Pacific (Singapore) both appear with serverless support; on Google Cloud the equivalents are asia-south1 (Mumbai) and asia-southeast1 (Singapore). Individual features — some Lakeflow connectors, some AI services — reach regions at different times, and the Azure list is maintained separately.
We check the current region and feature tables for the specific services a design depends on before anyone signs anything, because a feature that is missing in your region changes the architecture, not just the timeline.
Can we try it before committing?
There is a Free Edition for learning, and a trial of the full platform with credits. Free Edition is deliberately boxed in: one workspace, one metastore, one small SQL warehouse, serverless compute only, no SSO or SCIM, a handful of concurrent tasks, and it may not be used commercially. Features like Agent Bricks and clean rooms are not in it at all.
It is genuinely good for learning the interface and the SQL. It will not tell you what your workload costs, so we usually pair it with a short paid proof on your own data.
Bring us the argument you keep having about numbers.
In one session we will map the decisions that are costing you, say which part of this platform addresses them, and tell you plainly if the answer is something cheaper than a data platform. You will leave with a written view either way.
- The decisions, before the architecture
- Which of the eight parts you actually need
- What is generally available and what is beta
- A costed first domain, not a programme
- Cost controls designed in, not retrofitted
Databricks, Lakehouse, Lakeflow, Lakebase, Unity Catalog, Genie, Agent Bricks, OpenSharing, Delta Sharing and Lakewatch are trademarks of Databricks, Inc. Apache, Apache Spark, Apache Iceberg and Spark are trademarks of the Apache Software Foundation. Delta Lake and MLflow are projects of the Linux Foundation. SAP, SAP Business Data Cloud and SAP Datasphere are trademarks of SAP SE. PostgreSQL is a trademark of the PostgreSQL Community Association. All other names belong to their owners. Mawkish Technologies is not affiliated with, endorsed by or a reseller for Databricks, Inc. Everything described here is taken from material Databricks publishes; product names, availability and prices change, so check the current documentation before you commit to anything. Screens shown are our own illustrations with sample data, not screenshots.
At Mawkish, the Data Runway is how we keep Databricks work checked, owned and answered for, whoever does the build.