Getting the data out is the hard part. Not the SQL.
Almost every data project is sold on its transformations and sunk by its extracts. The nightly job that competes with month-end close. The schema change nobody announced. The connector written by someone who has since left. Lakeflow is Databricks’ answer to that whole category: managed connectors that read a source’s change stream, declarative pipelines where you state the table you want and the quality it must meet, and one scheduler over everything.
Three tools you were buying separately.
Most data teams have an ingestion product, a transformation framework and an orchestrator from three different vendors, with three sets of credentials, three monitoring screens and three renewal dates. Lakeflow is the argument for having one.
Lakeflow Connect
Managed connectors for databases, SaaS applications, event streams, file sources and cloud storage, with automated change-data capture and schema evolution. Each connector is a Unity Catalog connection, a pipeline and a set of destination streaming tables — nothing you maintain yourself.
Spark Declarative Pipelines
An open declarative framework, standardised on Apache Spark: you declare the tables you want and the expectations they must meet, and the same code runs for batch or streaming. Pipeline code is validated up front, with data quality checks inline, and a real-time mode delivers sub-second latency.
Lakeflow Jobs
One scheduler for ETL, analytics and AI work on serverless infrastructure, with observability and a wide range of task types, so a pipeline, a notebook, a dashboard refresh and a model run can be steps in the same dependency graph instead of four schedules that hope.
Lakeflow Designer and Genie Code
Visual, SQL and natural-language authoring for people who prepare data without writing Spark, and an agentic tool that drafts, tests and deploys a declarative pipeline from a description, using what Unity Catalog already knows about your tables.
Genie ZeroOps
Watches pipeline health, notices schema drift, and uses telemetry and lineage to point at a root cause and propose a fix — which is the difference between an alert that tells you something is broken and one that tells you what broke it.
What a managed connector actually sets up.
“Managed” is a word that hides a lot, so here is what appears in your workspace when you add one. Knowing these four pieces is what lets you reason about cost, about load on the source, and about what to check when it stops.
- 1
First run
Loads everything you selected. This is the run to plan a window for, not the ones after it.
- 2
Then change only
Later runs load what changed where the source supports it — change tracking or CDC on SQL Server, a cursor column on Salesforce.
- 3
On a schedule you set
One or more schedules per pipeline, each creating a job with the pipeline as a task, and more tasks alongside it if you want.
- 4
Or continuously
For database connectors the gateway runs as its own continuous task, so change is captured as it happens rather than in a window.
What is covered, and what you build yourself.
Databricks names examples in each category rather than publishing one exhaustive list, and the list moves every quarter. These are the ones it names today. For your systems specifically, we check the current table rather than assume — a connector that is not there yet changes an architecture, not just a date.
| Kind | Named sources | How it reads |
|---|---|---|
| SaaS applications | Salesforce, Workday, HubSpot, Jira, Confluence, Zendesk Support, Slack and Slack audit logs, Google Ads, Meta Ads, TikTok Ads — and more, Databricks says. | Incrementally where the API allows. Salesforce uses a cursor column from a fixed set of options. |
| Databases, with CDC | Microsoft SQL Server, PostgreSQL, MySQL. | Continuously, through an ingestion gateway and staging storage. SQL Server can use change tracking or CDC. |
| Databases, query-based | Databricks does not list these individually. | Queries the source on a schedule. No gateway, no staging — simpler to set up, heavier on the source. |
| File sources | SharePoint, Google Drive. | Picks up files as they appear, which is how contracts and scanned documents reach the AI functions. |
| Streaming | RabbitMQ, plus Auto Loader for cloud object storage. | Continuously. Structured Streaming underneath, with exactly-once processing guarantees. |
| Community and custom | Open-source connectors maintained by the community; anything else you write yourself. | Runs in your own workspace. This is where an Odoo instance or a local bank file usually lands. |
Will the first run hurt our production database?
It is a full read of everything you selected, so treat it like one: a read replica, an off-hours window, or a staged selection of tables. After that the gateway reads the change stream rather than re-scanning, which is far lighter than the nightly full extract most teams are replacing.
This is also the moment to be honest about table selection. Ingesting everything “because we might need it” is how a platform’s first bill becomes its last.
State the table. Not the orchestration.
The shift here is small to describe and large in practice. Instead of writing the steps that produce a table — read this, join that, write there, in this order, handling failure — you declare the table, the query behind it, and the expectations its rows must satisfy. The framework works out the dependency graph, the incremental processing and the recovery.
The same code runs as a batch job or as a stream, which removes an entire category of duplicated logic: the nightly version and the real-time version of the same business rule, which inevitably drift. Real-time mode brings sub-second latency when you want it.
Expectations are the underrated part. A quality rule written beside the table is enforced on every run, with violations visible rather than discovered in a meeting six weeks later. It is the cheapest data-quality programme available, because it does not need a programme.
What happens at two in the morning.
This is the section most vendor pages skip and the one that decides whether your team trusts the platform. Three mechanisms matter, and none of them removes the need for a named owner.
It retries itself
Failed connectors retry automatically with exponential backoff. Most transient failures — a network blip, a source under load — resolve before anyone is woken.
It keeps its place
When an error genuinely needs a person — an expired credential, a permission revoked — the connector stores its cursor position and resumes from there once fixed. It does not start the history again.
It points at the cause
Genie ZeroOps watches pipeline health, catches schema drift, and uses telemetry and lineage to surface a root cause and propose a fix, rather than handing you a stack trace and a dependency graph.
Every figure above is a customer result Databricks published, not a measurement of ours, and none comes with a method. We quote them because they indicate the shape of the gains people report — development speed and cost, rather than raw throughput. What yours would be is a question for a pilot on your own sources.
Where the data goes once it is in.
Ingestion is the start of the chain, not the point of it.
Unity Catalog
Tables, files, models, agents and dashboards in one permission model, with column-level lineage, attribute-based policies and a gateway that governs what AI is allowed to touch.
Read it →Data warehousingLakehouse
SQL analytics on Delta Lake and Apache Iceberg tables in your own cloud storage, with serverless compute, self-tuning layouts and a real-time tier for dashboards and apps.
Read it →Alongside your ERPSAP Business Data Cloud and Databricks
The connector SAP and Databricks built: SAP data products mounted as catalogues in Unity Catalog, Databricks tables exposed back to SAP, and the business meaning carried across.
Read it →Where this leads next.
© 2026 Mawkish Technologies. This guide, its diagrams and illustrations are protected by copyright. You’re welcome to learn from it and to quote it with credit and a link. Reproducing it, by hand or through AI, and publishing it as your own is not allowed. Read our copyright notice
What people ask about pipelines.
What does a managed connector actually set up?
Three things: a Unity Catalog connection holding the source credentials, an ingestion pipeline, and the destination streaming tables. Pipelines run on serverless compute by default. Database connectors add two more pieces — an ingestion gateway that reads the change stream and a staging area — because capturing change continuously is not the same job as querying a table.
Query-based connectors skip the gateway and simply query the source on a schedule, which is simpler and heavier on the source.
Which sources are covered?
Databricks names Salesforce, Workday, HubSpot, Jira, Confluence, Zendesk Support, Slack and its audit logs, Google Ads, Meta Ads and TikTok Ads among SaaS applications; SQL Server, PostgreSQL and MySQL for change-data capture; Google Drive and SharePoint as file sources; and RabbitMQ for streaming. Beyond those there are community connectors and the option to build your own in your workspace.
The list moves every quarter, and the method differs per source — SQL Server can use change tracking or CDC, Salesforce uses a cursor column from a fixed set. We check the current table for your specific systems rather than assuming.
Is the first run going to hammer our production database?
The first run does load everything you selected, and later runs load only what changed where the source supports it. That initial read is the moment to plan for: a read replica, an off-hours window, or a staged selection of tables.
After that the gateway reads the change stream rather than re-scanning, which is far lighter than the nightly full extract most teams are replacing.
What happens when a pipeline breaks at 2am?
Failed connectors retry on their own with exponential backoff, and when an error needs a human — an expired credential, for instance — the connector keeps its cursor position so it resumes from where it stopped instead of starting over. Databricks also offers Genie ZeroOps, which watches pipeline health, notices schema drift and uses telemetry and lineage to point at a root cause and propose a fix.
None of that removes the need for someone to own the alert. It removes the need for that someone to reconstruct the state of the world first.
Do we have to write Spark?
No. Spark Declarative Pipelines let you declare the tables you want and the quality expectations they must meet, and the same code runs for batch or streaming — there is a real-time mode with sub-second latency. Lakeflow Designer adds visual and natural-language authoring, and Genie Code can draft and test a pipeline from a description using the metadata in Unity Catalog.
Someone on the team still needs to understand what the pipeline is doing. The declarative form just means they are reading intent rather than orchestration code.
Show us the extract everyone is afraid of.
Every data team has one: the nightly job that must not fail, that nobody wants to touch, whose author has left. Show us that one. We will tell you whether a managed connector replaces it, what the first full read would cost your source, and what the migration looks like with the old job still running beside it.
- Your actual sources, checked against the current connector list
- A plan for the first full read
- Quality expectations written beside the tables
- One scheduler instead of three
- The old job left running until the new one reconciles
Databricks, Lakehouse, Lakeflow, Lakebase, Unity Catalog, Genie, Agent Bricks, OpenSharing, Delta Sharing and Lakewatch are trademarks of Databricks, Inc. Apache, Apache Spark, Apache Iceberg and Spark are trademarks of the Apache Software Foundation. Delta Lake and MLflow are projects of the Linux Foundation. SAP, SAP Business Data Cloud and SAP Datasphere are trademarks of SAP SE. PostgreSQL is a trademark of the PostgreSQL Community Association. All other names belong to their owners. Mawkish Technologies is not affiliated with, endorsed by or a reseller for Databricks, Inc. Everything described here is taken from material Databricks publishes; product names, availability and prices change, so check the current documentation before you commit to anything. Screens shown are our own illustrations with sample data, not screenshots.
The Uristead Ladder is our shorthand for looking after the details nobody sees in Databricks Lakeflow.