Building (and actually shipping) a custom Databricks Lakeflow connector for Xero
Having built a number of different integrations with Xero over the years, I finally decided to sit down and write a proper connector for it — one that could land Xero accounting data into a Databricks lakehouse the way I actually wanted it done. This should have been a solved problem — Xero is one of the most widely used accounting platforms in Australia, and Databricks has had a "Lakeflow Connect" story for a while now. It wasn't a solved problem. What follows is the honest version of how we got there: why we ended up writing our own connector from scratch, the specific bugs that nearly ate a week of my life, and why we're open-sourcing the result today.
Why We Built a Custom Databricks Lakeflow Connector for Xero
Databricks publishes a set of community connectors under databrickslabs/lakeflow-community-connectors — a framework for writing custom PySpark Python Data Sources that plug into Lakeflow Declarative Pipelines. It's a reasonable starting point, and I started there. But starting from a vendor authored copy of someone else's framework code creates a real problem the moment you want to actually own and ship the result: you're now maintaining a fork of code under a different license than your own project, with no clean line between "our logic" and "their scaffolding."
So partway through, I made the call to rewrite the whole integration layer — the LakeflowConnect interface, the spec parser, the PySpark DataSource/DataSourceReader implementation, the pipeline orchestration — as original code against the same public PySpark and Lakeflow APIs. Same platform integration, zero vendored source. It's slower than copy-pasting a working framework, but it's the only way to end up with something you can actually put your name on and give away.
That decision turned out to be the easy part.
"It deploys" is not the same as "it works"
Here's the thing nobody tells you about building a custom Python Data Source for Lakeflow Declarative Pipelines: the failure modes don't look like normal Python bugs. They look like platform-internal serialization errors with stack traces that go three layers deep into Databricks' own runtime before you see anything you wrote. I hit four of these, back to back, each one requiring a genuinely different kind of investigation to actually solve (not just work around).
Bug 1: the class you registered doesn't exist, according to the worker that needs it.
The first real deploy threw ModuleNotFoundError: No module named 'xero_connector' — but only inside Lakeflow's schema-inference worker pool, never in the main pipeline execution. The wheel was genuinely installed. import xero_connector worked fine everywhere else. It took a while to find the actual explanation, buried in the upstream framework's own build tooling: spark.dataSource.register()'s registered class gets cloudpickled by reference, and the schema-inference worker pool that needs to reconstruct it can't resolve arbitrary package imports — full stop, regardless of how correctly your wheel is installed elsewhere. The fix was to stop relying on imports entirely: an AST-based build step that walks the registered class's full dependency chain and inlines everything directly into the deployed notebook's own source text. Zero cross-module imports left for that worker to fail on.
Bug 2: two different pysparks, arguing about how to reconstruct a function.
Once that was fixed, a new error showed up, deeper in: AttributeError: Can't get attribute '_function_setstate' on <module 'pyspark.cloudpickle.cloudpickle'>. This is a cloudpickle version mismatch — something got pickled by a newer cloudpickle than the one trying to unpickle it. I chased this through several wrong theories (lambda closures in my own reader code, wheel dependency conflicts) before the real answer turned up in Databricks' own docs: setting a pipeline's environment_version silently switches execution to Spark Connect, which splits the notebook and the actual execution engine into two genuinely separate processes with independently bundled cloudpickle versions. environment_version had only ever been added to fix Bug 1 — and Bug 1's real fix (the AST merge) had already made it unnecessary. Dropping it entirely put everything back on classic, single-process execution, where this class of skew simply can't happen.
Bug 3: what does "return an iterator" actually mean.
DataSourceReader.read() needs to return an iterator, and I'd used a bare generator. That failed too — Spark's streaming source runner caches this between micro-batches via copy.copy(), and generators aren't copyable. Switching to map()-with-a-lambda fixed that, but reintroduced a version of Bug 2 (closures push cloudpickle down a more fragile reconstruction path). Switching again to a plain list got rejected outright by Spark's own type check ([DATA_SOURCE_INVALID_RETURN_TYPE]). The actual answer, once I stopped guessing and just tested each candidate directly, was iter(a_materialized_list) — the one form that's simultaneously copyable, picklable, and satisfies the type contract.
Bug 4 (my favorite, because it was invisible until real data changed):
Once live in production, someone added a new record in Xero and reran the pipeline. Nothing came through. Xero's incremental-fetch contract pairs an If-Modified-Since filter with a page parameter — but page paginates within a fixed filter value, it's not a global counter. My original code advanced modified_since on every page and never reset page back to 1 once a crawl caught up. After the very first successful run, the connector was permanently stuck requesting a page number that would never exist again against an ever-shrinking filtered result set. New records landed on page 1 of the new filter — a page number the connector had already stopped asking for.
Fixing that surfaced a second, subtler cousin of the same bug: If-Modified-Since is an HTTP-date header, which only has whole-second precision — no fractional seconds, full stop. My cursor was millisecond-precise. Persisting it verbatim meant every record updated in that exact second got re-matched (and, since this is an append-only flow with no dedup, re-appended) on every subsequent run. I watched several contacts land three times each in the destination table before tracking that one down.
What I'd tell someone starting this today
If you're building a custom Python Data Source for Lakeflow: assume the registered class needs to be entirely self-contained, no cross-module imports, before you even start debugging anything else. It will save you the exact week I lost to Bug 1.
Test environment_version and channel settings in isolation before you need them for something else — they interact with the execution model in ways that aren't obvious from the field names alone.
If your data source's read() return type is giving you grief, the answer is almost always "a materialized list wrapped in iter()" — not a generator, not a lambda-closure map.
If you're pairing an incremental HTTP filter with server-side pagination, write down explicitly what "page" means relative to your filter before you write the cursor logic. It's very easy to build something that works perfectly on the first run and silently breaks on the second.
None of these are documented anywhere I could find before I hit them. That's the actual reason we're open-sourcing this.
It's open source now
xero-lakeflow-connector is now public, Apache 2.0, under Cypher Agency. It's a from-scratch implementation — no vendored code, no third-party license carve-outs — covering all 26 of Xero's Accounting API list endpoints, client-credentials-only auth (no interactive OAuth dance to manage), automatic Unity Catalog schema provisioning, and every one of the bugs above fixed and covered by a regression test so they can't quietly come back.
If you're building something similar against Lakeflow, or you just want to land your own Xero data into Databricks without re-learning any of the above the hard way — it's yours. Issues and PRs welcome.
Keith
This article was written by Keith Jenneke, Principal Consultant & Practice Lead at Cypher Agency. Keith leads our data platform and AI engineering practice, building on Azure Databricks across professional services, resources, and government.
Comments