Work
4 companies, and the writing that came out of each.
145 merged pull requests to the open-source dlt framework: documentation across dozens of sources and destinations, benchmark studies, and Fivetran/Stitch-to-dlt migration engineering.
Data work at dltHub: documentation, migrations and benchmarks
Pipelines from Postgres, HubSpot and Freshdesk into BigQuery through bronze, silver and gold dlt layers, orchestrated on dltHub, served in Metabase dashboards.
Navit's data stack: three years in production, then its move to dltHub
Schema design for the in-house app, data quality checks on its database, incremental pipelines into BigQuery, and property and customer scores, shown in Looker.
Data work at RentLondonFlat.com: from app schema to dashboards
Nine years in capital projects: cost models and rate analysis, tender evaluation, and billing reconciliation across budgets of $50M+.
For nine years, my data had to be good enough for a judge
Published at dltHub
In March 2024, we published “dlt adds Reverse ETL”. This piece walks through an example GTM workflow: turning a hiring signal into enriched, sales-ready company and people records, then reverse-ETL them to Attio.
In this article, I walk through moving a HubSpot connector and its dbt models from Fivetran to dlt. I keep the old stack running as the reference, build against its schema, and diff until every mismatch has an explanation.
Schema evolution is a decision every data pipeline makes — most tools make it silently. This post discusses the five common failure modes every data pipeline sees, how dlt handles them, and how you can decide runtime policies for schema evolution with data contracts.
You pay for compute hours; what you actually want is data moved. This post measures the exchange rate across the five bottlenecks that dominate real pipelines: SQL copy, REST APIs, JSON files, Parquet, and MongoDB.
Write your access policy as a plain-English ontology. Schema evolves; the LLM reads the rules and decides.
dlt handles schema evolution efficiently but silently. Here's how to read dlt's metadata and be informed of what's shifting in your pipeline.
A "Success" exit code only tells you the pipeline ran. Use load_id to join _dlt_loads with your source table and check if the data is actually fresh.
Stop PII leaks before they hit your warehouse. By using dlt and Pydantic to enforce data contracts, you can sanitize or quarantine sensitive fields the moment they’re ingested.
Moved 5M rows from DuckDB to MySQL 3.7x faster, reducing time from 344s to 92s by switching from SQLAlchemy’s row-by-row path to Arrow + ADBC’s columnar pipeline.
11 practical, copy-paste data quality recipes for dlt. From schema freezes to alerts, learn how to keep pipelines clean, safe, and production-ready.
Start local with DuckLake, validate your data, then deploy to MotherDuck in minutes. Same pipeline, same code, just switch the destination.
Learn how dlt automates SCD2 for nested JSON data without complex SQL headaches. Real BigQuery benchmarks show incremental loading cuts costs by 25-35%.
Emmanuel built a slim framework on top of dlt that levels up the vanilla Kafka source into a production-ready setup.
We compared dlt and Sling for data ingestion performance, cost, and flexibility. See how they stack up and which might suit your data needs best.
Ajay Moorjani turned a deceptively simple JSON to Snowflake task into a rock solid pipeline using dlt, dbt, and Airflow, built in less than a coffee break.
REST API integrations come with hidden costs: pagination, schema drift, rate limits. With dlt + Cursor, you skip the boilerplate and build pipelines in minutes, not days.
Ingesting to Databricks should be simple. With dlt, it finally is. No config files, no staging, just Python and go.
The Write. Audit. Publish. framework brings discipline from software engineering: write in isolation, audit for correctness, publish with confidence. But can data engineering really follow suit?
Want to run SELECT * on your API data without setting up a database? dlt datasets let you query API data using SQL, following the Write-Audit-Publish pattern.
The sqlmesh init -t dlt command integrates dlt’s metadata with SQLMesh’s modeling, generating SQL models that handle incremental processing and schema changes without writing SQL.
2024 was a remarkable year for dltHub. Together with our users and partners, we streamlined workflows, introduced powerful capabilities, and laid a stronger foundation for the future.
SQL is key in data analysis, especially where production databases are used. We benchmarked Meltano, Airbyte, dlt, and Sling.
Moving data from SQL databases into BigQuery, Redshift, and Snowflake is core to modern data work. We benchmarked Fivetran, Stitch, Airbyte, and dlt to answer which tool fits which need.
A step-by-step migration of HubSpot pipelines from Fivetran to the open-source dlt, with strategies for unifying sources and keeping reporting intact through the switch.
How to migrate your SQL data pipeline from Airbyte to dlt, an open-source solution that offers greater control, speed, and cost-efficiency.
How to migrate your SQL data pipeline from Stitch to dlt, an open-source solution that offers greater control, speed, and cost-efficiency.
How to migrate your SQL data pipeline from Fivetran to dlt, an open-source solution that offers greater control, speed, and cost-efficiency.
The Operations team needed to gather information through Google Forms and land it in a Notion database. The Zapier connector did not work as expected, so we built the sync with dlt.
