LinkedIn Posts

dbt Core v2.0 is being built with AI agents as the primary consumer, not analysts running scheduled jobs.

The shift is subtle but real. dbt's semantic layer — the transformation code that defines what a customer is, what revenue means, how to calculate churn — has always been human-readable. v2.0 appears to be designed so that layer is also machine-readable in a way that lets an agent call into it, understand the lineage and intent, and compose transformations without a human in the loop.

That's different from dbt Cloud scheduling a job. That's different from an orchestrator triggering a dbt run. This is about making the semantic intent portable to systems that don't think in SQL or YAML. The exact mechanism hasn't been detailed yet, but the collaboration with Fivetran signals this is not speculative — it's already being built.

For teams on the Snowflake Native App with a November deadline, the migration is straightforward: move to Cloud or self-hosted. For everyone else, the question is whether your dbt project structure — the way you've organized transformations, documented lineage, defined metrics — will map cleanly to whatever v2.0 exposes to agents. The technical work doesn't disappear. It moves up a level.

What's your current dbt structure optimizing for — analyst-friendly readability or agent portability?

DuckDB 1.5.5 shipped stable support for Delta and Iceberg without the experimental tag, which means you can use it for Databricks lakehouse operations at production confidence levels.

The practical implication is that a single-machine analytical pipeline can now read and write to a Unity Catalog or Delta Lake directly, without staging data locally first or syncing to a separate system. That removes a whole class of infrastructure decisions — the intermediate landing zone, the sync job, the permissions model between your local environment and the warehouse.

For teams doing exploratory analysis, building data products, or running feature engineering pipelines, this matters because DuckDB becomes the natural staging layer. You're not choosing between "local analytics with imported data" and "query the warehouse directly." You're working with the same table formats your production system uses, locally, with sub-second latency.

The constraint to watch: you're still bound by local compute. Complex joins or aggregations over large Iceberg tables will still be slower than a warehouse query. But for 90% of analytical work — filtering, transforming, enriching — local is now a genuine option.

Are you currently staging Iceberg or Delta data locally, or pushing all work to the warehouse?

Apache Ossie entered ASF incubation, which means the Open Source Intelligence project is now governed and licensed under Apache terms — a stability signal for teams building systems that parse unstructured and semi-structured data at scale.

ASF incubation doesn't guarantee the tool will work for your use case. It does mean the governance is transparent, the codebase is legally portable, and the maintainers are committed to long-term stewardship. The forking risk drops dramatically.

For data teams ingesting logs, events, user-generated text, or any semi-structured source where schema inference and adaptive parsing are requirements, that's a meaningful shift. You're not evaluating a promising open-source project that might stall or change license. You're evaluating a project with institutional backing and a documented process for stability and deprecation.

The practical difference is small but worth noting: you can now recommend Ossie to other teams with more confidence, and you don't need to build contingency around the project being abandoned.

Have you hit scaling limits with your current parsing approach for messy data?