@PeterAvenant

Head of Data & AI Engineering at https://xitter.cf/t.co/jnFGXAsKGe

deltavault.ai
Joined September 2013
  • Tweets2K
  • Following152
  • Followers330
609Media
Silver layer modeling methods tell you which business thing a table is about. Only the canonical model says what each column means, and it rewrites every source to do it. Your AI assistant needs that meaning to stop guessing. deltavault.ai/blog/every-met…
Illustration for the article Every method stops at the concept

ALT Illustration for the article Every method stops at the concept

4
If nobody wrote down what a code means, your AI agent will guess it for you. Who on your team approves that meaning?
9
Written as an ontology over the same tables, business meaning lifted large language model accuracy from 16% to 54%. Meaning used to cost a table per concept. Now it costs a sentence mapped to a column. Where does your team keep it today?
Illustration for the article The economics of meaning flipped

ALT Illustration for the article The economics of meaning flipped

7
We recorded our own AI losing to a data engineer. Not on a hard problem: one view over two staging tables. It inner-joined across staging layers and flattened the history the view exists to keep. Catching that in a second is data engineering, not prompt engineering.
Illustration for the article This is why we need data engineers

ALT Illustration for the article This is why we need data engineers

11
Your data model lives in a load notebook, so every migration sends someone back to reinterpret it. Keep the model in the catalog as governed metadata and render native Databricks code from it. A new target changes the template, not the meaning. deltavault.com/blog/keep-the…
Illustration for the article Keep the model stable, change the template

ALT Illustration for the article Keep the model stable, change the template

18
Data Vault wrote business meaning into table structure because a table was all a machine could read. An AI agent reads sentences now, so that meaning belongs in your catalog. How many of your tables exist for something other than meaning?
Illustration for the article Modeling for Machines That Can Read

ALT Illustration for the article Modeling for Machines That Can Read

6
You don't need Data Vault in your lakehouse silver layer. Hubs, links and satellites encoded meaning as tables because tables were all a machine could read. Now a model reads the sentences, so a wrong definition gets read at scale.
Illustration for the article You don't need Data Vault anymore

ALT Illustration for the article You don't need Data Vault anymore

5
Top-down vs bottom-up was never a sequencing question, it's a mapping one. Neither side tells you which source table is which business entity. You find out the day Customer turns out to be a billing account. Map & Match proposes the link. deltavault.ai/blog/top-down-…
Illustration for the article Top-down and bottom-up data modeling: cart before the horse

ALT Illustration for the article Top-down and bottom-up data modeling: cart before the horse

1
12
Your steward is half of somebody's job. Your business expert gives you an hour a fortnight. That's the diagnosis, not the excuse: those hours belong on rulings only they can make, not on plumbing. Start with one subject area. /ebooks/the-data-lakehouse-for-everyone/
Illustration for the article No silver layer, no lakehouse

ALT Illustration for the article No silver layer, no lakehouse

1
10
Classify Email Address once, then find it unclassified in three staging tables and a vault satellite. That is a correspondence problem, not a coverage one. Record the mapping and the meaning travels. /ebooks/the-data-lakehouse-for-everyone/the-mapping-is-the-deliverable/
Illustration for the article The Mapping Is the Deliverable

ALT Illustration for the article The Mapping Is the Deliverable

1
20
You don't have 500 design problems in your integration layer. You have a dozen patterns, copied 500 times. Hand-written, your real design ends up in the diffs between files. Rendered from your catalog, it lives in one template you can read. deltavault.ai/blog/integrati…
Illustration for the article Integration Layer: Pattern, Not Project

ALT Illustration for the article Integration Layer: Pattern, Not Project

6
Databricks foreign keys are informational only, so an imported schema lands with no joins recorded. DeltaVault's discovery skill drafts them, each with a score and a reason. You approve every one. /blog/ai-drafts-the-tedious-part-you-keep-the-decision/
Illustration for the article AI drafts the tedious part. You keep the decision.

ALT Illustration for the article AI drafts the tedious part. You keep the decision.

10
A coding assistant reading your Jinja template guesses field names, because nobody gave it the contract. We documented that contract field by field. Drafts start from real names, then meet a live preview and validation on save. deltavault.com/blog/own-the-…
Illustration for the article The template engine is yours

ALT Illustration for the article The template engine is yours

6
Lineage harvested from query logs tells you what ran, not what anything was meant to be, and it lands mid-incident when you have no time to check it. Write the edge when the mapping is made instead: click any column, and the whole path lights up, ancestors and descendants.
Illustration for the article Meaning Leaks at Every Handoff

ALT Illustration for the article Meaning Leaks at Every Handoff

8
DeltaVault never pushes to your Databricks workspace. Build renders an immutable snapshot you read file by file, and a build with a failed artifact can't be committed at all. Deployment isn't the interesting moment. Review is. deltavault.com/blog/portabil…
Illustration for the article The model is portable. The code is native.

ALT Illustration for the article The model is portable. The code is native.

16
Re-run an AI recommendation two weeks later and most tools hand back suggestions that know nothing about the first, so review becomes deduplication. Accept context with the proposal and every later run inherits it, and so does every descendant. /blog/tags/how-we-use-ai/
Illustration for the article AI that explains how reality became data

ALT Illustration for the article AI that explains how reality became data

28
The modeling was the fast part. The next quarter goes to staging objects, hash keys and load order. Most of it is derivable: business keys become hubs, reference columns links, descriptive columns satellites, and the run order falls out of the lineage your catalog already holds.
Illustration for the article You already own the stack

ALT Illustration for the article You already own the stack

2
We built AI into every corner of DeltaVault, then watched it lose to a data engineer on one view over two staging tables. The panel flattened the history that view exists to keep. Right-click, Create View, correct the first time. youtu.be/UvPJF7Q6Pcc
Illustration for the article This is why we need data engineers

ALT Illustration for the article This is why we need data engineers

3
Your prompt is not the bottleneck. The glossary, the workshop notes, the dictionary nobody opened since go-live are. Upload them and DeltaVault pulls out the business entities and attributes, quoted verbatim, ready for the modeling workshop. deltavault.ai/blog/give-ai-c…
Illustration for the article Give AI Content, Not More Prompts

ALT Illustration for the article Give AI Content, Not More Prompts

7
Your catalog drifts because nothing ties it to the code your team maintains by hand. Declare a table once, let the platform write the create-table, staging, merge and workflow code. Lineage is captured when you declare it, not mid-incident. deltavault.ai/blog/manage-da…
Illustration for the article Manage data, not plumbing

ALT Illustration for the article Manage data, not plumbing

11
Load more