Reminded of Windows in 90s. The amount of confusion there is around Codex, Chat vs Work, Personal vs Business profile switching -- in ChatGPT.

Agree. Managing traces longer term and extracting patterns will be the data science equivalent for agents.

The thing that makes AI agents useful ("non-deterministic reasoning") is exactly what makes them impossible to debug. Traditional automation was boring because it was predictable. Same input, same code, same output. AI is useful precisely because it breaks that model. It can "think on its feet", at 1000x human speeds. So, if you give an agent the same goal twice and it may take a different path, call different tools, retrieve different context, and produce a different answer. That autonomy is the value prop. The balance lies in ensuring that when something goes wrong, you can reproduce what actually happened. Teach it, Coach it. For that, versioning the model is not enough. You need the exact state of everything that influenced the result: • source data • transforms • embeddings • indexes • policy/config • model checkpoint Miss one, and you are not replaying the same run. AI makes systems non-deterministic. Versioning is how we make that non-determinism debuggable, auditable, and reproducible.

Full breakdown: onehouse.ai/blog/announcing-…

The breakthrough is format separation + hardware alignment + efficient cache distribution. That’s what unlocks consistent sub-100ms query latencies on lakehouses.

⚡ To actually hit 10s-of-millisecond latencies on lakehouse storage, you need three things working together: 1️⃣ Table storage format ≠ cache query format Store as Parquet. Cache in Arrow-based layouts, enhanced with indexes and optimized for in-memory and SSD reads. 2️⃣ Hardware-native access patterns Use NVMe SSDs for random access instead of sequential scans. Read columns in parallel—SSDs love parallel reads. Latencies drop to microseconds. 3️⃣ Smart replication + predicate pushdown Cache any given record set 2–3x across the cluster for redundancy, not full duplication. Push query predicates to remote nodes instead of moving whole blocks.

Every query still has to decompress and decode pages. That work scales with the number of records per page—whether the data sits in memory, on an SSD, or in cloud storage.

Caching is supposed to make queries faster. But on lakehouses, simply moving Parquet files into memory or fast SSDs often barely moves the needle. The common assumption is that data access is network-bound. With modern cloud networks offering amazing throughput, the bottleneck often isn’t the network anymore. It’s the CPU.

That is closer to the AI experience we imagined before 2025. Very slick.

AWS EMR and GCP Dataproc both ship Hudi 1.x now. Most AI apps require four systems: Structured data → lakehouse Embeddings → vector DB Binary objects → object storage Semi-structured → document store Every write has to propagate across all four. Keeping them consistent is brutal. Hudi 1.2 collapses that. Native VECTOR, BLOB, and VARIANT types in one table. Vector search, binary storage, hierarchical queries—same transactional guarantees. The consistency problem disappears because there's nothing to sync.

Approaching now 10 yrs in production at some of the largest data lakes in the world. Collective accomplishments of the Apache Hudi community. 🫶

Lays out how it all connects

Full workflow: OneFlow ingests source data → Quanton curates training examples → Airflow coordinates the pipeline → provider trains → Lakegres serves context. onehouse.ai/blog/announcing-…

🔥 📣 Your lakehouse already has the data. Your training pipeline doesn't know how to use it. We're announcing AI training data pipelines on Onehouse for @baseten , @FireworksAI_HQ , and @togethercompute Training data curation today feels like carrying lake water in a leaky bucket. Most teams "version" training inputs by dumping JSONL files into a cloud folder. When a label changes or an evaluation surfaces a gap, there's no way to trace which examples came from where, or rebuild the dataset from updated source tables. Consider a support agent learning from past conversations. Useful examples require joining each conversation to its outcome: Was the issue resolved? Was the answer correct? Did someone intervene? The conversation alone isn't enough. The evidence lives in operational tables. When the model handles a class of requests poorly, the next training dataset needs to reflect updated labels, corrected data, and new selection logic. Teams need the selection query, the source-table versions it read, and the history of those corrections. 🔄 These are familiar data engineering requirements: joins, versioning, lineage, backfills. The lakehouse already handles them. Here's what we built: 1️⃣ Quanton curates examples from lakehouse tables using SQL selection logic—join conversations to resolution status, filter by outcome quality, batch vector search for good/bad examples at scale. It writes versioned exports as training datasets. 2️⃣ Onehouse submits the training job to Baseten, Fireworks, or Together AI and tracks status in the UI. The provider trains the model and returns the fine-tuned weights or adapter. 3️⃣ Lakegres serves point-in-time context during training or eval—queries lakehouse tables for the exact documents, records, or state needed to replay a past trace. The dataset remains reproducible. Engineers can inspect the selection query, trace examples to source records, and rebuild the dataset when labels change or evaluation surfaces gaps. Go to quanton.dev for a free account and give it a spin, let us know!

Companies spend billions on Spark compute every year. Most assume open-source accelerators can close the gap to commercial engines. We ran TPC-DS 10 TB across 5 engines on the same 11-node cluster to find out. OSS Spark: 12,200s (baseline) Comet (tuned): 9,122s (25% faster) Gluten (tuned): 8,563s (30% faster) Databricks Photon 18.2: 2,550s (5× faster) Quanton: 2,384s (5× faster) Open-source accelerators can get you partway there—under ideal conditions.

The 30x jump in open-model adoption this year hides a costly blind spot. As Meta, GLM, Kimi, Qwen, and Mistral models get more capable, more teams are customizing them. But SFT, preference tuning, and RFT aren't interchangeable—and the differences determine whether you spend $50 or $50,000 for the same result. 1️⃣ Supervised fine-tuning (SFT) You provide prompt/response pairs. The model learns by imitation. Think: prompt → ideal response 2️⃣ Preference tuning (DPO) For each prompt, provide multiple responses and mark which is better. The model learns from comparisons. Think: prompt → preferred response + rejected response 3️⃣ Reinforcement fine-tuning (RFT) The model produces an answer, a grader scores it, and that becomes the reward signal. The grader can verify correctness, run tests, or apply domain rules. Think: task → response → grader → reward 📊 Each approach demands different data. SFT needs accurate, representative prompt/response pairs with consistent formatting. Preference tuning needs the full comparison record: what outputs were judged, which won, and the context. RFT needs a representative task distribution, execution context, and clear lineage to the source. ⚠️ The bottleneck across all three: data management. Training data is a data engineering problem: 🔍 Curate datasets from production ✅ Validate before training 🔗 Track provenance for every example 📦 Version each dataset 🔄 Make every run reproducible The training API is the easy part. The leverage is in the data infrastructure underneath.

Turns out AWS pricing is negotiable if you blog about it. AWS S3 Tables launched at $5.04 to compact 100GB Iceberg . EMR: $0.17. 29× markup. When we published the teardown, AWS dropped the price by 10× right after. Then it was $0.52 for the same job. EMR still 3× cheaper. Who knew transparency was a feature request? AWS has so many amazing services. But, 2 years in, S3 Tables still ships with the same foundational limitations it had at launch. Sadly, still only seems to beat competition by calling customer execs with some FUD about risk and support gaps if they don't adopt the managed service. Price cuts are good. Price cuts after being caught aren't really a win. They just make you wonder what else sold this way.

@apachepolaris has done some good work here, by recognizing how a generic table abstraction at the catalog level is more extensible.