What does the data architecture for production AI agents actually look like?
Join MinIO and @lakeFS to dive into agent memory, data isolation, reproducibility, and governance + how to build those capabilities into the infrastructure layer without slowing development down.
๐
10/22 | 12 PM ET
๐: bit.ly/3WbJB15
๐ฆDVC retweeted
Every ML failure I have debugged traces back to unversioned data.
Code has Git.
ML data has DVC, lakeFS, Delta Lake, and Iceberg.
Most teams use none of them.
Here is the data versioning landscape and how to pick,
grouped into 3 capability areas:
๐๐ก๐ฒ ๐๐๐ซ๐ฌ๐ข๐จ๐ง ๐๐จ๐ฎ๐ซ ๐๐๐ญ๐
โ Reproduce a model months later without guessing
โก Debug regressions: was it the code or the data
โข A/B test with different data snapshots
โฃ Roll back bad training data before it hits production
๐๐จ๐จ๐ฅ ๐๐จ๐ฆ๐ฉ๐๐ซ๐ข๐ฌ๐จ๐ง
โค DVC: Git-like, works with any storage, best for file-level ML datasets
โฅ LakeFS: Git for object storage, branch/merge on S3, best for lakehouse workflows
โฆ Delta Lake: table-level time travel + versioning, best for Databricks/lakehouse
โง Iceberg: open table format with snapshots, best for multi-engine environments
๐๐ก๐ข๐๐ก ๐ญ๐จ ๐๐ข๐๐ค
โจ File-based ML datasets (images, audio, docs): DVC
โฉ Lakehouse-native workflows on S3: LakeFS or Iceberg
โช Databricks or Delta-committed shops: Delta Lake native
โซ Snowflake shops adopting open formats: Iceberg via Polaris
Unversioned data is the #1 hidden cause of "the model was working last week."
Real senior data engineers make data versioning the default,
Not an afterthought triggered by a production incident.
The gap between "our model is broken and we do not know why"
And "we rolled back to yesterday's data snapshot in 10 minutes"
Is where data versioning earns its keep.
Which tool are you using today? ๐
Follow for more.
If you are looking for a Data Engineering Roadmap, Resume Template or Interview questions with answers do check out my Product โ lnkd.in/gpcJ5rxj
hashtag#DataEngineering hashtag#ML hashtag#MLOps hashtag#Reproducibility
๐ฆDVC retweeted
DevOps to MLOPS ๐
How is MLOPS different from usual DevOps workflows?
MLOps is different from the usual DevOps workflows, just like machine learning is different from traditional software development.
For example,
In MLOps, you need to track three components: code, data, and models
While in DevOps, you only track code.
Additionally, MLOps introduces the following.
- ๐๐ฎ๐๐ฎ ๐๐ฒ๐ฟ๐๐ถ๐ผ๐ป๐ถ๐ป๐ด: To track changes in datasets using tools like DVC (Data Version Control) or LakeFS
- ๐๐
๐ฝ๐ฒ๐ฟ๐ถ๐บ๐ฒ๐ป๐ ๐๐ฟ๐ฎ๐ฐ๐ธ๐ถ๐ป๐ด: To log hyperparameters, metrics, and model versions for each training run
- ๐ ๐ผ๐ฑ๐ฒ๐น ๐ฟ๐ฒ๐ด๐ถ๐๐๐ฟ๐: To version and manage models separately from application artifacts
- ๐๐ผ๐ป๐๐ถ๐ป๐๐ผ๐๐ ๐ง๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด (๐๐ง): Beyond CI/CD, models need regular retraining as data changes
In the latest newsletter edition,
We break down everything about what MLOps means for DevOps engineers. It covers,
- What is MLOPS
- Skills Required For DevOps Engineers
- Key MLOPS Tools
- Do you need GPU-based clusters to learn MLOps?
So if you are ready to go โall inโ with MLOPS, this edition is for you.
๐ฅ๐ฒ๐ฎ๐ฑ ๐ถ๐ ๐ต๐ฒ๐ฟ๐ฒ: newsletter.devopscube.com/p/โฆ
Are you working on any MLOps projects right now?
Or are you planning to upskill in MLOps?
Either way, share your thoughts in the comments.
#devops #MLOPS
๐ฆDVC retweeted
You can now use @huggingface Buckets as a @DVCorg remote.
(DVC is a "git for data" tool, i.e. version datasets & models without stuffing large files into git.)
Buckets speak S3 now โ DVC points right at one: pointers in git, data in your bucket, no egress on pulls.
๐ฆDVC retweeted
We're co-hosting a webinar with @lakeFS to showcase how CytoReason are using @nextflowio, Seqera, and lakeFS to ship reproducible disease models as governed, versioned data products.
Register now: hubs.la/Q04lpHJk0
๐ฆDVC retweeted
"The manual model breaks": What happens when agents write to production data thenewstack.io/lakefs-agentiโฆ
๐ฆDVC retweeted
๐งตUnder the hood DataChain combines power of warehouses with distributed clusters with proper data access patterns to process millions of video, images audio files:
๐ฆDVC retweeted
how do you manage unstructured data? what issues, struggles do you have? I would be happy to discuss and learn!
๐ฆDVC retweeted
A small DataChain video on processing audio data from @huggingface with ๐ค models. We need more tools to do ETLs, analytics, governance, preparation for unstructured data at scale!
- stream files from tar or wds archives! ๐คฏ
- enrich, prepare, version, publish datasets ๐
- bonus! ๐ค is natively integrated like a storage provider!
๐ฆDVC retweeted
What is the trick to make progress bars look decent on Colab? Anyone, from the top of your head? :)
๐ฆDVC retweeted
@DVCorg : Git-like version control system for data, ensuring effective management of machine learning projects.
github.com/iterative/dvc
๐ฆDVC retweeted

elvis@omarsar05 Nov 2024
Caribbean!
CaribbeanConnected via Europe App StoreLocation may be affected by a proxy or VPN.Account-level information, not a live location or per-post device.
๐ฆDVC retweeted

elvis@omarsar05 Nov 2024
Caribbean!
CaribbeanConnected via Europe App StoreLocation may be affected by a proxy or VPN.Account-level information, not a live location or per-post device.
DataChain is a modern Pythonic data-frame library to efficiently organize unstructured data.
I haven't tested but it looks really interesting especially because it supports multimodal data and cares about efficiency.
๐ฆDVC retweeted
After trending in Hacker News, our open-source is now trending in GitHub.
Whatโs next - Netflix special?
github.com/iterative/datachaโฆ
๐ฆDVC retweeted
Datasets + LLMs + Pydantic = DataChain
...now with @huggingface !๐
DataChain by @DVCorg just added @huggingface support ! Create, Load, Transform HF Datasets with LLMs easily.
- Pydantic for dataset schema
- Use your own or public HF Datasets
- Run your own or public HF Models
๐ฆDVC retweeted
DVC is trending on Hacker News! news.ycombinator.com
Why does this always happen on weekends? ๐ค
๐ฆDVC retweeted
Thank you to everyone who joined us for the OSPO #InnovateWeek panel on Open Source and AI last week! Special thanks to our panelists: Paige Bailey @DynamicWebPaige, Will Falcon @williamfalcon, Dmitry Petrov @FullStackML, and moderator Nhi Le @ngocnhile.
๐ฆDVC retweeted
California is winning over regulations ๐
Thanks to all who helped!
Thank you Governor @GavinNewsom for vetoing SB-1047 -- your pro-innovation leadership is much appreciated!
And to the many people who've been pushing back on SB-1047, a huge thank you as well. Congratulations to all -- we won! ๐
Looking ahead, lets keep on protecting AI open-source and innovation. We must make sure AI policy is based on science, not science fiction!
๐ฆDVC retweeted
Gave a talk yesterda at @UWMadison where @KyleCranmer and team are building a great OSS center to get more students building opensource โก๏ธ โก๏ธ
Bonus โก๏ธ, got to hang with @DynamicWebPaige from the Gemini team, Dmitry from DVC and @alpha_fund
๐ฆDVC retweeted
๐ป Really enjoyed talking about open-source and its impact on AI last night at the Data Science Institute in @UWMadison!
Panel session and lightning demos with @williamfalcon (@LightningAI) and Dmitry Petrov (@iterativeai), hosted by @KyleCranmer and Nhi Lรช (Alpha Intelligence Capital).
The conversations immediately afterward with students and faculty were inspiring โ universities need more startup incubators. ๐ โ at Wisconsin, USA
Finally got to meet my fellow @RiceUniversity ๐ฆalum and open source enthusiast @DynamicWebPaige! The panel on open source, AI, and innovation was awesome! @datascience_uw
































