Open source tool for data, models, & experiment versioning for ML projects. Join our stellar community https://xitter.cf/t.co/RTCIKrZlmf for help, support and insights.

San Francisco, CA
Joined May 2018
๐Ÿฆ‰DVC retweeted
What does the data architecture for production AI agents actually look like? Join MinIO and @lakeFS to dive into agent memory, data isolation, reproducibility, and governance + how to build those capabilities into the infrastructure layer without slowing development down. ๐Ÿ“… 10/22 | 12 PM ET ๐Ÿ”—: bit.ly/3WbJB15
1
1
187
Every ML failure I have debugged traces back to unversioned data. Code has Git. ML data has DVC, lakeFS, Delta Lake, and Iceberg. Most teams use none of them. Here is the data versioning landscape and how to pick, grouped into 3 capability areas: ๐–๐ก๐ฒ ๐•๐ž๐ซ๐ฌ๐ข๐จ๐ง ๐˜๐จ๐ฎ๐ซ ๐ƒ๐š๐ญ๐š โ‘  Reproduce a model months later without guessing โ‘ก Debug regressions: was it the code or the data โ‘ข A/B test with different data snapshots โ‘ฃ Roll back bad training data before it hits production ๐“๐จ๐จ๐ฅ ๐‚๐จ๐ฆ๐ฉ๐š๐ซ๐ข๐ฌ๐จ๐ง โ‘ค DVC: Git-like, works with any storage, best for file-level ML datasets โ‘ฅ LakeFS: Git for object storage, branch/merge on S3, best for lakehouse workflows โ‘ฆ Delta Lake: table-level time travel + versioning, best for Databricks/lakehouse โ‘ง Iceberg: open table format with snapshots, best for multi-engine environments ๐–๐ก๐ข๐œ๐ก ๐ญ๐จ ๐๐ข๐œ๐ค โ‘จ File-based ML datasets (images, audio, docs): DVC โ‘ฉ Lakehouse-native workflows on S3: LakeFS or Iceberg โ‘ช Databricks or Delta-committed shops: Delta Lake native โ‘ซ Snowflake shops adopting open formats: Iceberg via Polaris Unversioned data is the #1 hidden cause of "the model was working last week." Real senior data engineers make data versioning the default, Not an afterthought triggered by a production incident. The gap between "our model is broken and we do not know why" And "we rolled back to yesterday's data snapshot in 10 minutes" Is where data versioning earns its keep. Which tool are you using today? ๐Ÿ‘‡ Follow for more. If you are looking for a Data Engineering Roadmap, Resume Template or Interview questions with answers do check out my Product โ€” lnkd.in/gpcJ5rxj hashtag#DataEngineering hashtag#ML hashtag#MLOps hashtag#Reproducibility
1
3
241
๐Ÿฆ‰DVC retweeted
DevOps to MLOPS ๐Ÿš€ How is MLOPS different from usual DevOps workflows? MLOps is different from the usual DevOps workflows, just like machine learning is different from traditional software development. For example, In MLOps, you need to track three components: code, data, and models While in DevOps, you only track code. Additionally, MLOps introduces the following. - ๐——๐—ฎ๐˜๐—ฎ ๐˜ƒ๐—ฒ๐—ฟ๐˜€๐—ถ๐—ผ๐—ป๐—ถ๐—ป๐—ด: To track changes in datasets using tools like DVC (Data Version Control) or LakeFS - ๐—˜๐˜…๐—ฝ๐—ฒ๐—ฟ๐—ถ๐—บ๐—ฒ๐—ป๐˜ ๐˜๐—ฟ๐—ฎ๐—ฐ๐—ธ๐—ถ๐—ป๐—ด: To log hyperparameters, metrics, and model versions for each training run - ๐— ๐—ผ๐—ฑ๐—ฒ๐—น ๐—ฟ๐—ฒ๐—ด๐—ถ๐˜€๐˜๐—ฟ๐˜†: To version and manage models separately from application artifacts - ๐—–๐—ผ๐—ป๐˜๐—ถ๐—ป๐˜‚๐—ผ๐˜‚๐˜€ ๐—ง๐—ฟ๐—ฎ๐—ถ๐—ป๐—ถ๐—ป๐—ด (๐—–๐—ง): Beyond CI/CD, models need regular retraining as data changes In the latest newsletter edition, We break down everything about what MLOps means for DevOps engineers. It covers, - What is MLOPS - Skills Required For DevOps Engineers - Key MLOPS Tools - Do you need GPU-based clusters to learn MLOps? So if you are ready to go โ€œall inโ€ with MLOPS, this edition is for you. ๐—ฅ๐—ฒ๐—ฎ๐—ฑ ๐—ถ๐˜ ๐—ต๐—ฒ๐—ฟ๐—ฒ: newsletter.devopscube.com/p/โ€ฆ Are you working on any MLOps projects right now? Or are you planning to upskill in MLOps? Either way, share your thoughts in the comments. #devops #MLOPS
1
14
40
1,674
You can now use @huggingface Buckets as a @DVCorg remote. (DVC is a "git for data" tool, i.e. version datasets & models without stuffing large files into git.) Buckets speak S3 now โ†’ DVC points right at one: pointers in git, data in your bucket, no egress on pulls.
1
4
17
1,162
๐Ÿฆ‰DVC retweeted
๐ŸงตUnder the hood DataChain combines power of warehouses with distributed clusters with proper data access patterns to process millions of video, images audio files:
1
2
5
480
๐Ÿฆ‰DVC retweeted
how do you manage unstructured data? what issues, struggles do you have? I would be happy to discuss and learn!
1
210
๐Ÿฆ‰DVC retweeted
A small DataChain video on processing audio data from @huggingface with ๐Ÿค— models. We need more tools to do ETLs, analytics, governance, preparation for unstructured data at scale! - stream files from tar or wds archives! ๐Ÿคฏ - enrich, prepare, version, publish datasets ๐Ÿš€ - bonus! ๐Ÿค— is natively integrated like a storage provider!
1
3
1
11
1,262
๐Ÿฆ‰DVC retweeted
What is the trick to make progress bars look decent on Colab? Anyone, from the top of your head? :)
2
2
622
๐Ÿฆ‰DVC retweeted
elvis
@omarsar05 Nov 2024
4
19
3,840
๐Ÿฆ‰DVC retweeted
elvis
@omarsar05 Nov 2024
DataChain is a modern Pythonic data-frame library to efficiently organize unstructured data. I haven't tested but it looks really interesting especially because it supports multimodal data and cares about efficiency.
5
46
2
252
21,689
๐Ÿฆ‰DVC retweeted
After trending in Hacker News, our open-source is now trending in GitHub. Whatโ€™s next - Netflix special? github.com/iterative/datachaโ€ฆ
4
20
1,221
Datasets + LLMs + Pydantic = DataChain ...now with @huggingface !๐Ÿ’› DataChain by @DVCorg just added @huggingface support ! Create, Load, Transform HF Datasets with LLMs easily. - Pydantic for dataset schema - Use your own or public HF Datasets - Run your own or public HF Models
1
5
1
36
1,971
๐Ÿฆ‰DVC retweeted
DVC is trending on Hacker News! news.ycombinator.com Why does this always happen on weekends? ๐Ÿค”
3
11
708
Thank you to everyone who joined us for the OSPO #InnovateWeek panel on Open Source and AI last week! Special thanks to our panelists: Paige Bailey @DynamicWebPaige, Will Falcon @williamfalcon, Dmitry Petrov @FullStackML, and moderator Nhi Le @ngocnhile.
3
1
10
5,913
๐Ÿฆ‰DVC retweeted
California is winning over regulations ๐ŸŽ‰ Thanks to all who helped!
Thank you Governor @GavinNewsom for vetoing SB-1047 -- your pro-innovation leadership is much appreciated! And to the many people who've been pushing back on SB-1047, a huge thank you as well. Congratulations to all -- we won! ๐ŸŽ‰ Looking ahead, lets keep on protecting AI open-source and innovation. We must make sure AI policy is based on science, not science fiction!
1
5
430
Gave a talk yesterda at @UWMadison where @KyleCranmer and team are building a great OSS center to get more students building opensource โšก๏ธ โšก๏ธ Bonus โšก๏ธ, got to hang with @DynamicWebPaige from the Gemini team, Dmitry from DVC and @alpha_fund
2
4
25
2,963
๐Ÿ’ป Really enjoyed talking about open-source and its impact on AI last night at the Data Science Institute in @UWMadison! Panel session and lightning demos with @williamfalcon (@LightningAI) and Dmitry Petrov (@iterativeai), hosted by @KyleCranmer and Nhi Lรช (Alpha Intelligence Capital). The conversations immediately afterward with students and faculty were inspiring โ€“ universities need more startup incubators. ๐Ÿš€ โ€“ at Wisconsin, USA
Finally got to meet my fellow @RiceUniversity ๐Ÿฆ‰alum and open source enthusiast @DynamicWebPaige! The panel on open source, AI, and innovation was awesome! @datascience_uw
2
5
28
6,803