software engineer in NYC. prev https://xitter.cf/t.co/Bx70AilrPF and https://xitter.cf/t.co/ZTUpgwSh6a. Now @powersetres.
New York
Joined November 2007
- Tweets991
- Following1.2K
- Followers875
- Likes11.7K
on @PowersetRes we run an LLM embedding pipeline to categorize repos based on their READMEs and website content.
we use embedding similarity to a category centroid to determine membership- it works, but seems like a place Jev could really shine
could simplify a whole lot and probably be faster/cheaper
We replicated the @SemiAnalysis_ estimate that ~4% of public GitHub commits are being authored by Claude Code. Moreover, this figure is growing exponentially.
unexpected easy win for LLM generated code - quickly add support for many languages to your library... 🌏
trying out SiriusDB: sirius-db.com/ on @modal GPUs
have it building, next step running the tpch suite
did some tuning and can now download the last 5 years of public @github events in ~15m
3.5 TB of data (50k+ json.gz files) big fanout made super easy using @modal 🎉, lots of parallel downloads + retries with exponential back-off
blog post and open-source repo coming soon!
using @modal to download every public event on GitHub (from @githubarchive) in 2024 in about 10 minutes, across 170+ containers, spun up almost instantly is very cool...
stored into a @modal volume, taking up ~775 GiB across 11k+ gzipped json files
