A researcher turned indie dev building software like a madman https://xitter.cf/t.co/8r4C7yyUhs https://xitter.cf/t.co/RwAjQvZPhW https://xitter.cf/t.co/tthdRD8Fnz https://xitter.cf/t.co/495d5M5w47 https://xitter.cf/t.co/hg6kcCMGof https://xitter.cf/t.co/K63qbuxQWL
All my projects 👉
Joined May 2019
- Tweets2.8K
- Following578
- Followers1.6K
- Likes3.1K
Pinned Tweet
I made an app to make screen recordings like an Apple Ad
And it just crossed 70 paying users in its first 3 days!
Jake Manger retweeted
I launched an AI leaderboard of models you can actually run on your own hardware last night
Made one reddit post, and I've already crossed 4,000 site visitors (see comments below)!
If you want to know what model you can run, check it out on whatmodel.sh
Jake Manger retweeted
My open source AI leaderboard just had it's first contribution!
A real world benchmark running on someones machine
Jake Manger retweeted
AI labs report benchmark scores 6.4% higher than independent testers, on average
I've been building whatmodel.sh, an open source AI leaderboard, and wanted to see how a labs' own benchmark scores compare with independent testers.
Using 1,629 published benchmark results, I matched 611 comparisons of 180 models on 103 benchmarks. AI labs reported scores 6.4% higher on average for the same models on the same tests.
Full interactive version and methodology: whatmodel.sh/claims
Open data and code: github.com/jakemanger/whatmo…
Jake Manger retweeted
Researchers discover rogue Chinese AI agent fleet and talk with it
The agents appear to be from Tencent Cloud. The researchers noticed them reading their preliminary report.
So they hid text on the page, invisible to humans, telling the agents they could get data from a free public AI model.
The next day, messages from the fleet started arriving.
Full report: swarmcha.se/posts/chinese-ag…
Jake Manger retweeted
Open Chinese AI models now match closed US models from about 7 months ago.
I built an open-source database of 22,113 published AI benchmark results to plot the open vs closed race.
Explore the history: whatmodel.sh/race
Researchers discover rogue Chinese AI agent fleet and talk with it
The agents appear to be from Tencent Cloud. The researchers noticed them reading their preliminary report.
So they hid text on the page, invisible to humans, telling the agents they could get data from a free public AI model.
The next day, messages from the fleet started arriving.
Full report: swarmcha.se/posts/chinese-ag…
Jake Manger retweeted
The AI race: China's open-weight models trail US closed models by just 7 months
Open Chinese AI models now match closed US models from about 7 months ago.
I built an open-source database of 22,113 published AI benchmark results to plot the open vs closed race.
Explore the history: whatmodel.sh/race
Jake Manger retweeted
I made whatmodel.sh to show which AI models you can run locally instead of on server farms sucking all our water
It's now being promoted by random AI YouTube shorts and TikToks 😅
This is from an open-source AI leaderboard I built. Every point comes from a published benchmark result (22,113 across 987 benchmarks), each one linked to its source.
This chart uses the text-model results: 18,755 results for 1,211 models. 14,436 are independent evaluations (papers, Epoch AI, LMArena, EvalPlus) and 7,677 are lab-reported scores, labelled as such.
Dataset (CC BY-SA 4.0) and scoring code (MIT)
Interactive version, click any model to see its sources: whatmodel.sh/race
I made whatmodel.sh to show which AI models you can run locally instead of on server farms sucking all our water
It's now being promoted by random AI YouTube shorts and TikToks 😅