AI evaluation platform
San Francisco, CA
Joined January 2023
- Tweets62
- Following7
- Followers179
- Likes63
Pinned Tweet
We've teamed up with @AiEleuther to make it super easy to visualize your evaluation results in Zeno!
Try it out the next time you run a benchmark: github.com/EleutherAI/lm-eva…
Zeno retweeted
@a13xba will give a presentation about @try_zeno, an interactive AI evaluation platform for exploring, debugging, and sharing how your AI systems perform. (co-founded with @a_a_cabrera)
xitter.cf/CarnegieMellon/status/…
An @SCSatCMU team has released a new interactive platform for data management and machine learning (ML) evaluation called Zeno. It empowers users to explore, visualize and analyze data and ML model performance across custom use cases.
cmu.is/zeno
ALT Zeno interface.
We just sent out the first issue of Zeno's Notes, our newsletter on AI evaluation.
In case you're not on the recipient list yet, read it here: zenoml.com/blog/newsletter-2…
Zeno retweeted
Lots of predictions of synthetic data for AI being big this year. Decided to look at the OG Alpaca dataset:
hub.zenoml.com/project/f192e…
Impressive for being GPT-4 generated w/ 1 prompt, but begs the question of how to generate diverse, OOD data
Zeno retweeted
Researchers from @CarnegieMellon, BerriAI explore the translation capabilities of Google’s Gemini and suggest Gemini Pro could be a valuable tool for MT.
@SNAT02792153 @yu_zichun52802 @AashiqMuhamed @tianyue_01 @a13xba @a_a_cabrera @krrish_dh @XiongChenyan
slator.com/is-google-gemini-…
Zeno retweeted
In case you missed it over the break - you can now visualize the outputs of any Eleuther LM Eval Harness run in @try_zeno with one command!
𝚙𝚢𝚝𝚑𝚘𝚗 𝚜𝚌𝚛𝚒𝚙𝚝𝚜/𝚣𝚎𝚗𝚘_𝚟𝚒𝚜𝚞𝚊𝚕𝚒𝚣𝚎.𝚙𝚢
We've teamed up with @AiEleuther to make it super easy to visualize your evaluation results in Zeno!
Try it out the next time you run a benchmark: github.com/EleutherAI/lm-eva…
Zeno retweeted
Google’s Gemini recently made waves as a major competitor to OpenAI’s GPT. Exciting! But we wondered:
How good is Gemini really?
At CMU, we performed an impartial, in-depth, and reproducible study comparing Gemini, GPT, and Mixtral.
Paper: arxiv.org/abs/2312.11444
🧵
Zeno retweeted
Since the initial release, we have significantly improved the usability of WebArena, accuracy of the evaluation, and provided interactive result analysis with @try_zeno
I am attending #NeurIPS2023 , say hi 👋 if you are interested in AI agent, code gen and their evaluations!
Zeno retweeted
Since some of you might be wondering whether Mamba 2.8B can serve as a drop-in replacement of some of the larger models, we've compared the Mamba model family to some of the most popular 7B models in @try_zeno
Report: hub.zenoml.com/report/2443/M…
🧵 1/5
Quadratic attention has been indispensable for information-dense modalities such as language... until now.
Announcing Mamba: a new SSM arch. that has linear-time scaling, ultra long context, and most importantly--outperforms Transformers everywhere we've tried.
With @tri_dao 1/
Zeno retweeted
Recently there were some great results from the new Mamba architecture (arxiv.org/abs/2312.00752) by @_albertgu and @tri_dao. We did a bit of third-party validation, and
1. The results are reproducible
2. Mamba 2.8B is competitive w/ some 7B models (!)
3. Mistral is still strong
Since some of you might be wondering whether Mamba 2.8B can serve as a drop-in replacement of some of the larger models, we've compared the Mamba model family to some of the most popular 7B models in @try_zeno
Report: hub.zenoml.com/report/2443/M…
🧵 1/5
Zeno retweeted
Google just released 𝑮𝒆𝒎𝒊𝒏𝒊, their long-awaited GPT-4 competitor.
Their report shows comparison across multiple common benchmarks, but 𝐡𝐨𝐰 𝐫𝐞𝐥𝐢𝐚𝐛𝐥𝐞 𝐚𝐫𝐞 𝐭𝐡𝐞𝐬𝐞 𝐫𝐞𝐬𝐮𝐥𝐭𝐬?
🧵 on potential issues with the benchmark scores
Zeno retweeted
Awesome blogpost by our friends @huggingface and @AiEleuther and a demonstration of how @try_zeno can be used to systematically spot issues with benchmark results. Give it a read!
Also:
Zeno Report: hub.zenoml.com/report/1255/D…
Zeno Project: hub.zenoml.com/project/2f5de…
⚠️ We are removing DROP from the Open LLM Leaderboard!
With leaderboard evaluation data openly shared on 2000+ models, we did a deep dive with our friends @AiEleuther and @try_zeno, & found out that its original implementation is unfair to many models 😱
huggingface.co/blog/leaderbo…
Zeno retweeted
Lots of info here on the several problems benchmarks can face
We loved collaborating with the @huggingface and @AiEleuther teams to investigate the odd behavior on the DROP benchmark! Check out the blog post and supporting Zeno report & project:
Report: hub.zenoml.com/report/1255/D…
Project: hub.zenoml.com/project/2f5de…
Zeno retweeted
We loved collaborating with the @huggingface and @AiEleuther teams to investigate the odd behavior on the DROP benchmark! Check out the blog post and supporting Zeno report & project:
Report: hub.zenoml.com/report/1255/D…
Project: hub.zenoml.com/project/2f5de…
⚠️ We are removing DROP from the Open LLM Leaderboard!
With leaderboard evaluation data openly shared on 2000+ models, we did a deep dive with our friends @AiEleuther and @try_zeno, & found out that its original implementation is unfair to many models 😱
huggingface.co/blog/leaderbo…
Zeno retweeted
⚠️ We are removing DROP from the Open LLM Leaderboard!
With leaderboard evaluation data openly shared on 2000+ models, we did a deep dive with our friends @AiEleuther and @try_zeno, & found out that its original implementation is unfair to many models 😱
huggingface.co/blog/leaderbo…
Zeno retweeted
𝐂𝐚𝐧 𝐀𝐈 𝐝𝐨 𝐲𝐨𝐮𝐫 𝐭𝐚𝐱𝐞𝐬? Probably not 💸
Quick @try_zeno report on @danielgross' benchmark of tax Qs. LLMs struggle with any math, and it's hard to validate text answers without external references
Report: hub.zenoml.com/report/1717/C…
Explore: hub.zenoml.com/project/e7519…
This quoted post is unavailable.
Happy Thanksgiving!
If you don't have one already, there's some turkey on Zeno: hub.zenoml.com/project/d7fdd…
Zeno now supports 3D 🧊 data!
We've uploaded over 1M @allen_ai ObjaverseXL models to a Zeno project to showcase how you can explore 3D data in a Zeno Project: hub.zenoml.com/project/d7fdd…
Zeno now supports 3D 🧊 data!
We've uploaded over 1M @allen_ai ObjaverseXL models to a Zeno project to showcase how you can explore 3D data in a Zeno Project: hub.zenoml.com/project/d7fdd…