Chen Xing (@LynetteSohn) profile picture

Deep Learner @ScaleAI, previous research scientist @ Salesforce Research, spent my PhD in Google, Mila and Microsoft Research.

Palo Alto
Joined April 2020
Introducing MultiChallenge by @scale_AI - a new multi-turn conversation benchmark. Current frontier LLMs score under 50% accuracy (top: 44.93%). The new Gemini 2.0 Flash model launched today has also been included to our SEAL leaderboard. 📄 Paper: arxiv.org/abs/2501.17399 🏆Leaderboard: scale.com/leaderboard/multic…
3
5
1
26
15,501
🚀 Introducing the SEAL Leaderboards! We rank LLMs using private datasets that can’t be gamed. Vetted experts handle the ratings, and we share our methods in detail openly! Check out our leaderboards at scale.com/leaderboard! Which evals should we build next?
10
32
11
188
50,838
Excited to share our brand new LLM evaluation benchmark 🐠FoFo🐠 on format-following! 🐠FOFO🐠 is a pioneering benchmark for evaluating large language models’ (LLMs) ability to follow complex, domain-specific formats, a crucial yet under-examined capability for their application as AI agents. Link: arxiv.org/pdf/2402.18667.pdf Our evaluation across both open-source (e.g., Llama 2, WizardLM) and closed-source (e.g., GPT-4, PALM2, Gemini) LLMs highlights three key findings: 1. open-source models significantly lag behind closed-source ones in format adherence; 2. LLMs’ format-following performance is independent of their content generation quality; 3. LLMs’ format proficiency varies across different domains. These observations suggest two key points: i) The format-following capacity of LLMs appears independent of their content-following capacity shown in AlpacaEval and MT-Bench, and may necessitate specialized alignment fine-tuning beyond the conventional instruction-tuning of open source LLMs. ii) Format-following capacity is not universally transferable across domains, highlighting the potential utility of our benchmark as a guiding and probing tool for selecting domain-specific AI agent foundation models.
3
15
4
91
11,618
Interesting model: github.com/OpenLemur/Lemur
2
21
1
119
37,207
Code LLaMA is finally here. Congrats to @MetaAI @ylecun @syhw, etc.. And we'd like to introduce the finetuned llama2-70B model -- 🐒Lemur-70B🐒 again😉, a complement to Code LLaMA (7B, 13B, 34B) and maintains strong performance in text tasks. Code: github.com/OpenLemur/Lemur
2
16
1
76
9,264
Chen Xing retweeted
🧵Lemur-70B-chat stands out as the top-performing open-source LLM, rivaling ChatGPT across a broader spectrum of tasks when compared to other available open-source LLMs.
Replying to @XLangNLP
4/6 Lemur-chat significantly outperforms other open-source supervised fine-tuned models across various dimensions.
5
21
2,203
Glad to be one of the lemurs! @SFResearch
1/6 Open LLMs have traditionally been tailored for either 📚text or 💻code, with limited ability to effectively balance both. 🚀 Introducing #Lemur70B! 🚀: the SOTA open LLM balancing 📚text & 💻code capabilities 🤗Model: huggingface.co/OpenLemur 📖Blog: xlang.ai/blog/openlemur
4
230
1/6 Open LLMs have traditionally been tailored for either 📚text or 💻code, with limited ability to effectively balance both. 🚀 Introducing #Lemur70B! 🚀: the SOTA open LLM balancing 📚text & 💻code capabilities 🤗Model: huggingface.co/OpenLemur 📖Blog: xlang.ai/blog/openlemur
2
33
12
71
25,892
I can't imagine being a woman in CS seeing this week's news. I feel shocked and devastated. Foremost for the victims. But also for all the other women who have to put their guard up higher, doubting the authenticity of the professional attention they receive. You deserve better.
3
17
2
334
The worst ever feature introduced online is the automatic redirection to localized websites. How can I convince the world that I don’t want them to turn into Finnish?
3
1
10
Chen Xing retweeted
Couldn’t let a really good meme go waste, so fixed it.
4
42
3
335
Chen Xing retweeted
What is starting to happen to me during quarantine because of no haircuts.
This sheep escaped a farm and spent 6 years in the mountains, during which time he grew 60 pounds of wool. Wolves tried to eat him, but their teeth could not penetrate the floof. You don't have to turn hard to survive the wolves, just be really, really soft and fluffy.
7
7
1
117
Chen Xing retweeted
❤️ For the canceled weddings, separated lovers, new loves found, and loved ones near and far. This is Elgar's "Salut d'Amour." #songsofcomfort ❤️
314
5,231
756
18,199