Working on making it easy to get trustworthy human-labeled data for ML @HelloSurgeAI
San Francisco, CA
Joined September 2009
- Tweets281
- Following2.1K
- Followers1.7K
- Likes1.6K
Andrew Mauboussin retweeted
"Prognosticative pastry." "A hound circling a tree, nose to bark."
These aren’t parodies - they’re actual quotes from SOTA models in response to creative writing prompts, and they’re winning leaderboards that are rewarding slop.
We’re introducing *Hemingway-bench*, a new AI writing leaderboard, to fix this:
surgehq.ai/leaderboard
surgehq.ai/blog/hemingway-be…
We designed Hemingway-bench to push frontier model writing toward genuine nuance and impact.
Instead of autograders and two-second vibe checks - both of which love fancy literary devices and dense formatting, over actual quality - we used expert human writers across a variety of fields to judge real-world writing tasks.
Why? I love writing. I love reading. Great science fiction is one of the things that's always inspired me. Even in terms of "enterprise value", so much of what we do in our day-to-day involves writing - we want crisp emails and insightful reports, not dry, verbose summaries.
Yeah, coding is important - but there's a reason I use CC-assisted apps, but still haven't read a full-fledged AI novel.
What did we find? Current leaderboards are easily hacked, and often negatively correlated with actual quality. If a model (over)uses all the stuff you learn about in school (metaphors in every sentence! transition words! complex, flowery phrases!), it ranks high on EQ-bench and LMArena.
But that’s not good writing that people actually want.
The winners of Hemingway-bench didn't sound like they were trying to win a poetry slam. Gemini 3 Flash, Pro, and Opus 4.5 took the top 3 spots because they had natural voices that didn't sound pretentious.
They were poetic and immersive, but in the right ways.
When they used wit, they didn't sound cringey and try-hard - they sounded like your naturally funny friend.
I'm waiting for the day AI wins a Pulitzer, and hopefully Hemingway-bench helps guide it on its way.
Check out the leaderboard and examples here: surgehq.ai/leaderboard
And our blog post describing it: surgehq.ai/blog/hemingway-be…
We’re founded by ML practitioners focused on producing the best data in the industry, so we do things a little differently. Instead of outsourcing to low wage countries, we hire our own workforce based in the US and write all of our own software for ensuring quality.
We partner with the top AI teams in the world on RLHF, evaluations, and more. One example project we work on is providing the fine-tuning data for Anthropic’s Claude: surgehq.ai/case-studies/anth…
Research papers using our data below (linked version at surgehq.ai/rlhf)
At @HelloSurgeAI, we are building the human data infrastructure to create useful + aligned AI.
We're in a period of unprecedented growth and are hiring across eng: full stack, infra, security, applied ml. SF/Remote. DM or email [email protected]
More details about us:
1. For supervised fine-tuning (training on prompt-completion pairs) quality is more important than quantity. The Meta team found that sticking to just 27k high-quality examples led to the best results. This matches the conclusions from the LIMA paper
I've done analyses in Jupyter notebook for years. Now Code Interpreter can write all the code, but I haven't used it daily because you can't tweak code, re-run analyses, install packages, etc.
I built Juno w/ @alexirobbins to bring GPT into Jupyter for the best of the worlds
Juno runs in your notebook context so you get working code out of the box. It can write, edit, and debug code. And it's a Jupyter extension so you pip install it and get started in <1 min: getjuno.ai
If you get a chance to try it out let me know what you think!
DALL·E 2 let anyone with an idea for an image to create it by just asking for it, no Photoshop required.
Data science could be done in the same way soon.
I created AutoPlot w/ @mihail_eric to explore this idea. It uses GPT-3 to do analysis and create charts on your command
It still makes occasional mistakes, but the technology is already good enough to be useful as a Copilot style tool. Removing the need to read through plotting library docs saves a lot of time.
Try it out here with public datasets, or upload your own: autoplot.app/
Andrew Mauboussin retweeted
I'm super excited to release AutoPlot, a tool @amaub and I have been working on to answer the question:
What if you could do any data analysis by just speaking in natural language?
No confusing scripts, Excel macros, or Matplotlib incantations.
Andrew Mauboussin retweeted
I've had a similar experience to Ethan. If you want to do an NLP data collection/labeling process and don't want/need to be managing annotators directly, Surge is remarkably easy to work with and their team does very good work.
The biggest game-changer for my research recently has been using @HelloSurgeAI for human data collection. With Surge, the workflow for collecting human data now looks closer to “launching a job on a cluster” which is wild to me. 🧵 of examples:
Andrew Mauboussin retweeted
The biggest game-changer for my research recently has been using @HelloSurgeAI for human data collection. With Surge, the workflow for collecting human data now looks closer to “launching a job on a cluster” which is wild to me. 🧵 of examples:
It's common wisdom that adding "Reddit" to Google searches gives better results, but is it actually true?
Yes! Our @HelloSurgeAI search quality evaluators found the top Reddit result was preferred to the top Google 30% of the time, but only shows up 3% of the time
Andrew Mauboussin retweeted
Describe your inverse scaling idea to @HelloSurgeAI over a call, & they'll turn your idea into a dataset. Rates are competitive w/ other data labeling platforms, but they handle all of the worker management/hiring, quality control, etc. Data quality is excellent in my experience
We're excited to partner with @EthanJPerez and team on the Inverse Scaling Prize!
Need help creating 300+ examples illustrating your inverse scaling task? We help AI companies create Alignment and LLM datasets, so we'll help you design a project on our data labeling platform.
Andrew Mauboussin retweeted
(1/) There’s a lot of talk about how bad Google has become. Ads out of control. SEO run amok.
So how does legacy Google compare to a modern search engine like Neeva?
We worked with @HelloSurgeAI on a scaled unbiased eval. The results will surprise you. Here’s what we found 🧵
Chain of thought prompting is one of the coolest ideas from recent LM research. The models can't solve an entire multi-step math problem in one pass. But if you just prompt it to come up with the first step of a solution, it can get that most of the time...
Language models are just trained to predict the next word on a massive text corpus. But researchers and API-consumers are discovering how to use next-word prediction for all sorts of other things, including these middle-school math problems
OpenAI created this math problem dataset with the @HelloSurgeAI workforce openai.com/blog/grade-school…
We have math teachers, engineers, and other domain-experts on staff to create and evaluate models. If you're also interested in exploring capabilities of new models we should chat!
