Building the future workforce of the AI economy šŸ¤

United States
Joined March 2014
Handshake retweeted
A user simulator isn’t useful just because it looks like a real user. AI agents should also succeed/fail with simulated users the same way they do with real ones. To fix this, we open-sourced CUE, a new simulator to help you build more ecologically valid multi-turn benchmarks.
āš ļø User simulators are increasingly used to evaluate AI agents in multi-turn interaction. But we find that looking like real users ≠ reproducing real evaluation outcomes. šŸ“¢ We introduce Calibrated User Embeddings (CUE) to bridge this gap across LLM simulators. 🧵 1/8
3
3
1
37
2,171
Handshake retweeted
Credit to @cgnorthcutt, @joinHandshake, & co-authors 🧵 Thread: xitter.cf/cgnorthcutt/status/210… šŸ“„ Paper: arxiv.org/abs/2609.28470 šŸ“Š Data: studentbench.org
Today we establish that AI is as good as expert human tutors for immediate GRE learning gains (p=.015, n=2,383 students, study conducted July-Sep 2026). One hour with an expert GRE tutor: $75. One equivalent hour with an AI tutor: 7 cents . 918x cheaper, equivalent learning (p=.044, n = 140). In 5 of 7 academic topics, the top AI tutor beat the expert human tutors on average. How we measured it: pre-test -> 1 hour with (AI or human or no) tutor -> post-test. There are no LLM judges. Learning gains are post-pre. Everything is real students. Why we did this: > 1 trillion USD is being spent per year on machines making machines better. StudentBench shows us how to shift these resources so machines also make humans better. Why I did this: I grew up in rural Kentucky. My dad, grandad, and great grandad were all mailmen. Getting into a good college changed my life. Now we have done a rigorous study to help AI labs evaluate LLMs and help show how bright AI can be for students of all incomes. I believe in good science that leads to the democratization of opportunity. The research paper behind StudentBench: arxiv.org/abs/2609.28470 Data, code, details: in this thread If you're wondering what I'm selling, sorry to disappoint: we released all study data for free on Hugging Face, and made studentbench.org free to use. Enjoy. lots more in the video / thread!!
2
4
2
32
8,018
Today we establish that AI is as good as expert human tutors for immediate GRE learning gains (p=.015, n=2,383 students, study conducted July-Sep 2026). One hour with an expert GRE tutor: $75. One equivalent hour with an AI tutor: 7 cents . 918x cheaper, equivalent learning (p=.044, n = 140). In 5 of 7 academic topics, the top AI tutor beat the expert human tutors on average. How we measured it: pre-test -> 1 hour with (AI or human or no) tutor -> post-test. There are no LLM judges. Learning gains are post-pre. Everything is real students. Why we did this: > 1 trillion USD is being spent per year on machines making machines better. StudentBench shows us how to shift these resources so machines also make humans better. Why I did this: I grew up in rural Kentucky. My dad, grandad, and great grandad were all mailmen. Getting into a good college changed my life. Now we have done a rigorous study to help AI labs evaluate LLMs and help show how bright AI can be for students of all incomes. I believe in good science that leads to the democratization of opportunity. The research paper behind StudentBench: arxiv.org/abs/2609.28470 Data, code, details: in this thread If you're wondering what I'm selling, sorry to disappoint: we released all study data for free on Hugging Face, and made studentbench.org free to use. Enjoy. lots more in the video / thread!!
44
125
49
918
78,401
Handshake retweeted
We audited thousands of agent rollouts in DeepSWE-1.1. Over 80% contained reasoning about an imagined grader. Yet no grader/verifier is mentioned in prompts nor accessible to the agents. Agents reasoned things like: "Let me look at the problem from the grader's perspective" and referred to "hidden tests", "test authors", and "the checker". We found this behavior across all six frontier models we analyzed, including recent models from OpenAI, Anthropic, Z ai, and Kimi. In 10-25% of cases, such reasoning pulled the agent's work away from the user's original specification (yet it often still earned full reward on the DeepSWE task). We call this speculative reward hacking where the agent's reasoning focuses on imagining a grader rather than what the user wants. Hui Wen & I published an article today with the problematic trajectories, quantitative findings, and a taxonomy of these reward hacking behaviors: joinhandshake.com/research/a… Our research community needs to do better. Better evaluations that penalize such reward hacking, and better model training that does not give rise to this grader obsession -- so that models focus instead on accomplishing what users actually want.
3
7
3
25
3,408
Handshake retweeted
How do we eval an agent’s ability to improve from interactive feedback? Today's evals are static, but in real work, professionals continually critique AI-generated artifacts and iteratively regenerate them. We introduce TAHI: a Test-time Adaptive agent framework through Human-agent Interaction, featuring research on different strategies for agents to adapt to user feedback (memory/skills, weight updates). TAHI's innovations include an evolving verifier+rubric module that translates raw human interaction signals into explicit training/evaluation criteria. Led by @ZhiruoW, our study evaluates how well agents adapt to user feedback in creative writing and data visualization tasks, using professionals sourced through Handshake’s talent network. Our team at @joinHandshake is excited to support research advancing how we evaluate human-AI collaboration!
Agents trained on population-scale data can do a lot of things. But ask a professional to stake their reputation on an AI-generated artifact? ā€œPretty goodā€ isn’t good enough. We introduce TAHI: a Test-time Adaptive agent framework through Human-agent Interaction. Featuring: šŸ“ˆ Test-time adaptation via context (memory, skills) and weight training ⚔ Efficient adaptation to individual expertise within tens of tasks āœ”ļø Creating comprehensive rubrics for ā€œnon-verifiableā€ tasks šŸ” Analysis of shared community guidelines vs. personalized tacit expertise
2
2
8
963
Kudos to @anishathalye and @jomulr for co-chairing the RL agentic benchmarks workshop track for the inaugural ACM CAIS conference this week. We presented two separate Handshake AI Research papers in: (1) AI agentic systems - first evaluation of grader frameworks, and (2) AI benchmarks - first investment banking benchmark. Their posters had big crowds all afternoon. Great job!
6
13
2,279
Demo gods were on my side for this guest lecture on AI Agent Security at @MIT_CSAIL: I was able to show a prompt injection attack against @AnthropicAI's Opus 4.6 model. Agent security is still an unsolved problem!
6
6
61
5,534
Handshake retweeted
This @joinHandshake event with @OpenAI was so energizing. Not surprisingly, when you give young people powerful tools, their creativity and ambition run wild. The @UCBerkeley students were incredible. With AI, your career will be more about showing than telling. Build something real, not just a pretty resume. This is just the start.
Students are learning to build with Codex, and building to learn. Here’s what @UCBerkeley students built at the Codex Creator Challenge with @joinHandshake.
1
11
5,592
Handshake retweeted
šŸ“£ Second speaker announcement for Acquired Unplugged: Garrett Lord, co-founder and CEO of Handshake, will join Ben Gilbert and David Rosenthal for a live conversation on building Handshake into the career network for the AI economy. Space is limited. Register here: luma.com/acq-workos
3
11
2,667
Handshake retweeted
I asked @GarrettLord how @joinHandshake came out of nowhere to become a top data labeling partner to the AI labs: "We started to see incredible demand for individuals with PhDs and Master's on the Handshake platform. People studying to be lawyers, doctors, consultants, and getting their Master's in Tax Accounting. All the data labeling companies were trying to recruit them. We saw this, and were also hearing from those same people that it was a frustrating experience. They weren't getting paid on time. They weren't getting trained properly. And they weren't being treated as experts in their domain. We realized we had an opportunity to go direct. And we could then pass along the customer acquisition costs as savings to the AI labs, leapfrogging others in the space. Human AI data labeling is a very operationally intensive business. The only durable advantage in this space is access to an audience. Otherwise, it's a commoditized set of companies competing with each other for margin. If you can build loyalty, improve retention, and treat these people the way they expect to be treated, you can pass along many benefits to the labs. The three things they care a bout is data quality, speed, and volume. And you can do this by building a much better product experience." From our conversation published in July, 2025. Full episode linked in the replies.
5
8
24
7,644
The week before ICLR, we're hosting a tight-knit AI research symposium in SF on the future of economically valuable AI agents w/ @TheAndiPenguin (co-founder, humans&) @patrick_tammer (AI strategy/operations lead, Google), Linda Lu (head of strategic initiatives, Berkeley RDI)
1
2
18
3,738
Handshake retweeted
Love that @mbdailyshow remembered using Handshake in college. So did millions of other people. The internet has been scraped. What moves models now is signal that has to be generated by real people in real time. How humans actually talk. How experts actually reason. How professionals actually evaluate work. That's why we're paying improv performers $74/hr. And paying doctors, nuclear technicians, farm supervisors, and accountants A LOT more than that. The models need what only people can generate.
5
2
12
4,239
Handshake retweeted
Someone on our subreddit just posted this. Paid off their student loans. $235K earned on HAI since July. Now they want to send their mom to Italy and start a college fund for their nephew. This is the part of building an AI company that never makes the headlines.
9
15
1
193
24,383
Each year the #UpfrontSummit brings together over 1,000 investors, entrepreneurs, and leadersĀ to showcase the best of innovation, technology and creativity. This year we were proud to see some of our founders grace the stage to share what they've learned on their journeys: - @JamieSiminoff, founder of @ring, reflected on grit, near-death moments, selling to Amazon, and why AI has made Ring feel like a startup again. – @hametwatt , founder of Feno, joined the conversation on building enduring companies in moments of technological shift. – And @GarrettLord, founder of @joinHandshake, shared his 12-year building journey, giving a shoutout to True partner @tonidotorg as first to believe in him when he was still sleeping in college parking lots trying to close deals. These are the stories that matter: conviction, resilience, reinvention. Proud to back founders building through cycles — and building for the long term. #UpfrontSummit2026 #FounderLed #FrontierTech #AIBuilders #AIEntrepreneurs #GenZ
1
2
3
1,926
Handshake retweeted
Big news: Taro is joining @joinHandshake AI šŸŽ‰ For the past 4 years, we’ve been singularly focused on helping software engineers achieve career growth. As AI reshapes the role of engineers in fundamental ways, the trend is clear: adapt with AI or get left behind.
2
3
8
1,853
Handshake retweeted
Huge moment for Handshake AI. Excited to welcome @rpandey1234 , @AlexChiou , and the @join_taro (YC S22) community to Handshake AI. 175,000+ engineers across 300+ top tech companies. That's the kind of community you want building the future of AI. We’re launching the Software Engineer Talent Network - connecting vetted engineers directly to leading labs for paid, flexible work on model eval, code generation, and alignment. Looking for 10,000+ engineers. First 500 accepted get a $150 onboarding bonus. Apply: joinhandshake.com/ai/mkt/swe
8
7
3
100
198,995
News: @joinHandshake acquires @CleanlabAI! This "ten-year old job marketplace" has quietly become a top human data lab for AI--building an AI research org, acquiring top AI talent, and advancing Cleanlab tech and research to lead data foundations for frontier AI. 1 of 4
2
4
3
22
4,434
Handshake retweeted
Big news: Handshake has acquired @CleanlabAI. šŸš€ The world-class research team has earned top recognition in the field, and brings additional academic pedigree, technical credibility, and cultural alignment to the @joinHandshake team. techcrunch.com/2026/01/28/ai…
1
12
1
63
197,558
TITV from Davos: The World Economic Forum (@wef) Our CEO & Editor-in-Chief @Jessicalessin will be interviewing: -@btaylor (Sierra) -@StevenBartlett (FlightStory & Diary of a CEO) -@zackbogue (DCVC) -@GarrettLord (Handshake) Tune in tomorrow at 10 am PT / 1pm ET on X or thein.fo/3LVSKGp #wef26
1
2
6
3,362
Career Readiness took center stage at the Student Academic Success Winter Retreat. We shared strategies to embed career development across #Marist. Thank you to Christine Cruzvergara from @joinHandshake for an inspiring fireside chat on career services best practices.
2
4
810