MIMESIS Paper: arxiv.org/abs/2610.09484 Dscout Ground Truth: dscout.com/lp/dscout-ground-…

We need humans, or human proxies, in reinforcement learning and evaluation loops now more than ever. Agents keep topping benchmarks, then stumbling in the real world. No surprise -- for one, there’s a level of scientific rigor needed to ensure benchmarks don’t test models with tasks they’ve seen in training data. For another, tasks and rubrics included in benchmarks are usually much simpler than what real users hold as good. Finally, tasks and answer keys are often wrong (about 6.5% of MMLU questions have errors, Terminal Bench was recently found to contain errors). How many legal documents can be reviewed without back-and-forth with a client? How many tax returns can be prepared without input from specialists, departments, and regulatory websites? To handle real-world workflows, agents need to learn the real-world messiness during training. That often means integrating humans into training and evaluation environments. But people don’t scale, so teams turn to LLMs to play the user. New research from Meta shows why that’s a problem. The MIMESIS paper measured how often the same GPT-5.5 agent succeeds: With real users: 63.6% success. With frontier-model playing real users: 82.4–84.4%. Too tidy, helpful, and nice. With pretrained Qwen3.5-4B/9B playing real users: Success drops by 16.0 and 14.9 points. Too obstructive. The authors trained simulators on how humans actually interact, closing the gap. Every trained simulator landed within 10.3 points of real users; every off-the-shelf model missed by at least 14.9. The best, MIMESIS-4B and 9B, came within 2.4 points. The key takeaway: trained simulators captured how people actually behave, their directness, emotions, and context, while exposing agents to more diverse interaction paths. The team then added a coaching loop, using the simulator’s private reasoning and next reaction as feedback for the agent. Agents trained this way outperformed those trained against GPT-5.5 across all nine unseen user simulators. Kudos to Hoang Phan and the MIMESIS team at @Meta. This is rigorous, practical work for the industry Our read: The next leap in agents comes from understanding how people really behave. They’re vague, impatient, and change their minds mid-sentence. That’s where we are spending time at Dscout, and we’re happy to help anyone experimenting here. What does your “user” look like when you test your agents? #AIAgents #HumanAI #UserResearch

A big shoutout to the researchers/builders/founders that joined our #SFTechWeek dinner to discuss how the next wave of RL & post-training will be driven by real users, in real context, in the loop. @siyan_zhao's work on on-policy self-distillation showed how agents can learn from their own failed trajectories. Now that idea is showing up in production. @perplexity_ai shared a great example last month: use real user sessions to see where an agent went wrong, turn that into better feedback, and train the system not to repeat the same mistake. In the same vein, where the agent was in fact successful, use those trajectories as demonstration data that gets imitated. As agents take on longer, messier, real-world work, the real-user feedback loop helps hone in on "Did the model get the answer right?", "Where did the agent get stuck?", "What did the user correct?" and "Which steps worked? Which failed?" That's exactly what we're helping unlock at @dscout. Thanks all -- to more such riffs.

Looking forward to making new friends at #config2024!

Validating makes it seem like you're already correct. It also signals a one-and-done approach to testing. Was the design validated or not? Instead, try using words will help you (and stakeholders) shift into a mindset of curiosity and open-mindedness. ⤵️ #UXResearch #UX

If you want to take your #usability testing up a notch, spend: ➖ less time on manual tasks ➕ more time gathering and discussing insights Our newest feature will help you do all of that and more. Learn more about the "more" here: bit.ly/44GRUk7 #UX

With some creativity, your #usability testing practice can be more efficient, innovative, and customer-centered. It might actually start to be more fun, too...bit.ly/3EnMlMW #UXR

Feeling analysis paralysis around the glut of tools, solutions, and vendors available for research? Use this list to narrow on the offering that best serves your organization's ongoing needs: bit.ly/44op48f #UX #UserResearch

Hey there👋 We're not frequent Tweeters (Xers?). But if you need: • The latest from our People Nerds community➡️join us on LinkedIn • Support➡️email [email protected] • Anything else➡️email [email protected] Don't be a stranger.

This checklist has it all: know what to do before, during, and after a usability testing session—including a template on hand to make everything easier 🗒️ bit.ly/3KBGk2B #UXResearch

B2B user research comes with its own unique set of challenges. Here are 4 you should be aware of 👉 bit.ly/3qoqDVA #UXResearch

Stakeholders don't always understand the value of the research you provide. Luckily, we've got some tips on how to create some research readout FOMO: bit.ly/3QyAoeK #UXDesign

Make sure every #usability test goes off without a hitch. We’ve built a robust checklist to ensure you’re prepared for everything prior, during, and after the test: bit.ly/3OJcdss #UX

#Usability tests are a powerful way to understand how users interact with a service or product. This guide shows you how to avoid common pitfalls: bit.ly/3OvlFQc #UXDesign

Once you've delivered your insights, don't just let them stagnate. Using ideation and co-creation can jumpstart them into meaningful action: bit.ly/4793jLX #UXR

Sometimes stakeholders don’t realize the real value of what you’re providing. Facilitating entertaining sessions can make all the difference: bit.ly/47j8DfW #UXR

B2B #UserResearch comes with a unique set of challenges. Here's how to help the process go smoother: bit.ly/3OIL2OB #UX

Unmoderated user tests are ideal for specific situations. Here's how to know when to use them to the greatest effect: bit.ly/459spZs #UXR

If you're crafting a new product for your #UXResearch business, you don't want it to fall flat. These ideas help you ask the right questions—and get on the right path: bit.ly/458XBYP #UXDesign