@madiator

CEO @bespokelabsai. RL/Envs/Posttraining/Agents. Created Generative Retrieval at @GoogleDeepMind.

 Inside an RL Env
bespokelabs.ai
Joined February 2008
  • Tweets4.9K
  • Following1.6K
  • Followers17.5K
Good to see Bespoke Nimble at #3 on the API providers page of Jevbench, above Jev 1.13.0 and Jev 1.14.0. benchmarkheaven.com/jev-mode… It's available here: console.bespokelabs.ai/
1
16
959
Very cool work by Pref model! We have an internal QA system which has been finding issues as well, and we have submitted several fixes to Terminal Bench so that Terminal Bench continues to improve. This story is not just for Terminal Bench, wink wink.
Jennifer Zhou
@chem_safetyOct 19
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
44
2,735
Garden harvest. Lots and lots of tomatoes this time. The tomato plant needed a harness..
21
853
Replying to @madiator @bespokelabsai
Nimble-v4 is 9B and it beats several other models that are much larger.
3
8
516
Ah oh btw, it's also pretty darn good at Chess!
1
1
12
604
1
7
607
Hey nimble, please kill AI slop.
1
1
10
1,125
Hey nimble, make no mistakes.
2
10
1,007
Today, we are announcing Nimble-v4 and Nimble-v4-max which are pushing the frontier of decision models, outperforming Jev and other variants. On the Decision Index public scores, nimble-v4-max and nimble-v4 score +10 and +6 against Jev 1.13, making nimble-v4-max the #1 model. Beyond benchmarks, we have done a lot of internal evals, and see meaningful improvements: 43% fewer mistakes than Jev, 50pp improvement on model routing decisions, being great at coding use cases, fact checking etc. The models are available via our API service. Go build nimbly: console.bespokelabs.ai
23
39
12
242
82,790
Perplexity uses Nimble codebase in various ways, and no credit given. @AravSrinivas please fix :)
Antoine Chaffin
@antoine_chaffinOct 11
Congratulations on the release! Open source is a race to the top, btw :) Happy to see credit given, and hopefully all the various effort around open decider models converge towards insanely capable models! huggingface.co/vllm-sr/d3/bl…
2
6
96
10,831
Wrapped up COLM!
4
64
2,703
Load more