New paper from @RekaAILabs 🔥 (yes an actual paper). This time we're releasing part of our internal evals which we call Vibe-Eval 😃 This comprises of a hard set which imo is pretty challenging for frontier models today. The fun part here is that we constructed it by trying to wreck our own model (e.g., Reka Core). 😈 However, it turns out on >50% of them, none of the frontier models can do them anyways. Making hard prompts has a lot of challenges that people often don't talk too much about. We discuss some of them in the paper. Notably, because our benchmark is hard, we are able to also induce further separability across frontier models. We benchmark 13 representative multimodal models and share some findings and insights. 🔹 Gemini Pro 1.5 and GPT-4V are roughly equally capable. Gemini 1.5 is really really strong and surpassed our expectations from Gemini 1.0. 🔹 Claude-3 and Reka models are a slight tier behind but substantially much better than any OSS model. Reka Core and Flash are roughly better than Claude-3 on normal tasks. However, on hard tasks they are nerf-ed due to our data collection process. More details in the paper (link below). 👇