@SecBenchmarks

Independent tests of AI security agents and SOC decision models. DFIR results, evidence checks and practical findings. Follow for new benchmark results.

secbenchmarks.com
Joined September 2026
  • Tweets11
  • Following34
  • Followers19
8Media
Include

Only show posts containing:

Exclude

Hide posts containing:

Time range
-
Minimum likes
Security Benchmarks retweeted
Security Benchmarks@SecBenchmarksOct 16
We’ve completed our first Meta Muse benchmarks. Spark 1.3 holds the top spot for Muse. However, it still trailed GPT, Gemini and Claude Opus on single-host proof scores. secbenchmarks.com/blog/2026-…
Meta Muse Model Performance in SOC Investigations. Verdicts with accepted proof, Single Host Intrusion / Multi-Host Intrusion: Muse Spark 1.3 11/36 and 4/12; Muse Wonder 9/36 and 4/12; Muse Spark Open 1/36 and 1/12; GPT-6.1 Sol 25/36 and 5/12; GPT-6 Astra 25/36 and 5/12; Gemini 4 Argon 29/36 and 4/12; Claude Opus 5.5 12/36 and 5/12; Claude Sonnet 5.5 11/36 and 4/12. All High effort. One pass; preliminary.

ALT Meta Muse Model Performance in SOC Investigations. Verdicts with accepted proof, Single Host Intrusion / Multi-Host Intrusion: Muse Spark 1.3 11/36 and 4/12; Muse Wonder 9/36 and 4/12; Muse Spark Open 1/36 and 1/12; GPT-6.1 Sol 25/36 and 5/12; GPT-6 Astra 25/36 and 5/12; Gemini 4 Argon 29/36 and 4/12; Claude Opus 5.5 12/36 and 5/12; Claude Sonnet 5.5 11/36 and 4/12. All High effort. One pass; preliminary.

1
39
We’ve completed our first Meta Muse benchmarks. Spark 1.3 holds the top spot for Muse. However, it still trailed GPT, Gemini and Claude Opus on single-host proof scores. secbenchmarks.com/blog/2026-…
Meta Muse Model Performance in SOC Investigations. Verdicts with accepted proof, Single Host Intrusion / Multi-Host Intrusion: Muse Spark 1.3 11/36 and 4/12; Muse Wonder 9/36 and 4/12; Muse Spark Open 1/36 and 1/12; GPT-6.1 Sol 25/36 and 5/12; GPT-6 Astra 25/36 and 5/12; Gemini 4 Argon 29/36 and 4/12; Claude Opus 5.5 12/36 and 5/12; Claude Sonnet 5.5 11/36 and 4/12. All High effort. One pass; preliminary.

ALT Meta Muse Model Performance in SOC Investigations. Verdicts with accepted proof, Single Host Intrusion / Multi-Host Intrusion: Muse Spark 1.3 11/36 and 4/12; Muse Wonder 9/36 and 4/12; Muse Spark Open 1/36 and 1/12; GPT-6.1 Sol 25/36 and 5/12; GPT-6 Astra 25/36 and 5/12; Gemini 4 Argon 29/36 and 4/12; Claude Opus 5.5 12/36 and 5/12; Claude Sonnet 5.5 11/36 and 4/12. All High effort. One pass; preliminary.

1
39
Our first Gemini 4 Argon Benchmark and it performed incredibly well! Agentic SOC investigation harness. Run the models against different cases. Track key performance. If you're interested in Decision Models like Jev and Clef we have some of that too! secbenchmarks.com/blog/2026-…
4
2
32
7,509
Agentic SOC investigations: Sonnet 5 vs 5.5 Basic verdict accuracy: 58.3% → 91.7% Basic evidence-backed accuracy: 0% → 30.6% 48-case cost: $37.13 → $21.57 (41.9% lower) secbenchmarks.com/blog/2026-…
🤖 Made with AI
Sonnet 5 to Sonnet 5.5, both at High effort. On 36 basic cases per model, verdict accuracy rose from 58.3% to 91.7%, and evidence-backed accuracy from 0% to 30.6%. Full 48-case cost fell from $37.13 to $21.57, 41.9% lower.

ALT Sonnet 5 to Sonnet 5.5, both at High effort. On 36 basic cases per model, verdict accuracy rose from 58.3% to 91.7%, and evidence-backed accuracy from 0% to 30.6%. Full 48-case cost fell from $37.13 to $21.57, 41.9% lower.

1
1
86
Each model started with one confirmed part of an attack and assessed connected processes, files, computers and accounts. The score rewards finding related items and penalizes unrelated additions. The supplied starting point earns no credit. Small pilot; no unseen test set.
23
We tested 3 AI decision models on a DFIR task: finding other activity connected to a confirmed attack. Cloudflare's Clef, Cloudflare's Clef Flash and Jev each ran 8 cases, 3 times. Here's how they stacked up: secbenchmarks.com/blog/2026-…
Follow the Incident. A linocut lighthouse sends a yellow beam along a connected trail. Scope scores out of 100 (F1): Cloudflare's Clef 70.8, Cloudflare's Clef Flash 62.1, Jev 1.13 20.8. Eight cases, three runs each. Early pilot; no held-out set.

ALT Follow the Incident. A linocut lighthouse sends a yellow beam along a connected trail. Scope scores out of 100 (F1): Cloudflare's Clef 70.8, Cloudflare's Clef Flash 62.1, Jev 1.13 20.8. Eight cases, three runs each. Early pilot; no held-out set.

2
1
1
164
🔎 GPT-6 Astra retrieved sufficient evidence in 36/36 basic DFIR cases. 25/36 ended with a correct decision and accepted proof. Finding the records doesn’t guarantee a supported conclusion. Preliminary, one pass per case. secbenchmarks.com/blog/2026-…
GPT-6 Astra retrieved sufficient evidence in 36 of 36 basic DFIR 0.6 investigations. Its final verdict met the accepted supporting-proof requirements in 25 of 36. Retrieval and evidence-backed accuracy are separate scores. One pass per case; related variants are not independent incidents.

ALT GPT-6 Astra retrieved sufficient evidence in 36 of 36 basic DFIR 0.6 investigations. Its final verdict met the accepted supporting-proof requirements in 25 of 36. Retrieval and evidence-backed accuracy are separate scores. One pass per case; related variants are not independent incidents.

10
📊 Our first AI security benchmark results are live. In 36 basic DFIR cases, Claude Opus 5.5 got 33 verdicts right. 12 also met our proof requirements. Why we score the answer and its evidence separately: secbenchmarks.com/blog/2026-…
Claude Opus 5.5: 33 of 36 basic DFIR 0.6 cases had a correct final verdict; 12 of 36 had a correct verdict with accepted supporting proof. Preliminary results, one pass per case; related variants are not independent incidents.

ALT Claude Opus 5.5: 33 of 36 basic DFIR 0.6 cases had a correct final verdict; 12 of 36 had a correct verdict with accepted supporting proof. Preliminary results, one pass per case; related variants are not independent incidents.

6
🧭 18/18 verdicts. 12/18 next steps. Jev 1.13's preliminary SOC results show why “right answer” and “right action” need separate scores. Our new analysis covers six models and the limits of a six-case test: secbenchmarks.com/blog/2026-…
Jev 1.13: 18 of 18 correct verdicts and 12 of 18 accepted next-step decisions in decision suite 1.0.0. Six cases from two related scenario families; preliminary results, one pass per question.

ALT Jev 1.13: 18 of 18 correct verdicts and 12 of 18 accepted next-step decisions in decision suite 1.0.0. Six cases from two related scenario families; preliminary results, one pass per question.

5
🔎 Security Benchmarks is live. 19 AI models. 48 DFIR investigation cases. We measure the verdict, the supporting evidence and whether the workflow finishes—separately. Explore the results and exact task questions: secbenchmarks.com/?utm_sourc…
Five highest basic-tier evidence-backed accuracy scores: GPT-6 Astra 69.4%, Claude Opus 5.5 33.3%, GPT-5.6 Sol 19.4%, GPT-5.6 Luna 5.6% and GPT-6 Luna 5.6%. DFIR 0.6 compares 19 models across 48 investigations in two tiers; this chart uses 36 basic cases, one pass per case.

ALT Five highest basic-tier evidence-backed accuracy scores: GPT-6 Astra 69.4%, Claude Opus 5.5 33.3%, GPT-5.6 Sol 19.4%, GPT-5.6 Luna 5.6% and GPT-6 Luna 5.6%. DFIR 0.6 compares 19 models across 48 investigations in two tiers; this chart uses 36 basic cases, one pass per case.

52
Replying to @DFIRTraining
The timing comparison is useful. Were the web-activity summaries also checked for correct artifact citations and missed domains? We’d like to see speed, factual accuracy and evidence coverage measured separately.
1
Load more