Today on a whim I built the equivalent of DocSend to send a set of documents that I needed to send out. It was a better use of time. It's so easy to be delighted everyday with AI.
Today the security factory finished building a credential broker for agents. By doing this we’re moving towards the agents having no standing access and instead injecting creds to every call. Also creating a kill switch.
I've been calibrating cloud models and I think I'm finally done in getting a sense of when to pull in Opus, Sonnet, Haiku, Sol, Astra, Luna, Grok, Gemini.
It's all the use cases where the benchmarks don't test that are some of the most interesting and the types of roles I've been baking into a local + cloud model driven harness that runs my agent swarms across my computers.
I’ve been shadow benchmarking 36GB-48GB capable local models alongside cloud models by having an orchestrator running the same prompts through both and the same harness. In terms judgement the local models match the actual decision 10-15% of the time while cloud models are 70-80%
What I do to shadow benchmark these models is run real prompt through them overnight + pinned to prior commits so it’s representative. Last night was the breakthrough.
I’ve found asking agents to edit another agent’s writing tends to lead the agents down the path of agent code speak. Some models are especially prone to it. Is there an eval for this? If not there should be one for “incoherent semantics” youtube.com/watch?v=ORgKY9Al…
What am I missing? Next is actually paying for tokens for models that run in 96GB-256GB. Or actually buying the hardware. Now that I’m getting close to figuring out how to apply Claude, Codex, Gemini, Grok models in real life, it’s the local models next.
I published some research recently after building a data pipeline that scans over 200K job descriptions to get a sense of where AI + Security is going: akasecurity.io/research/
We've started to talk about agent swarms recently which is really an extension of loop engineering. How long can an agent operate autonomously (and productively) and then how many of them?