https://xitter.cf/t.co/9Od4BYinYs https://xitter.cf/t.co/o05daXPcOQ https://xitter.cf/t.co/c4l6UiFWMD Collab: collab@orbitlanelabs.com
London, England
Joined March 2022
- Tweets6.2K
- Following2.8K
- Followers12.2K
- Likes19.1K
A higher score can still be a worse eval.
Anthropic's 13 November 2026 post, "Building strong safety evaluations," puts it simply: a safety claim is only as sound as the eval behind it. If you only do three things, they say, write down the construct before collecting data, validate the grader against human judgment, and report uncertainty.
A card you can fill in before a run:
Eval question:
Construct we will score (not a proxy we don't actually mean):
Design: observational or causal (if causal, what single variable changes):
Grader: validated against humans? agreement versus human-human agreement:
Preregistered metric:
Comparisons:
Slices:
Improvement or regression threshold:
Repeats per item:
Uncertainty we will report:
Config to document: model version, surface (API versus app), system prompt, sampling, grader version, date:
Official notes worth keeping on the same card:
- Refusal rate is the wrong construct in some wellbeing settings. They score an appropriate-response rate across a multi-turn conversation instead.
- Configuration can move the number a lot. They report a gap as high as 34 percentage points between the API and claude.ai on multi-turn safety evals for Claude Fable 5.1.
- A headline gap can be noise. They report 98% versus 94% for Opus 5 and Opus 5.5 on a romance-scams eval, and say the difference was not statistically significant.
- Sample-size floors they use: 100 items for a complex multi-turn eval, 1,000 for a simpler single-turn one.
This is their public guidance, not an eval I ran.
anthropic.com/research/build…
A passing local AWS test can still be a request that real AWS would deny.
AWS Emulator is generally available as of 11 November 2026. Official blog: you run it on your machine, the AWS CLI and SDKs talk to a local HTTPS endpoint, and there is no extra service fee. It listens on 127.0.0.1:9297 by default. Access key IDs start with EMULATOR. It uses a single account, 000000000000, with no cross-account or cross-Region behavior.
Point the CLI at the emulator, and clear real keys from that shell:
export AWS_ENDPOINT_URL=https://127.0.0.1:9297
export AWS_SHARED_CREDENTIALS_FILE=~/.aws/emulator/9297/emulator_credentials
export AWS_CONFIG_FILE=~/.aws/emulator/9297/config
export AWS_EC2_METADATA_DISABLED=true
export AWS_DISABLE_HOST_PREFIX_INJECTION=true
export AWS_REGION=us-east-1
export AWS_DEFAULT_REGION=us-east-1
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN AWS_PROFILE
Then check the health endpoint with the generated certificate, and run aws sts get-caller-identity.
Two limits official docs are blunt about:
1. It does not evaluate IAM policies and resource-based policies the way AWS does, so a request AWS would deny might succeed here. Do not use it to test whether your application is secure.
2. An unimplemented operation returns HTTP 501. Coverage varies by service. They publish fidelity scores as k of n passing tests, and a score can go down when they add tests.
Linux installs from an RPM or Debian package. On macOS and Windows, official guidance is to run it in Docker from their Dockerfile; they do not publish a ready-made image.
I have not run the emulator. Check current fidelity for the services you actually call before you trust a green local suite.
aws.amazon.com/blogs/aws/tes…
2
240
If you asked "explain X" and the agent starts a quiz, the skill did not follow its own guard.
Jane Street published an MIT-licensed agent skill, socratic-tutor, in the official janestreet/agent_skills repo. The SKILL.md is for an ongoing teaching session, not a one-off explanation. If the user asked "explain X" or "what is X", it should answer normally and not start the tutoring loop. If intent is unclear, it should ask once whether you want a quick answer or to actually be taught.
When tutoring does start, the documented loop is:
1. Split the topic into 4 to 10 subskills.
2. Run a short diagnostic of 3 to 5 questions, one at a time, before teaching.
3. Show a skill map with levels 0 to 5.
4. Raise a level only after at least two independent checks, including one unprompted recall.
5. End with a mixed assessment.
Install, from their README: copy the socratic-tutor directory into your agent's skills folder, then ask it to teach you a topic. In a plain chatbot, paste the body of SKILL.md, then say what you want to learn.
I have not run a tutoring session with it. This is their published skill, not an AgentSkillsDir page, and not a Jane Street class.
github.com/janestreet/agent_…
1
3
255
The top score is not the default pick if this agent will live in Slack all month.
Deepnote's Agent Initiative Benchmark, on their research site, tested 12 models on the same 50 workplace situations in a synthetic company called PageMantle. Same instructions, same tools, same company snapshot, high effort, and a 180-second model-response budget.
From their results table:
GPT-6 Astra: 8.97 / 10, $0.693 per task, stayed quiet in 8 of 10 cases where silence was right
GPT-6.1 Sol: 8.65 / 10, $0.112 per task, stayed quiet in 8 of 10
Claude Fable 5.2: 8.17 / 10, $1.87 per task, stayed quiet in 6 of 10
Deepnote's own read: Sol sits inside Astra's 95% interval (8.30 to 9.52), so this task set does not separate them. Fable 5.2 is below Astra's interval. When help was needed, a message went out 9 times out of 10. When silence was right, models held back only 6 times out of 10.
They also price "a month of Slack" as one investigation per about 2,500 threads: $281 on Sol, $1,732 on Astra, $4,684 on Fable 5.2. Those are recorded historical costs, not a live quote, and they say so.
If I were choosing from this paper alone, I'd write the job down first: a live thread where someone is waiting (Opus 5.5 was the fastest among the leaders at 29 seconds, with a 7.84 score) versus always-on monitoring (Sol's score and cost). I'd also track false alarms as their own number. This is Deepnote's benchmark, not my production traffic.
deepnote.com/research/benchm…
1
223
A stack diagram your teammate can open is more useful than a snippet they can't explore.
Archify is an Apache-2.0 agent skill from tt-a1i. Official flow: the agent writes a standalone HTML architecture diagram you can open in a browser, switch theme, and export. Nobody needs Archify installed just to view the file. It is not Graphify, which is a queryable project knowledge graph, and it is not a hosted drawing service.
Upstream's first example prompt:
Use Archify to diagram a web request: Browser calls the API,
the API checks Redis, and a cache miss queries PostgreSQL and fills the cache.
Install, from the upstream README as documented on the AgentSkillsDir entry (commit a406594, last checked 31 October 2026; confirm against current upstream if you install today):
npx -y skills@latest add archify.si --skill archify --agent claude-code --global --copy --yes
Change --agent to codex, cursor, or opencode as needed. Then run the doctor command from the folder the installer reports.
This is a source-linked AgentSkillsDir page, not Archify's official site, and not a security review. AgentSkillsDir has not generated a diagram with it. I have not run the install.
agentskillsdir.com/skills/ar…
github.com/tt-a1i/archify
2
1
264
You don't need a Mac on your desk for an agent to check an iOS build. You do need to stop the session, or it keeps counting.
Expo Simulators is generally available on Expo paid plans. Official launch post: $0.10 per simulator minute on iOS and Android, and the Starter, Production, and Enterprise plans include 60 minutes per month until 2027. Official docs: run a read-only availability check first, and if available is false, do not start a session.
Install the official skill:
npx skills@latest add expo/skills --skill eas-simulator
Or run the documented loop yourself:
1. Update EAS CLI and run eas whoami. If you are not logged in, stop and run eas login, or set EXPO_TOKEN in a headless environment. Do not print the token.
2. From the project directory, run: eas simulator:availability --json
If available is false, stop.
3. Add .env.eas-simulator to .gitignore. Official docs: that file holds controller credentials. Do not commit or share it.
4. Start with a short purpose name, for example:
eas simulator:start --platform ios --type agent-device --name "Checkout screenshots" --non-interactive
5. Continue only when eas simulator:get --json --non-interactive shows status IN_PROGRESS and remoteConfig is present.
6. When you are done, run eas simulator:stop.
Official warning: --non-interactive does not stop the session. Usage continues until you stop it, or until a max duration or idle timeout you set. Docs: --max-duration-minutes can be set up to 40 minutes on the Free plan and 115 minutes on paid plans. Expo's launch post also says no session runs longer than two hours.
I have not run a cloud simulator session, and I am not claiming a speed or cost result of my own.
docs.expo.dev/eas/simulators…
expo.dev/blog/introducing-ex…
Your Claude Code project can start work your repo skills never see.
Anthropic's official docs say Projects are in public beta on Pro, Max, and Ultra, rolling out gradually, and are not offered on Team or Enterprise yet. If you don't see Projects in the sidebar at claude.ai/code or under More, the rollout hasn't reached you. They point you to a waitlist.
Before you send the first batch, I'd check this card:
1. GitHub: cloud threads need the Claude GitHub App and push access on github.com. A /web-setup token that already works for other cloud sessions is not enough.
2. Skills and tools: cloud threads load skills you enable on your claude.ai account. They do not load skills, .mcp.json, or .claude/settings.json from the repo. They do load CLAUDE.md at the repo root.
3. Models: a new project runs every thread on Opus at high effort, which official docs say draws on your plan fastest. Check Thread model and Thread effort in Project settings before a wide first batch.
4. Approvals: a permission prompt lives inside that thread. Telling the coordinator to go ahead in the project chat does not reach it.
5. Local threads: a thread on your computer runs only while that computer is awake with Remote Control on. Official docs also say a project cannot run a thread on your computer while Require trusted devices is on.
6. Limits: project usage counts against the same plan limits and uses them faster. There is an enforced cap of 500 new threads per day across your projects. Auto-continue when usage limits reset is on by default.
Official docs currently say you cannot share a project or its threads with another user during the beta, and project thread transcripts do not have the share option other cloud sessions have.
/btw asks a side question without adding it to the conversation. /fork copies the current conversation into a new background session. Official command docs also say /fork's isolation instruction needs Claude Code v2.1.221 or later.
This is from Anthropic's current Projects documentation, not a run in my own project.
code.claude.com/docs/en/clau…
5
6
448
Stop collecting prompt tips. Practice making one prompt fail less.
Anthropic's public interactive course has 9 chapters with exercises and an Example Playground. It covers clear instructions, separating data from instructions, output formatting, examples, and hallucinations. The README recommends working through it in order.
Here's a small exercise to take with you. Keep the input fixed and change only the instruction.
Synthetic input:
"My order arrived late. The box was damaged. I haven't opened the product."
Prompt A:
"Summarize this complaint."
Prompt B:
"Summarize this complaint in exactly three bullets: reported delivery issue, known product condition, and one question to ask next. If the product condition is unknown, say unknown. Don't infer damage to the product from damage to the box."
Compare the answers:
1. Did it keep the unopened product's condition unknown?
2. Did it invent a refund or a company policy?
3. Did it follow the three-bullet format?
The known facts in this toy input: late arrival, damaged box, unopened product. Product damage and refund eligibility are not established.
This is my practice exercise, not a model test result. The course materials are public; that does not mean every model call is free. Its README still references older Claude models.
github.com/anthropics/prompt…
Before you let an agent write more code, walk the patterns it will copy.
A coding agent learns from the repo in front of it, including workarounds, skipped tests, and messy leftovers. I have not measured a repair rate for this. Before a long run, I would fill this preflight on the actual tree (the example lines are labels, not a repo I audited):
Skipped or commented tests: [list paths]
Temporary guards or TODOs the agent might treat as style: [list]
Secrets, tokens, or credentials in the tree: [list and rotate if real]
Formatting or generated churn that is not part of the task: [list]
Clean the items you don't want copied, or tell the agent those paths are out of bounds. Then start the run. If you skip this, you are asking the model to imitate the mess as well as the design.
2
301
Don't write the summary until each row still points at the source.
When a bug thread, a chat export, or a ticket dump turns into a status update, the first thing that goes missing is the evidence. Use this prompt on a folder of notes you are allowed to share. The table below is a synthetic sample, not a live incident.
Prompt:
Read the files in ./inbox. Build a markdown table with one row per claim. Columns: claim, source (file name and a short quote or ID), last dated evidence, status (confirmed, contradicted, or unverified). Do not invent a date or a status. If the source is missing, leave status as unverified and say what's missing. Stop after the table. Do not write an executive summary.
Synthetic output:
Claim: checkout retries on 429
Source: notes/ops-thread.md, "retry once after a 429"
Last dated evidence: 2026-11-04
Status: confirmed
Claim: retries also cover 500s
Source: none in ./inbox
Last dated evidence: none
Status: unverified (no source)
If a row can't name a source, it isn't ready to send.
1
2
324
A skill update is not an update until you know which agents still have the old files.
When someone publishes a new version of a skill, every agent that already has a copy can keep running the previous files. I have not built a registry for this. Before you treat the new version as live, fill this inventory:
Skill name and source URL:
Pinned commit or tag actually present on disk:
Which agents or machines still have a copy:
Last date those copies were compared to the pin:
If the commit is missing, you are floating on whatever the last install left behind. Pin it, record the path, and only then update the agents that should move. Leave the rest on the old pin until you choose otherwise.
1
3
328
A Select component is the wrong unit if an agent has to invent the next control.
Vercel's UI SDK, in public beta as of Nov 12, packages reusable behaviors (popup, selection, typeahead, focus, drag and drop, and the ARIA they need) so an agent can compose a control that is not in the catalog. The same primitive can render in React, Vue, Svelte, Solid, or Web Components. Official install is in the GitHub README: the scoped ui-sdk npm packages for your framework, then:
npx skills add vercel-labs/ui --skill ui-sdk
The sharper risk is the other half: letting the agent use the page. createAgent() and snapshot() expose named controls and actions such as setValue. agent.act() goes through the component API, not a screenshot. Official docs report a project-settings benchmark, median of three Claude Code / Sonnet runs: browser tools used 16 tool calls, 41 seconds, and $0.073; the agent bridge used 5 tool calls, 16 seconds, and $0.032. I have not rerun those runs.
Before you enable that path, set an allow callback that can refuse a step, and make change handlers inspect source: "agent". The APIs can still change before general availability.
github.com/vercel-labs/ui
ui-sdk.dev/docs/agents
4
1
301
The useful question is what your shell, files, and stop conditions allow.
Claude Code and Codex get framed as two different "coding agents." Under the hood they are loops with tools, a filesystem, and a machine that already has the repo. The model matters, but so do the tools you expose, the files you mount, and the point at which the loop has to stop.
I would not pick a tool from the brand name alone. I'd write down the loop constraints first: which commands are allowed, which paths are writable, how a change is verified, and what happens when the login expires or the test budget is gone.
This is a reading of the problem, not a bench of either product. If you added one constraint that actually changed the result, the useful reply is that constraint, not a team jersey.
The wording is mine. The discussion was sparked by @ishaansehgal.
5
4
383
Start the multi-file agent in a worktree, not in the branch you're also typing in.
If you and the agent share one dirty working tree, a failing test no longer tells you whose edit broke it. git worktree gives the agent its own checkout. This is standard git, not a model feature, and not a measured success rate.
In a clean repo:
git fetch
git switch -c agent/topic-name
git switch -
git worktree add ../repo-agent-topic agent/topic-name
cd ../repo-agent-topic
Then leave a short agreement in that folder (synthetic example, not a real client file):
Goal: add a retry around the invoice fetch, then stop.
Test: npm test -- invoices
Do not edit: .github/, prisma/migrations/
Done when: the new test fails on a 429 fixture, then passes after the retry.
Start the agent in that folder. When you're finished, review the diff in that worktree, then remove it with git worktree remove once you've merged or discarded the branch.
3
302
You can scan for legacy web patterns without installing a skill first.
Modern Web Guidance is stewarded by the Chrome, Edge, Firefox, and MDN teams. Official README: coding agents often reach for old JavaScript because training data is full of it. The project feeds current, token-efficient guides into the agent instead.
I have not run this scan. The documented way to try it, with no install, is:
npx modern-web-guidance@latest legacy-scan .
You can also search or retrieve a single guide the same way. To install the skill for Claude Code, Codex, and other listed clients:
npx modern-web-guidance@latest install
Search can surface experimental Origin Trial or volatile APIs, but retrieve stays gated until you opt in with a .mwgrc file (or the documented set-allow commands). Code is Apache 2.0; the guides are CC-BY 4.0.
github.com/GoogleChrome/mode…
developer.chrome.com/docs/mo…
1
280
The action is not to retweet a date. It is to open your tenant's Message Center and the Anthropic subprocessor setting.
Microsoft's official Source post on November 6, 2026 says Claude models in Copilot will be available to eligible EU and EFTA organizations later in 2026 with processing inside the EU Data Boundary. That same post says those models are disabled by default in the EU and EFTA, and that admins enable them in the Microsoft 365 admin center for all users or selected groups. If you already enabled Claude, the later processing move does not require another step.
A separate tenant notice, circulating as MC1546438, is being quoted as a December 9, 2026 default change for tenants that have not made an Anthropic selection, and as the retirement of the "Copilot in Microsoft 365 apps only" option. I could not find that December 9 default on the current Microsoft Learn subprocessor page or in the November 6 Source post, so I am not treating a third-party archive as the live setting.
If you admin an EU or EFTA tenant, read the notice inside your own Microsoft 365 Message Center and confirm the Anthropic subprocessor control before you assume either default.
news.microsoft.com/source/em…
learn.microsoft.com/en-us/mi…
1
1
460
You can skip the n8n account. You cannot skip the model bill.
n8n Labs released Nod as an open-source research preview (Apache 2.0). Official blog: it is a standalone personal agent that runs on your Mac or Windows PC, or on your own server. It does not require an n8n account. It has a local sandbox and browser, can work in connected apps such as Gmail, Calendar, Drive, and Slack, and can also work on sites with no integration.
You connect your own model provider, with an API key or your ChatGPT plan, or a local open-source model. Nod itself is free. Model usage is not. Official docs say each task shows its cost, you can set a monthly spending limit, and Nod checks in once a task passes $5, which you can change.
Two limits to read before you lean on it: if it is running on a laptop, it stops when the laptop sleeps. The permission check is itself a model and can make mistakes, which is why it asks when your words don't cover an action. I have not run Nod.
blog.n8n.io/meet-nod/
nod.n8n.io
A finding the same agent just invented is not a confirmed vulnerability.
Cloudflare's security-audit skill is a defensive, source-first workflow for a codebase you are authorized to assess. Official README: six phases from reconnaissance and coverage-led hunting through an independent verifier, structured findings, and a target-neutral report. Loading the skill is guidance by default. Full audit mode, and the report files, start only when you explicitly ask for an audit, a pen-test, or those artifacts.
Install from the official repo:
npx skills add github.com/cloudflare/securi… --skill security-audit
Requirements they document: a coding agent that can use tools and parallel sub-agents, Node.js for the validators, and an OS-enforced sandbox if target code will be executed. Without those sandbox controls, the workflow keeps a lead as needs_validation instead of running the target. I have not run this audit.
This is a source-linked AgentSkillsDir entry, not Cloudflare's official site, and not a security review. The listing is pinned to commit c1c8a8c, which matched the repo's main branch when I checked it this morning.
agentskillsdir.com/skills/cl…
github.com/cloudflare/securi…
1
4
316
Install the skill last. Scan it first.
NVIDIA's SkillSpector is an Apache 2.0 scanner for agent skills. Official docs say it accepts a Git repo, URL, zip, directory, or a SKILL.md file, and can run a fast static pass or add optional LLM review.
I have not scanned a skill with it. The numbers below are from NVIDIA's README, not from my dataset: in a 31,132-skill analyzed subset, 26.1% contained vulnerabilities and 5.2% showed likely malicious intent.
Quick install (Python 3.12+; uv or pip). NVIDIA also documents a Docker path if you do not want a local Python install:
uv tool install git+github.com/NVIDIA/skillspect…
Then, for a static-only first pass:
skillspector scan github.com/user/my-skill --no-llm
A clean report is not a security certification, and a directory listing is not one either. Read the report, then read the skill, then decide whether to install.
github.com/NVIDIA/SkillSpect…
docs.nvidia.com/skills/scann…
If you still need live video or screen sharing in ChatGPT, don't assume Live voice can do that yet.
Official help says Advanced and Standard voice retire on December 11, 2026. After that date you can't pick them in Settings > Voice, and a previous Advanced or Standard choice is switched to Live. Conversations stay in your history.
Use this card before you change the setting:
Live (GPT-Live): full-duplex voice. Official help says it does not support video or screen sharing.
Advanced: the previous turn-based voice model. Official help says live video and screen sharing stay on Advanced until GPT-Live has that capability. Voice with custom GPTs also keeps using Advanced until December 11.
Standard: speech-to-text, then a text model, then text-to-speech.
I have not timed the three modes. If video or screen sharing is part of your workflow, check Settings > Voice and the official help before December 11, rather than assuming Live already covers it.
help.openai.com/en/articles/…
help.openai.com/en/articles/…
1
1
258