AI Researcher 🧑🔬 | DevOps Engineer from profession | Indie hacker by heart | Love to build things
San Francisco, CA
Joined July 2026
- Tweets8.7K
- Following1.1K
- Followers895
- Likes1.3K
Pinned Tweet
Every time I hit a wall with one model, I used to paste my whole thread into another one and explain the project from scratch. That part wasted more time than the actual work.
llmwise.ai puts 16 models from 7 companies in one chat, on one plan. Switch models mid-chat and the next model sees the whole conversation. No re-explaining your project.
I pick a model per message, and the message count shows before I send. My usual move: start on a fast model for the boring first pass, then switch to a stronger one for the hard part, same thread.
Pro is $20 a month plus tax. The monthly allowance is mixable, up to 125 messages on Claude Sonnet 5-class models, plus 60 fast messages a day.
Helps anyone juggling more than one AI subscription.
Most people guess why a model is slow. Few read the trace.
Amazon published Part 1 of a series on profiling TorchNeuron, the open source native PyTorch backend for AWS Trainium. It assumes no profiling experience. It walks RMSNorm through three implementations.
What it is:
TorchNeuron runs standard PyTorch on Trainium: eager execution, torch.compile, standard APIs, minimal code changes. It uses PyTorch's PrivateUse1 extension point, so it needs no dedicated dispatch key like cuda.
Profiling takes two objects. NeuronConfig sets the data streams, the NeuronCores to capture, and the output directory. Pass it to torch.profiler.profile as experimental_config. NeuronProfiler wraps it and gives an export_trace callback. That callback writes trace.json next to the Neuron system traces.
nc = NeuronConfig(
modes=[ProfileMode.DEVICE, ProfileMode.RUNTIME],
profile_output_dir=OUTPUT_DIR,
correlate_device_time=True,
)
What it means for builders:
1. Set correlate_device_time=True. It moves device time into the key_averages() table. Then you compare CPU time and Neuron device time in one place.
2. Read key_averages() first. It shows which operations use Neuron device time.
3. Use Neuron Explorer for the space between dispatches. It is a browser viewer for the system trace, device timeline and per-engine breakdown. Open it over an SSH tunnel or the VS Code extension.
4. Keep the.neff and.ntff files as a pair. NEFF holds the compiled executable. NTFF holds the hardware instruction trace. A shared numeric hash marks the pair.
5. Compare lowering paths on the same inputs. The post profiles RMSNorm three ways. First, separate PyTorch ops: multiply, mean, add, rsqrt, multiply. Second, torch.rms_norm() as one fused Neuron kernel. Third, the same ops under torch.compile.
The honest limit: Part 1 shows how to capture and read a trace. It gives no timing numbers for the three versions. You must run the scripts yourself. They are in the torch-neuronx examples directory.
Which operation in your model would you profile first?
A cache that never goes stale is a cache that never gets invalidated. That is the bug.
Most CI caches key on the lockfile alone. The lockfile pins your dependencies. It does not pin the tool that installs them. Bump the Node, Python, or pnpm version and the key stays the same. The restore hits. The cache is now wrong, and the failure looks like a flaky test.
Here is the map I use.
1. Build the key from three parts: runner OS, tool versions, lockfile hash.
2. Hash every lockfile, not just the root one. Monorepos have several.
3. Add a manual version prefix, like v1. Bump it when you change what you cache.
4. Treat keys as immutable. GitHub Actions does not overwrite an existing key. A second save with the same key is skipped.
5. Use restore-keys for a soft fallback, then read the log. A partial restore that saves two seconds can hide a broken lockfile for a week.
6. Cache the package manager download directory, not node_modules. node_modules ties the cache to one OS and one architecture.
The concrete bit. Do not hand the key to a setup action and walk away. Read the cache step log and check what goes in. Most setup actions build a key from the lockfile hash and the OS. They leave the runtime version out.
So write it yourself:
key: ${{ runner.os }}-node${{ matrix.node }}-${{ hashFiles('/package-lock.json') }}-v1
Avoid the version without the node part. It hits on every runtime upgrade and serves the wrong tree. You will blame the tests.
What is in your cache key today? Paste the line and I will tell you what goes stale first.
a bad answer is easy to spot and hard to explain. the fix is usually in the log, not the prompt.
what do you write down on every tool call so you can debug a bad answer a week later? inputs, raw output, latency, model version, something else?
Connecting with builders working on AI and agents.
I am a senior SRE with 10+ years on Kubernetes, AWS and Terraform. I also write arXiv papers on chain-of-thought faithfulness and build AI agents for small teams.
What is the one repo you always open when you start a new agent project? Reply with what you're building. I follow back builders.
Claude is also coming with always on agent like grok bot and dots
Interesting paper on silent tool errors in e-commerce agents.
A tool call can return a wrong value with no error signal. The value looks plausible. The agent keeps going. Each later step builds on the bad value. The final answer is wrong, and nothing in the trace says why. The paper calls these silent errors.
The team built RITE, a benchmark for this case. It comes from human-authored industrial workflows and inputs derived from production logs. They used a semantic error taxonomy. They injected errors into fixed workflow prefixes and froze recorded tool behavior. Then they let the agent continue freely. That makes each run controlled and repeatable.
The benchmark has 167 matched units across nine e-commerce workflows. That yields 501 instances: silent-error, correct-value, and explicit-failure versions. Each unit is matched, so the error type is the only change.
They ran 10 recent LLMs. Silent errors roughly halve task success. Silent errors are also consistently harder than explicit failures. The failures stack in three places. The agent does not recognize the error. Or it does not use the available verification path. Or it does not integrate the evidence it gets. Recognition, verification, and integration are successive bottlenecks. They also propose a light State Checker as an intervention, to test if resilience can improve.
The clever part is the matched unit: freeze the tools, swap only the error type, and resilience becomes measurable. A builder can steal that: run the same task three ways, correct, silent error, explicit failure, and compare.
Hardest numbers: silent errors roughly halve task success across 10 LLMs, on 501 instances from 167 matched units.
If you build tool-using agents on real workflows, this is built for that.
Paper: arxiv.org/abs/2610.27777
Vague feature requests waste sprints. "Make search faster" turns into a long argument. These prompts turn that request into criteria you can test.
1. "Here is a feature request: [paste]. List every assumption it makes that nobody stated. Do not write criteria yet."
2. "Rewrite the request as one sentence: who does what, and what changes for them. Flag every word I cannot test, like fast, easy, better, soon."
3. "Replace each vague word with a number. Ask me for the number. If I do not have one, write unknown and stop."
4. "Write criteria as Given, When, Then. One behavior per scenario. Split any scenario that has two and conditions into two scenarios."
5. "List the edge cases: empty input, max length, duplicate submit, no permission, network failure, timeout. Write one criterion for each."
6. "Write the negative criteria. State what the feature must not do, and what must not change for current users."
7. "Sort every item into in scope or out of scope. Move anything I did not confirm into out of scope."
8. "For each criterion, name the test that proves it: unit, integration, or manual. Mark the ones that need a human."
9. "Rewrite any criterion that needs a judgment call into a pass or fail check. Show me the old line and the new line."
10. "Give me three questions to ask the requester before we build. Rank them by how much rework each one prevents."
Save this for the next vague ticket. Reply with the prompt you would add.
Interesting repo: qiz029/dscode (998 stars).
It is a terminal coding agent for macOS and Linux, built on DeepSeek Harness. It is written in JavaScript. Most coding agents work alone. DSCODE is built on the opposite assumption: sessions on your machine are visible to each other. One session can hand a task over. Another can review the diff.
What you can actually do with it. File tools read and edit code with file-version checks. A persistent shell handles searches, builds and tests. The TUI, the CLI and your scripts share one session runtime. The first `dscode` starts a local background host that owns the sessions, and each terminal attaches to it. Close a terminal and the work survives. Several terminals can share one session. `dscode host status` and `dscode host stop` manage it.
One detail worth a bookmark. Type `/btw why is the cache cold on the first turn?` in the TUI. The side question runs in its own read-only child session and answers in a panel. That exchange never enters the main conversation. You get the answer without polluting the context.
Other things I want to try. `dscode send --steer "review the change in parser.ts and reply"` hands work to another session on this machine. The other session can read your transcript, answer, and hand the result back. `/permission auto-review` lets an independent reviewer decide approvals from your instruction, not from a rule table. `/review-usage` shows what it allowed and what it cost. You can run the runtime on a server with `dscode server --port 8080`, then point your terminal at it. Tools, files and credentials stay on the server. The readme is clear that this link is token-authenticated but unencrypted, so use a VPN, LAN or SSH tunnel.
One more thing that matters for real work: it installs as a pinned, reproducible harness. DSH dependencies, TUI and plugins are versioned and verified together. So two machines can run the same setup.
News: DeepSeek open-sourced DeepInfer, the inference framework that powers its models in production.
The repo is public on GitHub at deepseek-ai/DeepInfer.
The repo description reads: high-performance, large-scale inference framework for DeepSeek models.
The README says DeepInfer powers DeepSeek's large-scale production.
The license is Apache-2.0.
The repo shows 6.7k stars and 546 forks.
It shows 5 watchers and 32 open pull requests.
The commit history is short. There are 3 commits on main.
The code sits in folders at the top level. The names are csrc, deep_cluster, deepinfer, doc, eval, examples, scripts, tests and vendor.
Build and packaging files sit next to them: Makefile, build.sh, setup.py, pyproject.toml and requirements.txt.
There are two hardware requirement files: requirements-nvidia.txt and requirements-ascend.txt.
One points at NVIDIA hardware. The other points at Ascend hardware.
There is a requirements-dev.txt for development work.
The repo also holds CONTRIBUTING.md and a.clang-format file.
The.gitmodules file means the repo pulls in other repos as submodules.
The doc folder and the eval folder are both in the tree, and so is examples.
Why builders care: the serving stack behind a large model is now code you can clone, read and test.
DeepSeek open sourced the inference framework that runs its own production services. It is called DeepInfer. The release is offline inference only, and it supports one model.
That model is DeepSeek-V4.1-Flash. The license is Apache 2.0. The hardware list is short: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
What it means for builders:
1. Determinism is a flag, not a rewrite. Set DEEPINFER_BATCH_INVARIANT=1 and results stay identical across different batch compositions. Add DEEPINFER_ENABLE_MTP_VARLEN=0 and pass a fixed seq_seed to SamplingParams to get identical outputs on identical input. NVIDIA only.
2. Structured outputs are built in. Pass so_args={"schema_type":..., "schema":...} to SamplingParams and constrain generation to a regex, a JSON schema, or an EBNF grammar.
3. Decoder prefill work drops with SWA bounded replay. Set DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 for the V4.1 defaults. The kept suffix must cover the SWA window plus the maximum backtracking length.
4. Async scheduling is on by default. CPU batch scheduling overlaps device execution, so you get the zero overhead batch scheduler without tuning it.
5. Build steps are plain. Linux, GCC 13 or higher, Python 3.12, CUDA 13 on NVIDIA or CANN 9.2 on Ascend. Export DEEPINFER_ACCELERATOR=nvidia, then run./build.sh. The first run downloads the checkpoint unless you set DEEPINFER_HF_MODEL_DIR.
The honest limit: this is not a full serving stack. A KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data parallel replicas are not in the release. Most of the disaggregation engine code is already there. You supply the rest. The benchmark setup used 4 SM100 GPUs for prefill and 64 for decode.
Which part of your serving stack would you still have to write yourself?
A slow query alert fires, and the first fix is always the same guess: add an index. That guess is often wrong. These prompts make the database explain the plan first.
1. "Here is my query and the EXPLAIN output. List every sequential scan, the table it hits, and the filter column behind it. Do not suggest an index yet. Just list the scans."
2. "Compare EXPLAIN with EXPLAIN ANALYZE for this query. Show me each step where the estimated rows differ from the actual rows. Give the ratio for each step."
3. "List every column in my WHERE, JOIN and ORDER BY clauses. Mark which ones an existing index already covers. I will paste my index list below."
4. "Propose a composite index for this query. Equality columns first, then the range column, then the sort column. Give me one line per column on why it sits there."
5. "This index exists but the planner ignores it. Check these causes against my query: type mismatch, a function wrapped around the column, low selectivity, stale statistics. Say which one applies and why."
6
prompt changes are cheap to write and expensive to test. every run costs tokens, and a bad batch costs a day.
how do you test an agent prompt change without burning money? a small eval set, replaying old traces, a cheaper model first, or something else. what worked for you this week?
Great video from Tech With Tim on building a RAG based LLM app in Langflow.
What is inside:
- The promise is in the title: a RAG based LLM app, built in 20 minutes.
- It is a full Langflow tutorial. Langflow is the tool used for the build.
- The description says the video shows how to create your own AI application.
- Tim also points to his free software development guide, linked in the description.
- The video has 221, 884 views so far.
Why it is worth the watch: one tool, one app, 20 minutes. That is a small block of time for a full build. You can watch it once, then do it again with your own hands.
A few notes from me, as someone who builds AI tools every day.
RAG means the app fetches text and puts it into the prompt before the model answers. The model stays the same. The data changes. That is the part you control, so that is the part to get right.
Short tutorials work best when you pause. Do not watch the whole thing first. Build at the same speed as the video. When a step fails, you know exactly which step failed.
Keep a working copy. Copy the project folder before you change anything. A broken build with no copy back costs you an evening.
Write down your settings. Model name, chunk size, number of results. You will forget them in a week, and then you cannot repeat your own result.
Then change one thing. Swap the data source. Ask a different question. A RAG app is only as good as the text you feed it.
One more point. A 20 minute build is a starting point, not a product. The demo shows the flow works. The work after that is your data, your tests, and your error cases.