Web Summit starting today in Lisbon. To kick start the new @monitive uptime monitoring tool, I started monitoring all the exhibitors from this year. Let's see how their site holds 👀
2
1
63
I'm getting a regular newsletter from Indie Hustle on how some random dude made tens of thousands of $$$ with some simple site, so I had my AI agent read all 123 founder stories in the Indie Hustle newsletter and sort out how each business got its customers.
SEO came up most, in 64 of them. Referrals 43, X 42, directories like Product Hunt and the app stores 40.
But 66 stories list channels without saying which one mattered most. Of the 57 that do, SEO leads with 15, then directories 8, X 7, referrals 6.
Paid ads: 19 stories say they failed, 3 call them the main channel.
Reddit: 30 mentions, main channel in 3, failed in 4.
This is what survivors say about themselves, not what works. Failed businesses don't get profiled.
Filter it by business type and revenue, and read the quote behind each one.
pages.daniliuc.com/a/indie-d…
3
92
I pointed a code reviewer at the implementation plan instead of the diff, before any code existed.
The first run reproduced both findings that had escalated during implementation on earlier cycles, and surfaced three nobody had seen.
Diff review is structurally late. By the time there is a diff, the design is already paid for, and the reviewer ends up arguing about lines instead of decisions.
plan review finds a different class of thing: missing states, failure paths with no owner, contracts nobody wrote down.
It does not replace reviewing the code. A fresh P1 still turned up after the replan, which is the argument for both, not against one.
Cheapest review in the cycle, and it runs before anyone has written anything to defend. Blog post which touches this subject coming soon.
28
I was about to put two applications in one secrets config, and stopped when I noticed both are Laravel.
@doppler configs are a flat namespace. Every Laravel app defines APP_KEY, APP_URL, DB_DATABASE, DB_USERNAME. Two of them in one config do not merge, they overwrite. The second app you add silently takes the first one's database credentials and encryption key.
The failure would not be a startup error. It would be one app decrypting sessions with the other app's key, and connecting to the wrong database, in whichever environment you deployed second.
One project per application, not per company and not per developer. It reads like bureaucracy until you notice the shared namespace guarantees a collision between any two apps built on the same framework.
Frameworks standardise variable names. That is the feature and, in a shared config, the bug. #lessonlearned
33
A service of mine restarted 13,400 times over about 20 hours. Nothing alerted. Nothing looked wrong.
It crashed every five seconds because one environment variable was set without its partner: a signing key path with no issuer URL. The process refused to start, systemd restarted it, and the cycle repeated all night.
It surfaced only when I tried to connect a client and got a Cloudflare 521.
The gap was not the config. Config gets things wrong every week. The gap was that a service dying every five seconds looked exactly like a service nobody had asked for anything.
Two alarms cover this and neither is expensive. Alert on restart count over a window, not on the process being absent, because a crash loop is technically running. Then add a post-deploy smoke check that calls the service once and fails the pipeline when it answers wrong. Mine is about fifteen lines.
I broke it on purpose afterwards to watch the alert fire. Test the alarm, not only the fix. #lessonlearned
27
A code review caught a bug that would have silently switched off alerting for a specific set of users: everyone whose monitor name contains an emoji or Chinese characters.
The notification provider put the monitor name into an HTTP header. HTTP headers are ASCII. In JavaScript, fetch does not transliterate or strip a non-ASCII header value, it throws, so the alert is never sent at all.
The users who lose alerting are exactly the ones who never see an error. Their monitor looks configured. The failure lives in your dispatch path.
The rule that comes out of it: user-controlled text does not belong in a header. Move it into the JSON body, where UTF-8 is defined behaviour.
The regression test is a monitor called "café 🔥 服务器". If your alerting survives that name, it survives your users. #lessonlearned
30
I tagged two releases at 2:39 in the morning. By 8:25 I had answered a customer who had been waiting since February.
The trick is that a release note is a search query you have already written. Every line is a feature somebody asked you for, in the words they used when they asked.
So after tagging, I paste each line into my mail search. That morning "SSO" surfaced a February thread where a customer had asked for Microsoft SSO and been told it was in development. It had shipped an hour earlier. He got a reply the same morning it went live.
Nobody has to remember who asked for what. The changelog remembers, and your inbox is the index.
Ship, search, reply. It takes ten minutes and it lands better than any newsletter.
2
1
31
I had an expensive model implementing and a cheap one nowhere. Then a job stalled on review round seven, I threw a cheap model at the same scope, and it finished in two and a half minutes.
The split I use now: the big model plans and judges, the cheap model implements and fixes. Judgement needs context. Applying a review comment needs almost none.
The second rule cost me more to learn. Never let an implementation worker run its own code review inside its own context window. It has already spent that window on the work. Mine hit the 200k ceiling mid-review and died with the fixes unmade.
So review moved out to the main session permanently. The worker writes code. The session that has room to think reads it.
Model choice matters less than deciding which job gets the context. #lessonlearned
26
Six repositories returned 404 to a token that the GitHub UI said had access to all of them.
The checkbox is labelled "All repositories". It means all repositories owned by the account you selected when you created the token. A fine-grained personal access token is scoped to exactly one resource owner. Repositories on other accounts, where you are only a collaborator, are not in scope and cannot be added.
There is no error for this. The API answers 404, which is the same answer it gives for a repository that does not exist, because GitHub will not confirm the existence of something your token cannot see.
So you spend an hour re-checking permissions that were never the problem.
If your work spans repositories owned by several accounts or organisations, a classic token with repo and read:org still does what you expect. Fine-grained tokens are better scoped and worse at exactly this. #lessonlearned
29
ghost crash-looped 4,300 times overnight on my blog server. It booted clean when I ran it by hand. MySQL was fine. Permissions were fine.
The disk was 100% full. Every log write hit ENOSPC, the process died, systemd restarted it, and midnight logrotate had lit the fuse.
Here is the part worth your attention. An endpoint check would have told me the blog was down. It could not have told me the disk crossed 90% six hours earlier, which is the only warning that would have prevented the outage instead of reporting it.
HTTP checks are a lagging indicator. They fire once your users are already looking at the error.
I pruned 15 GB of docker build cache and unused images, got the blog back, then put host alerts next to the endpoint checks: disk 85%, memory 90%, CPU 90%, on every box.
It fired the next morning at 86% on the same server, from a 4 AM model pull that left 25 GB of failed partial downloads behind. Caught before anything went down.
Watch the hosts, not only the URLs. #lessonlearned
37
My ideal customer does not use social media.
I found that out by asking where his spare minutes actually go. Gym. Kids. A DIY project he is halfway through. A book. A run. Reddit never came up. X barely did.
Awkward, because the entire growth playbook for a small dev tool is roughly "go be interesting on Reddit and X".
Two channels survived that interview. Podcasts, the one format that fits an hour under a barbell. And search, because when the thing finally breaks at 11pm he types the problem into Google, and I would rather be the answer than the ad above it.
The lesson was not really about channels. I had been optimizing for the customer who is easy to reach instead of the one who is easy to serve, and those are not the same person. #lessonlearned
16
Half a shipped feature sat inert in production behind a green merge. The code was fine. The container simply could not see the new environment variable.
The value was in the .env file at 14:18. The container had been created at 13:22. docker compose restart restarts the process inside the existing container, and a container's environment is frozen at create time. A restart cannot pick up a file edited afterwards.
The trap is that every check you run on the host agrees with you. cat .env shows the value. The compose file lists env_file. The deploy log is clean.
The only check that counts runs inside the container: docker compose exec <svc> printenv VAR. If that prints nothing, the process never had the variable, whatever the host file says.
The fix is docker compose up -d, which recreates the container. Compare the container id before and after, because "up" can report success without recreating anything. #lessonlearned
22
A cron job had been doing nothing for two days. No error, no log line, no clue.
The command worked when I ran it by hand. It was a shell function defined in .bashrc, and cron never sources .bashrc. Under cron the name simply does not exist.
So I inlined the real command. Still nothing. The function also called docker compose exec without -T, which wants a terminal. Cron has no TTY, so the call dies the moment it asks for one.
Two independent reasons, and fixing only the first leaves you with exactly the same silence.
What shipped: a wrapper script with an absolute PATH, set -e so a failing step exits non-zero, -T on every exec, and a lock so a slow run cannot overlap the next tick. The following tick pushed 19 records with zero errors.
A scheduled job working in your shell proves nothing about cron. Test it the way cron runs it: env -i /bin/sh -c '<your line>'. #lessonlearned
So, I nearly shipped a rate limiter that would have flagged half the internet as down. ❌
On paper, at least.
The intent was polite. Cap how often we check any one domain, counted across every account, so a popular site does not get hammered by us.
The bug was not the cap. It was where a throttled check went afterwards: straight down the failure path. So the busiest domains, the ones the most people watch, would have been reported down for everyone at once. Incidents opened. Webhooks fired. For sites that never went off.
A check you skip on purpose is not a check that failed. It has to record as skipped, or your politeness turns into an outage you invented.
#lessonlearned
1
37
For weeks every deploy to my server finished green and then painted a red X on the run ❌
The deploy worked fine. The health check at the end of it did not: it exited with "api-key is required" because I had never wired the key in. So the last step of a successful deploy failed every single time, for its own reasons, and I learned to scroll past it.
That is the real cost. A check that fails for reasons of its own does not just give you one wrong answer. It retires the signal. By the time a deploy fails for a reason that matters, nobody is reading that line any more.
Fixed by wiring the key and the monitor id, which took two minutes. The part worth remembering is how long I had been training myself to ignore it. #lessonlearn
25
Was working on my DailyCoinRecap service a while ago. It shows Bitcoin prices, news summaries and more. Of course I didn't want to depend on a single pricing provider (CryptoCompare at that time), so my price feed had a backup provider, CoinGecko.
Then the primary failed (they actually dropped their free quota) and I quickly learned that the backup had never run once.
No key was ever configured, so two bugs sat there unseen for months.
An untested failover path is not redundancy; it is a comment.
Now I break the primary on purpose. #lessonlearned
19
Protect today’s income while deliberately buying tomorrow’s independence. #notetoself
17
A migration merged green and then broke the deploy.
CI ran the suite on SQLite: pass. Production runs Postgres: SQLSTATE 0A000, unique constraint on virtual generated columns not supported. The release never activated.
The code wasn't wrong twice. The same DDL is legal in one engine and impossible in the other, and a test suite can only certify the engine it actually runs on.
Two fixes shipped: a partial unique index both engines accept, verified against a real PG18 container before merging, and a CI leg that runs the whole suite on Postgres, so parity is checked by a machine instead of by the deploy log.
If prod and CI disagree on the database engine, your migrations are untested by construction 💥
#lessonslearned
3
2
50
Two automated planning jobs hit the same repo. Both used their own git worktree. They still collided: "cannot lock ref".
💡A worktree is not an isolated repo. You get a separate working tree and index, but one shared .git underneath: same refs, same packed-refs, same locks. Two processes fetching or branching at once are still writing to the same place.
Worktrees isolate files, not git operations. If two jobs can touch refs concurrently, you need a lock around the git ops, or truly separate clones. We went with a lock; a clone per job costs disk and setup time the pipeline didn't have.
The rule that survived: parallelize the work inside the tree, serialize everything that writes to .git.
32




