Late night coffee runs are a thing these days.

Unpopular opinion: If you are buying AI courses from all these course sellers who have been in the business for quite a few years and haven't done any development/deployment, etc., for years, then you are the biggest fool. Just think about it: You are buying a course to learn AI development from those who have been selling courses for years and haven't had any AI production experience. AI is self learnable and if you need to pay thousands/lakhs for that, I would rather have you buy enough credits and experiment, break things and learn from free YouTube videos. --- Bhai agar LLM agent integration seekhna hai toh mai free me workshop krduga X pr. Comment krdo. Agar ache free utube tutorials ka idea hai toh wo bhi comment krdo. Let's make this the best twitter thread to learn production grade AI integration for free.

Anyone can say “learn these 10 things in distributed systems” and make a nice checklist. Heck, I do that too. But real experience starts showing when someone can explain especially on X - what they actually do when prod is on fire. Like p95 suddenly jumping from 200ms to 2s. Do you just stare at the API dashboard? Or do you correlate that spike with downstream services, check DB latency, queue depth, cache hit rate, CPU/GC, then follow the same request using correlation IDs across services? Can you explain why p50 looks perfectly healthy while p95/p99 is dying? Can you tell whether retries are actually making the outage worse? Can you look at traces and figure out that one dependency is adding 800ms only for a certain request path? Tbh kind of knowledge rarely comes from interview preparation. It comes from debugging ugly systems for months/years, breaking things, fixing them and slowly building intuition.

Senior engineer prod scenario: an AI agent hits a paid API and gets stuck in a loop. Cost goes from $50/day to $5k/hour. You also need an audit trail of every call and output for compliance/debugging. What do you do in the first 5 minutes, and what do you ship after? I’m looking for: hard kill switch (per org + global), spend/rate limits with budgets, idempotency keys so retries don’t double-bill, rollback strategy if bad outputs already wrote to DB, and observability that ties agent step -> request -> downstream charge -> persisted data. Also: how do you keep logs append-only and queryable without logging secrets or PII, and how do you prove data correctness after the incident?

For those who don't know .. 1. Strong read-after-write consistency If you write an object to S3 and the write succeeds, any read immediately after will see that latest object. Example: PUT logs/100 = "event A" → success GET logs/100 → "event A" You do not have to worry about another reader temporarily seeing an older version after the successful write. --- 2. Conditional writes with If-None-Match: This basically means: > "Create this object only if this key does not already exist." Suppose two servers both try to claim log position 101: Writer A → PUT logs/101 with If-None-Match: * Writer B → PUT logs/101 with If-None-Match: * Only one succeeds. The other gets a failure because logs/101 now exists. So you can build something like: logs/100 logs/101 logs/102 ... Multiple writers can race for the next slot, but only one wins each slot. Combined with strong consistency, the loser can immediately see who won and retry the next position. That is the interesting part: S3 itself becomes the coordination primitive, so for certain designs you may not need a separate Raft leader just to decide who gets to append next. But S3 does not automatically provide an ordered distributed log - you still need to design the append/retry protocol around these guarantees.

Tech story time !! One year ago, we had one of those production bugs that was almost impossible to notice. We had a Kafka consumer processing events from another service. 99.9% of the time everything worked fine. Consume message -> process it -> update DB -> move on. But once in a while, processing would fail. Maybe the downstream service timed out. Maybe some payload had unexpected data (some edge cases here that weren't handled). Sometimes the DB connection dropped for a few seconds. Nothing unusual in distributed systems. The problem was our retry logic. We retried the message a few times, and after that... we basically gave up. No DLQ. The consumer kept moving forward, so from the outside the system looked completely healthy. CPU fine. Kafka lag looked fine. Error rate almost zero. No major alerts. But that tiny percentage of messages were effectively lost from the business workflow. And because it happened so rarely, nobody noticed immediately. Weeks later someone found a mismatch between two systems. We started tracing IDs manually and eventually found events that had entered Kafka but never completed processing. That was the scary part actually. The system was not "down" per se. But it was silently wrong. We fixed it by introducing a proper retry + DLQ flow. Normal topic -> consumer -> temporary failure -> retry topic with backoff -> retry again -> after max attempts -> DLQ But just adding a DLQ was not enough. We also added the original payload, error reason, retry count, timestamp and correlation ID so someone could actually debug the failed message. Then alerts for DLQ growth. Then a small replay tool so after fixing the underlying bug, we could safely send failed events back through the pipeline. We also had another bug which led us to make consumers idempotent because replaying messages without idempotency can create an entirely new problem. Duplicate payments, duplicate rows, duplicate notifications... whatever your consumer happens to do. That incident really taught me something important about event-driven systems: Message loss does not always look like message loss. Most of the times Kafka is healthy, consumers are healthy, dashboards are green... But your business data is slowly becoming inconsistent. A DLQ is not just a queue where "bad messages go". It is a safety net for failures you did not predict. And in distributed systems, there will always be failures you did not predict.

Your database CPU is at 95%. P99 latency just crossed 8 seconds. You have 10 minutes before the payment service starts timing out. What do you check first?

Your cache-aside pattern has a race condition: two requests miss cache simultaneously, both fetch from DB, both write to cache. One overwrites with stale data. How will you prevent this? [Asked at Twitter interview]

Most distributed systems bugs are not networking bugs. They are two machines disagreeing about what time it is. One node expires a lease. Another still trusts it. Now both think they are correct. And that is how simple clock skew can actually become a production incident.

Top 10 Kubernetes mistakes I keep seeing: 1) No resource requests/limits, then nodes OOM and noisy neighbors win 2) Liveness/readiness probes missing or wrong, so traffic hits half-started pods 3) Deploying without PDBs, then a node drain drops 100% of replicas 4) HPA on CPU only (or none), ignoring latency/queue depth and real bottlenecks 5) Treating pods as pets: writing to local disk instead of PVC/object storage 6) Secrets in env vars or in Git, no rotation, no RBAC boundaries 7) No network policies, everything can talk to everything, lateral movement is easy 8) Ingress/TLS as an afterthought: timeouts, headers, and cert renewals break prod 9) No observability: missing request IDs, no metrics per endpoint, logs not centralized 10) Using default namespaces and wide cluster-admin, then one bad kubectl ruins a day

I have been working on a product for indie creators. @indiehash is the agentic payment infra on solana. Upload premium content, share it with your audience and get paid in USDC on solana network. For the end user it's as simple as probe->pay->unlock premium content from your favourite creators. You can also host an api endpoint with some ai agent doing complicated work behind the scene and get paid per api call. No KYC. Instant payment on the solana network.

Uber-style system design: ride matching + dispatch. What I look for in interviews: 1) Core services: Location ingest, Driver state, Rider requests, Matching, Pricing/ETA, Dispatch, Trip lifecycle, Payments, Notifications, Support. 2) Hot path: driver GPS updates (1-5s). Don’t write every ping to SQL. Use stream + in-memory/grid index; async persist. 3) Data model: Driver(id, status, geo_cell, last_seen), RiderRequest(id, pickup, dropoff, constraints), Match(id, driver_id, request_id, score, expires_at), Trip(id, state machine, idempotency_key). 4) APIs (idempotent): POST /requests, POST /drivers/{id}/location, POST /match, POST /dispatch, POST /trips/{id}/events, GET /eta. 5) Bottlenecks: geo queries, fanout notifications, lock contention on driver assignment, retry storms, thundering herd on hotspots (airports). 6) Scaling tradeoffs: sharding by geo_cell, eventual consistency for location, bounded staleness, per-cell leaders vs global queue, push vs pull matching. 7) Failure ca

High-scale database modeling is mostly deciding what you refuse to do at runtime. 1. Start from access patterns: top 5 queries, write paths, latency SLOs. Model for those, not for elegance. 2. Pick your consistency boundaries early: one aggregate per transaction, clear ownership, avoid cross-service joins. 3. Denormalize on purpose: duplicate the 3-5 fields you need for reads; keep a source of truth + async repair. 4. Plan keys like you plan outages: avoid hot partitions, use time-bucketing, add random suffixes when needed. 5. Index budget is real: every index is extra write cost and storage. Track p95 write latency after adding one. 6. Version everything: schema versions, event versions, backfill jobs. Assume online migration with dual writes. 7. Model deletion upfront: TTL vs hard delete, GDPR, tombstones, and how it affects secondary indexes

Your Postgres database is at 95% disk usage. You delete 1 million rows but disk space doesn't decrease. Why not and how do you reclaim space? [We faced this last quarter]

10 super practical tips to ace 95% of your Senior Backend Engineer interviews in the age of AI in 2026 (and beyond).

You added caching and your memory usage grows until server crashes (OOM). Cache hit rate is 95%. What's wrong with your cache implementation? [Topic: Memory Management]

Just talked to a person via my 1:1 call and he said that after 9yrs of experience he is still not staff. He works at a reputed company but writing APIs in python. I suggested him to use AI to get better at system design etc and also get better at understanding the fundamentals. And tbh this is more common than people think. 9 years of experience does not automatically become Staff. A lot of people spend years doing scoped API work, closing tickets, staying reliable, but never intentionally building the skills that higher levels actually demand. Staff is not just writing more code. It is system design. It is driving ambiguity into clarity. It is taking ownership beyond your Jira board. It is influencing roadmap, mentoring people, seeing tradeoffs early, and communicating well with managers and teams. He said he is earning 27 LPA fixed which is not some disaster tbh. It is a decent base. One solid switch with better prep can move that to 40 to 45+ LPA very realistically. But the bigger point is this: - Stop thinking in "how do I escape next month" - Start thinking in "what can I become in the next 6 months" - Use AI daily. - Study fundamentals properly. - Practice system design. - Do Leetcode consistently. - Talk more with your manager. - Ask for ownership. - Get closer to architecture and decision making. Most careers do not change in one interview. They change when someone spends 6 months very seriously becoming a different engineer. Consistency looks boring in the moment. But in software, boring consistency is what usually creates the big jump. --- You can book 1:1 call if you want: topmate.io/abhishek_singh452 Practice System Design for free: 0xffdevs.com/design Read interview Experiences: 0xffdevs.com/interviews Follow pathways: 0xffdevs.com/pathways And invest and buy the community token on Solana: 0xffdevs.com/token --- Thanks for reading. Godspeed.