@recursiveagi
Account based inUnited States
About this account
- Account based in
- United States
- Connected via
- United States App Store
Account-level information from X, not a live location or the device used for a specific post.
Pinned Tweet
Lately, I have been trying to beat Google.
Ok not really xD
I have been building a search engine over niche web data and it's been one of the most fun projects I've ever worked on.
The idea of querying the internet exactly the way I want, at scale, has always fascinated me. Especially what I could do with it: look at 1,000 companies' job postings and find hiring trends using natural language, map competitive landscapes including the lesser known players, find alternative data for investing. Or just a personalized corner of the web, curated for me.
It's been an interesting journey.
Started out with 200 workers on localhost, rewrote the crawler from Python to Rust to get up to 1,600 workers in parallel. Moved it to the cloud. Hit challenges that made the whole Rust rewrite useless xD
Then I woke up to an abuse report from a French website with the words "European Commission", "appeal", "court" in it. For a second I genuinely thought I was about to have legal proceedings against me. Phew. Nevertheless it's been the most fun project I've worked on. It's surprising how you can cover a lot of information starting from a modest number of seed URLs.
I'll be sharing everything I learned in the open. Code will be open sourced soon and the search engine will be live too.
If you're curious how to build a distributed web crawler that covers 10M+ pages under $200/month, do read my blogpost.
Blogpost Link: lnkd.in/g7MsaXYA
hrishikesh kamath retweeted
OpenAI serving open-source models through Baseten is a big deal. They are embracing open source.
Until now, OpenAI's pitch was "use our models."
Now it is "commit your annual AI budget to us". We will get you the best model for each job, whether it's ours, someone else's, or open source.
So if you're a large bank looking to commit $100M in AI spend, you centralize with OpenAI. One throat to choke. GPT handles the credit memos. Open-source handles the KYC volume at a fraction of the cost. A post-trained / specialized model handles fraud.
This changes OpenAI's position. It stops competing only on whose model is best this season and starts competing to own the enterprise AI budget. A layer above being just an AI lab.
Also proves AI is positive sum, not zero sum between closed and open source!
hrishikesh kamath retweeted
Shower thought: Maybe PDFs should come with instructions for how they should be parsed — obviously, having the raw data is the best kind of instruction, but can be a pain to include correctly + there are ambiguities that instructions could easily resolve.
hrishikesh kamath retweeted
For Reference, Zhipu AI has -439% PAT margins and Minimax -2368% for FY2025. The two publicly listed pure play foundational model companies.
hrishikesh kamath retweeted
Everyday intelligence as @noahrshinn put it is an interesting way to view the contrast of what you would use for a personal assistant vs. frontier models you would use at work
General purpose auto routing is the biggest scam. It’s impossible to have a low latency model to know nuances of every single kind task to route it right. Big believer in domain specific routing, most applied ai companies are doing it right. Cursor being the earliest.
hrishikesh kamath retweeted
we’re probably the last humans who will ever be the smartest things alive
everything after this is just us managing something smarter
$META makes like $20 per month per user for US users only. Not sure if this can sustain browser use actions regularly. On the brighter side I am sure a lot of sites will become more agent friendly!
Best example for an asset that doesn’t go up much even after great news: robinhood:0xd0601ce157db5bdc3162bbac2a2c8af5320d9eec, one that goes up even with slightest good news and not down too much: $META
This is a heuristic I learned early in my career that has saved me a good deal of money.
Generally speaking:
If you are short an asset and it experiences a stream of negative news that fails to push the asset lower, get out.
If you are long an asset and it experiences a stream of positive news that fails to push the asset higher, get out.
hrishikesh kamath retweeted
i think everyone should build something at least once - an app, a newsletter, a tiny business, a youtube channel, anything
creating something nobody is obligated to care about teaches you more about people than a surprising amount of formal education does
hrishikesh kamath retweeted
Losses are a part of life
Missing out on the next big one is even worse than taking a loss.
Best of the investors have a 60-65% hit rate. Many retail folks need to internalise this
Yep it’s simple as that
This is a heuristic I learned early in my career that has saved me a good deal of money.
Generally speaking:
If you are short an asset and it experiences a stream of negative news that fails to push the asset lower, get out.
If you are long an asset and it experiences a stream of positive news that fails to push the asset higher, get out.
hrishikesh kamath retweeted
Just spoke to one of the big data labeling businesses.
Few interesting insights:
- They predict the majority of their revenue will come from Fortune 1000 enterprises, not labs in a few years
- They believe every company will want to own their intelligence, but owning intelligence does not necessarily mean using open source models
- A company’s evals will become their main proprietary IP given the improvement in agent performance after properly setting up & running internal eval environments
- Most enterprises haven’t graduated from coding agents and it’s largely due to not having the proper eval infrastructure to make non-Eng agents performant
SLMs are everywhere! Took me a Jev to realize.
This instance was caused by a hallucination (the model fabricated a proper noun) that was further amplified by its causal thinking trace. This was not a data breach, no user data was shared, and no user isolation boundary was violated.
Our users place a lot of trust in Instinct to keep their data private and secure. We've done a lot of work to make sure that users’ data are truly partitioned, from isolated sandboxes to short-lived local credentials to identity-signed tool execution.
Regarding hallucination prevention: We worked over the last 48 hours to build an active hallucination detection system that now scans and verifies every token that flows through the platform. This layer is powered by small models that are trained to predict and detect hallucinations caused by ungrounded claims, creative brainstorming in thinking traces, or rare random sampling errors. Our systems are adversarially trained and battle-tested by world-class agent exploitation security teams to ensure that they can detect very subtle and nuanced hallucination cases. This layer has the ability to steer or intercept Instinct from proceeding with the next thinking trace or tool call execution before it is generated or executed.
In the coming weeks, we’ll share a deeper dive into how we’re building a proactive engine to protect Instinct from hallucinations.
We take every opportunity to learn and create, and we’re excited by the performance and potential for this line of work. These models are going to continue to be hardened every day through continuous testing, sampling, and creative adversarial training.
