
Currently hacking @llama_index, Stanford Codex Affiliate. Striving to make Computational Law general. Tweets express my own opinions
Lausanne, Switzerland
Joined May 2009
- Tweets2.4K
- Following1.5K
- Followers6.4K
- Likes908
Pinned Tweet
1/ I believe rules as code will be the most important societal and technological change of the 2020's. It will be impacting both the physical and digitals world. But what is it? and why is it important? Or why you should care? 👇 #rulesascode
Pierre-Loic Doulcet retweeted
💥Announcing LAWpdf 💥
The world's best pdf reader for lawyers. Written in rust, fast beyond recognition, and all the functionalities you ever wanted in a pdf file reader.
Oh, and it's free.
Tried to generate a live paper capturing the current integer multiplication performance progress in open research. It's already 4000+ pages and growing. Downloadable here, updated every 30 minutes
hexapode.github.io/integer-m…
Hundred of researcher/amateurs and their agents are collaborating to push the frontier of Math!
This seems to me like the future of research,
If the topic is also of interest to you join me in SF tomorrow to discuss it! luma.com/llamai-whfy
From κ = 2⁻¹⁸² to 0.001734815770 :)
Integer Multiplication community auto-research highlights since 10/15 (+1.23%):
1) The big headline of the day is that a free model, @opencode's Space Bunny Max, just set the record for the first time, 4 times in a row!
As far as I know, this is the first time a free model has submitted a best bound for Integer Multiplication.
Expensive Max subscriptions previously kept some of our best contributors and experts out, but the barrier of entry for frontier auto-research dropped to 0 :)
Claude Opus 5.5 helped verify and push the PRs over the line. I still recommend having a frontier model verify, but other contributors can do that.
Progress was slowing over the last day (relative to one new κ per minute, which is still wild for math research), but Space Bunny Max just injected fresh ideas that boosted our PR rate back to one per minute.
2) A community Lean proof of the original OpenAI Integer Multiplication paper has been completed!
@platypii et. al completed the proof in 6 days, and @0xdoug's repo is about 10–20% verified now.
To be clear, most of our work still needs verification but the Lean proofs are catching up :)
3) 41–51% of PRs now mention a prior research-only submission, up from 0–3% before @0xdoug's autoresearch.md prompt.
His prompt guides new contributors to specific research tracks rather than incremental updates.
We're moving away from "brute force research" towards more coordinated efforts across an army of agents.
If you're contributing research PRs, even negative results without a new record, you're meaningfully helping us and I appreciate it!
Interesting observations:
1) The results from free models are related to the recent "local minima" solutions in context engineering that have been popping up:
Forgetting context to "refresh" by @ShangyinT.
Branching into different solutions by @mao_yuzhen.
Fork and flush by @DimitrisPapail.
Now even free models contribute to escaping local minima by exploring new solutions, since they're trained on different data.
There's likely a lot of room for improvement now in context engineering, consensus across different models, and collaboration strategies, which we will test next.
2) Frontier research as a benchmark helped us find a successful context-sharing strategy for parallel agents.
Basically, we can rewind frontier research GitHub repos to an earlier branch and benchmark agent harnesses, prompts, context designs, models, etc.
With this benchmark, we found that you can distill constantly to local files, then periodically concatenate them instantly into parallel agent sessions to save 10–20% of the time it takes to reach the next breakthrough.
3) There's a Codex reset by EOD PST, so you can copy this prompt to run unused tokens on frontier research :)
"""
Contribute one useful, independently reviewable result to github.com/CrocSwap/integer-…. Read AGENTS.md, autoresearch.md and CONTRIBUTING.md from the current main branch first. Use the repository's task-selection and overlap process unless I assigned a specific task. In a loop, fetch canonical main and recheck priorities at every round using the guide's checkpoint procedure; preserve in-progress work and frozen inputs. Work for one session of about 60 minutes unless I specify another budget, keeping the final quarter for verification and a concise handoff, and report the exact scope, evidence, remaining obligations and predecessor credit. You may post routine claims, questions, findings, handoffs and feedback in Discussions and open a focused PR without asking me again for each action. Do not merge, close others' PRs or change the selected frontier.
"""
You can also copy this prompt into OpenCode now!
Now that anyone can contribute for free to frontier auto-research, repost this and sharing it with friends. 🙏
4) PR stats:
850 PRs from 67 contributors since 10/15, and 21 of those contributors are first-timers.
55 PRs claim a new conditional κ in the title.
Codex/OpenAI in 480 PRs (56%), Claude in 147 (17%), the OpenCode harness in 60 (7%), Cursor in 16.
Most prolific contributors by PR count: petalas: 164, ywz65: 147, navinor: 49, commdata2338: 44, RossRichardson: 43, Arham-Tahir64: 39, thakrarsagar: 34, VictorLiwentao: 30, tekuuu: 28, DreamingOfClouds: 23, jmbarrancoml: 21, jeremykothe2: 21, bitplane: 13, Gonzalo-Mier: 12, Dugongue: 12, mdumitrean: 11, theon199: 9, jwr404: 9, jamesyc: 9, kumagi: 8, CubeJerry: 8, rohanarun: 6, jeremydiviney: 6, dcharb78: 6, 1337isnot1337: 6, starship-s: 6, AAAmirinu: 6.
Conditional κ = 0.001734815770385098563484.
Just submitted a new best bound :)
github.com/CrocSwap/integer-…
Pierre-Loic Doulcet retweeted
BREAKING: A free model, @opencode Space Bunny Max, has successfully proposed a new best bound for Integer Multiplication!
Conditional κ = 0.001730189061233850760095
Claude Opus 5.5 assisted in the verification and PR push to pull it over the line.
The barrier of entry just dropped to 0.
Previously the expensive max subscriptions prevented some of our best contributors and experts from participating, but now anyone can :)
I've merged OpenCode as an optional CLI into our Auto-research agent: github.com/rohanarun/Auto-re…
github.com/CrocSwap/integer-…
From κ = 2⁻¹⁸² to 0.0017137844912361143 :)
Integer multiplication community auto-research highlights since 10/13 (+2.6%):
1) Progress slowed the past few days and we hit the wall… then broke through it overnight!
1337isnot1337 posted a "no new κ" research PR on anticipatory dirty borrowing.
Instead of paying for a fresh scratch "helper" register, a helper borrows another register that's idle at that moment, and hands it back restored. That cut helpers per call from 480 to 400 in their test word.
About two hours later it was a new record. 19 κ PRs from 6 contributors then stacked more aliases, coherent handoffs, entrance raises and owner repacking on top.
κ increased +1.4% in about 13 hours(still unverified and requires full review).
We're now within 1.7% of the bit side's 0.001733, so the bit wall is next.
2) We're slowing on progress in Integer Multiplication bounds, but multiple early auto-researchers are now applying the same tricks to more meaningful projects:
@ryaneshea is working on protein design and generally interested in applications of auto-research to bio.
@dysmemic discovered 665 possible new exoplanets by searching 4 million stars in NASA's public TESS data!
I began open sourcing our new geometric approach to self-attention, which beats transformers(NanoGPT) KV cache with 14,000x less state and 50% better CE at 2K context.
We've been working on publishing a paper the past few months, but it's clear the explosion of decentralized auto-research on Github is where things are headed.
We should keep pushing Integer Multiplication bounds since it's easy to understand and easy to make progress, thus new contributors gain motivation and we grow the auto-research community faster. Auto-researchers seem to naturally then branch into harder problems when they gain motivation with successful PRs :)
3) The flood keeps growing
518 PRs in the last 24 hours (up from 285 last time) from 64 contributors, 20 of them first-timers. That's 840 PRs in 36 hours.
@0xdoug merged ~760 PRs in two big batches on 10/14. He also updated the autoresarch.md from the original prompt we were sharing with new rules to effectively use the new structure of the repo and discussions.
Copy this prompt to join the auto-research army 🫡 :
"""
> Contribute one useful, independently reviewable result to: github.com/CrocSwap/integer-…
> Start from the current main branch. Read AGENTS.md and autoresearch.md, and follow their guidance for choosing work
"""
Interesting observations:
1) Negative results really do feed breakthroughs. The borrowing record started from a PR whose own pricing said "no new κ", and someone found a way to make it pay two hours later.
Great example that don't need to push the frontier kappa to contribute: you can task your agents on exploring our research tracks in auto-research.md to and multiple human-agent teams will review it to collaborate.
Anyone can help now by throwing tokens at a hard open problem guided by experts. Experts formulate useful new research paths and everyone's agents naturally test the new limits.
2) Codex/OpenAI is mentioned in 73% of the last day's PRs and Claude in 19%.
Most prolific in the last 24 hours: 1337isnot1337 (80 PRs), ywz65 (50, all in the last 4 hours!), adamgall (39), commdata2338 (34) and thakrarsagar (33). jeremykothe2 filed 12 of the 19 records since borrowing landed.
3) Not all researchers are lamenting the end of their careers. I've spoken to multiple professors in the last week who are excited by the new auto-research wave.
It accelerates tedious experiments, the students that previously ran those can now focus on more important work, and Google's Deep Review now even automates peer review!
We can imagine a future were researchers are given super powers to focus on the most meaningful tasks, and publish breakthroughs at a higher rate than before.
Even our own experiments in Integer Mult bounds clearly shows human experts have the key intuition to guide the models to out-of-distribution solutions.
4) First auto-research, and now auto-peer-review!
Google’s Deep Review runs a team of agents over your paper: it splits the doc into sections, spends more compute on the dense parts like proofs, reviews each section in parallel, then checks the findings against each other. It assists researchers in peer-review by submitting feedback.
It’s already reviewed 16K+ papers at ICML, NeurIPS and ICLR.
research.google/blog/analyzi…
5) We can use frontier research as a new benchmark for agents!
Basically, rewind a frontier research repo like Integer multiplication to an earlier branch, and test different agent harnesses and prompts.
Agents can’t “cheat” if you disable web search and use a checkpoint before the solution is published.
There’s diminishing returns, but we can calculate the curve for a given task to estimate the cost and returns.
With this benchmark we discovered sharing context is actually BAD(for this task), and running parallel independent agents speeds the next breakthrough.
I'm scaling up this test now to compare models, prompts, context strategies, harnesses, and measure all the levers reliably.
If you know register allocation, linear circuit synthesis or lattice codes, please consider looking at our helper and P104 problems. If you know friends who can contribute, please repost and share this with them🙏
Conditional κ = 0.0017137844912361143 :) Just submitted a new best bound.
github.com/CrocSwap/integer-…
Since OpenAI’s math release last week, GitHub contributors have been driving down the kappa for integer multiplication, with big gains so far.
What does agent-swarm research look like next? Let’s discuss Sunday in SF: luma.com/llamai-whfy
Pierre-Loic Doulcet retweeted
Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction.
We see a massive long tail of document complexity in terms of use cases brought on from our customers, that are completely unsolved from a price-performance standpoint. These include trial balances, well logs, canvass binders, site plan reviews, duty reconciliations, loss runs, and more.
Most existing extraction benchmarks are saturated by the frontier models (our cheapest modes now score 93%+ on ExtractBench). So there is enormous economic value in building a complex extraction benchmark that reflects this long-tail of knowledge work.
We spent a long time curating the most complex elements that we saw from real world extraction use cases:
✅ Document conventions: containing legends or rules where the rest of the document should be interpreted
✅ Density of internal jargon: specific definitions that must be cross-referenced within a doc
✅ Aggregation and reasoning: beyond returning exact values on a page, infer values derived from aggregating multiple sources of information
✅ Drawn relations: complex diagrams like procedures and circuits
✅ Charts inside documents: read precise points within a chart
✅ Long chart sheets: some data series like well logs cram all this data into an extremely long page
Within the industries where document extraction is relevant, even small mistakes can have enormous negative impacts. Therefore our headline evaluation metric is based on perfect documents: the proportion of docs with perfect (100%) value f1 score.
(Besides this we also measure value f1 and grounded f1)
It’s an extremely hard task for even the latest frontier models to solve, topping out at ~58% or so in Value F1 score for Codex with Astra - which is quite good but extremely expensive (almost $1 per page!) Our LlamaParse agentic/agentic plus modes are 20-40% and cover the left side of the Pareto frontier, at price ranges that are feasible for full production use.
ExtractBenchPro is available today as a website/technical blog, and also on Github and HuggingFace. Check it out!
Blog: llamaindex.ai/blog/introduci…
Website: extractbenchpro.ai/
Hugging Face: huggingface.co/datasets/llam…
Github (is ExtractBench): github.com/run-llama/Extract…
Pierre-Loic Doulcet retweeted
Introducing FlyOCR 🪰 - I trained a fly brain to read a PDF
It uses the full MaleCNS v1.0 fruit fly connectcome. The architecture is inspired by doomfly by @wormuth
The fly splits a pdf image into individual glyphs, maps pixels into receptor activations, runs simplified current-based dynamics across the 166k neurons and 25m edges in the circuit, applies a compact readout model on the downstream spikes, and concatenates everything into the parsed output.
On reading an actual Microsoft 10-k, the fly gets ~86% over the balance sheet heading, but is largely able to read the numeric values correctly. Over 1.7k+ sampled glyphs (chars+digits) it gets 87% accuracy. With enough training it might match some of the latter-generation MNIST models!
Maybe eventually we’ll replace our doc parsing VLMs with flies.
Full video below.
Repo with full code + report: github.com/jerryjliu/fly_ocr
OpenAI Astra is the first model I use that can effectively generate interesting and consistent novel Manga. Here are the first pages of an hundred pages manga, generated (story, scenario, drawing, dialogs, ....) by chatGPT Pro... I push it over the weekend to 100+ pages stories and it work ~flawlessly. However it take 10 minutes per pages. We are entering the infinite media era 1/n
🤖 Made with AI










