managing context windows
Copenhagen
Joined May 2018
- Tweets124
- Following1.5K
- Followers273
- Likes876
Maxim Khomiakov retweeted
saw a good comment on “software is dead”
technically anyone can 3d print anything they want. they can either get a printer themselves or just print stuff online
in reality, buying mass produced products is both cheaper and better quality. people couldn’t care less about making everyday products “personal”
vibe coding is the new 3d printer
> we must write our text ourselves, and LLMs could only be used to review and give suggestions
Absolutely, nobody wants to read sloptent - accelerate experiments, the grunt work - sure. But artisan crafts are artisinal for a reason.
One sad realization about that however, we're just feeding more quality datapoints to the next generation llms, so if one would want to actually fight the slop, we would need to poison the content in a way for llms to struggle pretraining on them - and then thats just another headache.
Maxim Khomiakov retweeted
We are releasing the latest version of BrokenArXiv and ArXivMath! These benchmarks now focus on conjectures that were refuted in the last month on ArXiv, and models are executed within a harness instead of directly via API.
Performance remains impressive, with GPT-6 Astra on top
1/5 the size and outperforming K3?
mp.weixin.qq.com/s/qg0NU3NNU…
DeepSeek V4.1 Flash 为 552B 参数的 MoE 模型，采用了全新的 Causal-Encoder-Decoder 结构，输入和输出不对称，输入激活只有 8B，输出激活 16B，成本显著低于已知的同尺寸模型。同时，V4.1 Flash 还采用了新的预训练方式、经过了更大规模的强化学习后训练，在基准测试中，成功超越了包括 DeepSeek V4 Pro 在内的一众旗舰模型的智能水平。
Overprovisioned compute capacity? No slowing down just yet
This model represents a step-function improvement on many benchmarks, and its training is ongoing.
Our internal model group arrived at the Navier–Stokes solution in 88 hours, using around 10,000 coordinating AI agents.
Throughout the effort, we maintained the strict safeguards—including monitoring and isolation—that we apply to all our frontier evaluations.
Keeping a drawer of unsolved puzzles is indeed one of the best evals
Astra does appear to check out as a step change. Many vibe checks on Astra (or any new model) throw some grand task like an epic 3D game, etc. I've got a more boring approach.
If I've got some tasks that are stalled waiting on my intervention / focus time because bouncing between say Fable 5 and 5.6 Sol is still yielding crap, I throw the blocked tasks at the new model. If it cuts through the noise and pulls out a concise, functional, clean result, it might be a winner. Fable 5.1 stayed in ping-pong mode. 6.0 Astra 'got it.'
I've run into this pattern a few times over the past 18 or so months as the models keep improving. It's funny how used to the new capability you become though, delegating more and throwing harder tasks at the models, then getting frustrated once more as they end up stuck in a shit ping-pong at some point... until next model at least.
Maxim Khomiakov retweeted
Most latent generative models still follow the same two-stage recipe: train a VAE for reconstruction, freeze it, then force the generative model to learn whatever latent space it gets. But reconstruction-optimal latents are not necessarily generation-friendly — so why not let the generative model shape the latent space directly?
ByteDance Seed presents GenFirst to make stable end-to-end latent learning work.
This work revisit the VAE objective and identify a prior–entropy imbalance as the key cause of latent collapse in naive end-to-end training. After fixing collapse with explicit entropy preservation, they find that generation and reconstruction have strongly asymmetric learning dynamics.
This leads to Generation Before Reconstruction (GenFirst): first let the generative objective shape a generation-friendly latent space, then strengthen reconstruction to recover visual details.
The results are strong. With end-to-end SiT + VAE training, GenFirst achieves 0.97 gFID with CFG / 1.45 without CFG on ImageNet-256, and reaches the final SiT result with at least 70× fewer training steps. This is the first diffusion model to reach FID < 1.0 without an FD loss. On text-to-image generation, it reaches 0.90 GenEval.
More importantly, this end-to-end framework goes beyond image generation: it learn a shared visual latent space that supports generation, representation learning, and reconstruction, and further extend end-to-end latent learning to continuous text latents, enabling unified text-to-image and image-to-text diffusion modeling.
GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling
Paper: arxiv.org/abs/2608.29335
Project: tom-zgt.github.io/GenFirst/
HF Daily Paper: huggingface.co/papers/2608.2…
Seq-to-seq is the answer to the universe
Maxim Khomiakov retweeted
Why pay full compute for pixels you're not even looking at?
In our new work, Foveated Diffusion, we introduce a new concept for efficient image and video generation, motivated by how the human visual system works.
(See full thread below)
High-resolution image and video generation is hitting a wall because attention in DiTs scales quadratically with token count. But does every pixel need to be in full resolution?
Introducing Foveated Diffusion: a new approach for efficient diffusion-based generation that allocates compute where it matters most.
1/7🧵
Maxim Khomiakov retweeted
When I worked at Apple, I remember seeing code that was 15 years old. Every file had a comment at the top with the date and the author’s name. You’d open a file and read through it, sometimes touching a feature no one had touched in a decade.
Everything felt very permanent, like a house made of bricks. Everyone who made a brick put their name on it before slotting it into the house. Even in a massive company, you were proud of your bricks. Each one fit so satisfyingly into the bricks around it.
When someone was renovating a room you had worked on years ago, they’d reach out and say, “I found this brick with your name on it. Can you tell me what it was doing here?”
Sometimes old bricks would get taken out during a renovation, but they were usually kept in a pile in the corner to reuse in the future.
Now you can will whole houses and neighborhoods into existence. You don’t even see the bricks go in. They just tumble in from the sky. You walk into the new house and wave your hands to push walls back, throw up a chimney, or drop in a kitchen.
It's so fucking cool. But you forget what a brick feels like.
And it’s hard to know how sad to be about that. The houses are better, they go up faster, and building them increasingly feels like magic.
Still, every once in a while you miss holding a brick.
Good moment today to remind everyone to stay frosty with AI. I was working on making a server stoppable via the CLI, and Codex decided the best path forward was to implement an unauthenticated network API to every server to shut itself down. Excellent work.
I responded with "I don't know about that." (I usually give more direct feedback but this was so absurd) and it thought for like 20 seconds and responded "Server stopping should not be part of an unauthenticated network API." Yes. lol.
By the way, the result I was looking for was that we automatically start a locally discovery Unix-socket bound server so I wanted it to discover that (using prior mechanisms already made) and gracefully stop that.
The driver is absolutely accountable here (me). My lazy prompting led AI to think I wanted to be able to stop ANY configured server I was talking to. Shame on me for thinking it was obvious here. But, I also reviewed it and found this idiocy, so, that's where good AI drivers come in. Also me.
No matter the model capacity, if the same input is mapping to multiple outputs, you will get average results. This is why the ability to provide essential and accurate details, that activate the right part of the model's weight space is so important.
Maxim Khomiakov retweeted
GANs are back!
"Scalable GANs with Transformers" shows that GANs are finicky but with the right setup, they can be scaled.
arxiv.org/abs/2509.24935
Maxim Khomiakov retweeted
In the world of agents benefits of static typing are quickly increasing over the downsides.
Amazingly quick shift that is an earthquake in the market of programming languages
Maxim Khomiakov retweeted
"Latent Reasoning with Normalizing Flows"
NF-CoT makes latent reasoning feel native to LLMs. So instead of forcing every intermediate thought through verbose CoT text, it learns compact continuous thoughts with a normalizing flow inside the causal LLM stream.
The key move is that latent thoughts become sampleable, scoreable, and RL-trainable like tokens, with exact likelihoods and KV-cache friendly decoding.
This beats explicit CoT and prior latent methods, while using 64 latent tokens to compress roughly 385 CoT tokens and running much faster than diffusion-based latent reasoning.
pretty impressive if it works as described
What if computer-use agents could do real work?
We built Gym-Anything: a framework that turns any software into a computer-use agent environment.
We used it to create CUA-World: 200+ real software, 10,000+ tasks and environments, across all major occupation groups, from medical imaging to financial trading.
🧵
This is interesting - Loop over a specific range of layers in the middle of your LLM, and performance increases.
Maxim Khomiakov retweeted
my results on AI for autonomous progress:
we ran codex+claude for 43 days straight to build a System Verilog compiler/simulator
normalcomputing.com/blog/bui…
