@SpiralDB
Account based inUnited States
About this account
- Account based in
- United States
- Connected via
- United States App Store
Account-level information from X, not a live location or the device used for a specific post.
Multimodal data platform. Petabytes in, signal out. From the creators of @vortexdotdev.
NYC & London
Joined March 2023
- Tweets116
- Following10
- Followers886
- Likes223
The team cooked with this one.
Spiral now has first-class support for geospatial data types, with optimized query operators for things like spatial joins.
If you work with satellite images, vector data, or anything multidimensional, you're gonna wanna check this out.
spiraldb.com/deep-dives/geom…
Your data shouldn’t have to fit your file format.
We built Vortex so one columnar format can fit any shape of data and still be fast and small.
Check out the post by Connor Tsui: spiraldb.com/blog/vortex-one…
Spiral retweeted
This is exactly why Spiral is focusing on AI data infrastructure. Training on the right data is a surprisingly outsized lever
Pretraining progress seems to be coming mostly from data improvements.
@who_is_jerbear and I pretrained combinations of year-representative open model recipes and data corpuses across 2019 to 2025 at various small scales.
Data improvements contributed 3.24x as many compute multipliers as model improvements did (12.0x vs 3.7x).
And the gains stack independently - a better dataset helps every architecture about equally, and vice versa.
Here are full results, plus what we think this means for the future of AI progress:
dwarkesh.com/p/pretraining-p…
Always Be Spiralin'
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.
The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.
The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
Check out the QCon London 2026 talk by @onur_satici "From S3 to GPU in One Copy: Rethinking Data Loading for ML Training" spiraldb.com/blog/from-s3-to…
Spiral retweeted
great pattern! powers a lot of @SpiralDB under the hood, with a few twists:
- journaled data structures enable time travel and reversibility (why -> spiraldb.com/blog/reversibil…)
- multimodal storage engine adds a cool design challenge: you need a query language that offloads heavy, complex IO orchestration entirely to the engine
i'm experimenting with a new substrate for distributed systems: s3collections
durable data structures (lru, queue, tree) shared across a fleet with a simple API
metadata lives in SlateDB, larger data stays as plain blobs. nothing revolutionary
github.com/DamianB-BitFlippe…
"You probably do need a custom file format. You just shouldn't have to build one from scratch."
I’ve written enough file formats to know the bytes are the easy part.
I/O, pruning, scheduling, pushdown eat the weekends.
I built Vortex so I never have to do this again!
spiraldb.com/blog/vortex-the…
Spiral retweeted
their work is excellent! I always viewed what we do at @SpiralDB as redesigning an end-to-end data platform with considerations that (1) data must be training-ready at all times and (2) multimodality must be first-class.
export into a training format or multimodal support through file system APIs is never going to enable "researchers can now ingest, curate, and experiment with orders of magnitude larger data without rebuilding the path each time"
When people talk about robotics, they usually talk about models, data, or hardware. Few people talk about the infrastructure that lets you iterate on all three quickly. Today we're publishing how we trained Dyna-2 on over 1,000,000 hours of egocentric video, repeatably. At this scale, most of what worked at ten thousand hours did not hold up:
• ingestion throughput was capped at 14,000 episode-hours per week — a million hours would have taken over a year
• building a training manifest took 48 hours before a run could even start
• reading a petabyte from cloud storage during training left GPUs exposed to latency and packet loss
🧵
Spiral retweeted
💯 your data should always be training-ready.
"training can consume data quickly enough for the dataloader itself to become a bottleneck" yes! this was the core thesis behind some of our design decisions at @SpiralDB
At a million hours, building a training manifest took 48 hours before a run could even start. We moved manifest construction from a file-by-file walk to a single SQL query over a data warehouse:
• manifest build time: ~48 hours → under 1 minute
• manifest load time: 737s → 12.4s via memory-mapped columnar tables
• memory footprint per node: 2,151 GB → 218 GB
