Agents love grep. So we built HypGrep: grep over huge text datasets in S3, no server. Built for code and logs full of symbols and syntax that BM25 engines choke on. Index is Parquet next to your data, compute on the client. 3.2M logs, $0.38/mo. github.com/hyparam/hypgrep

Someone leaked our architecture diagram 😲 What if our competitors learn that you don’t actually need a backend??

Today I'm excited to announce that we are launching @hyperparamapp, an AI-powered Swiss Army knife for massive LLM datasets. It lets you view, score, filter, label, and transform LLM data directly in the browser. I started Hyperparam one year ago because I knew that the world of data was changing, and existing tools like Python and Jupyter Notebooks were not built for the scale of LLM data. The weights of LLMs may be tensors, but the input and output of LLMs are massive piles of text. The training set of LLMs is a large corpus of text from various sources, meticulously cleaned and preprocessed. The output of LLMs is also text, and it’s being produced in even greater quantities than the training data. No human has the patience to sift through all that text, so we need better tools to help us understand and analyze it. That's why I built Hyperparam to be the first tool specifically designed for working with LLM data at scale. To accomplish this required rethinking how data analysis tools work. I started Hyperparam as a side project and wanted to see if I could build it entirely in the browser. No Python, no servers, just pure interactive web experience. I'm pretty excited about how it turned out. Hyperparam is fast, powerful, and easy to use. It can handle datasets with millions of rows of text and provides a rich set of tools for exploring and analyzing that data using LLMs agents for assistance. Every company is now producing volumes of LLM data. Chat logs, agent traces, coding agent logs, and more. Hyperparam is designed to help you make sense of all that data. If you're working with LLM data (and let's be honest... every company is producing LLM data now), I encourage you to give Hyperparam a try. It’s free while in beta. 🚀

What is the software equivalent of this? No dependencies, avoid abstraction, vertically integrate your software stack. Good engineering is subtraction.

Just wrote a new blog post: how I spent a year making the world's fastest Parquet loader in JavaScript, and all the optimizations that went into it. TLDR: Hyparquet can load a parquet file from S3 in 155ms. While duckdb-wasm took 3466ms the same file!

Shots fired at the @ApacheIceberg Meetup: "What I see is a community that's gotten way too complacent with complexity. I think there's a way you can do this a LOT more simply." ...so I ported Iceberg to JavaScript.

duckdb is great! but a couple points for hyparquet: - duckdb wasm is ~40mb (often larger than the data being loaded) - bundling wasm is a pain - hyparquet is tiny (10kb) and easy to deploy - if you want to minimize time-to-displayed-data in the browser, hyparquet usually wins.

"before we were largely in a compute-constrained environment. now we are much more data-bound."

Just found this very cool project OpenTimes where you pick a starting point, and it gives you travel time to all points around you. Uses hyparquet to efficiently fetch geospatial data and visualize it. I absolutely love this kind of browser engineering!

Anthropic is going to win because they understand the power of javascript

Created and released a new library today: GeoParquet on NPM It uses hyparquet to parse @GeoParquet files into geojson (for google maps, etc). Combined Hyparquet.js + a geoparquet file is often vastly smaller than an equivalent geojson file, and loads faster!

Cost per trillion input tokens (updated 2024-08-06)

I’m incredibly proud to release Hyparquet 1.0.0 —javascript parser for Apache Parquet files. Hyparquet is the world’s most conformant Parquet parser! It opens files that even PyArrow and DuckDB can't handle. I dare you to find a file that Hyparquet can't open!

"Weird how every Parquet parser for javascript has gone abandoned. I'm sure I'll be fine..." @hyperparamapp

As of today, Hyparquet, an open-source parquet parsing library I created, is being used by @huggingface in their dataset viewer! 4 months ago I started thinking about the problem of how you could view and explore massive text datasets with the interactivity of the browser. Most ML datasets today are in parquet format. Despite this, I couldn’t find a good browser-based parquet parser. (to be fair, parquet is a nightmarishly complicated format) Long story short, I created Hyparquet as an open-source parquet reader for javascript. I built it from scratch from the ground up, with no dependencies. It is lightweight, fast and has native support for viewing parquet files over http. This allows for streaming data to the browser, on demand, with no backend needed. I’m shocked by how quickly it has already seen adoption. In the 4 months since creating hyparquet, it’s been added by multiple projects including CanvasXpress and Hugging Face 🤗 (thanks @severo_dev!) Give it a star: github.com/hyparam/hyparquet