Lyon, France
Joined November 2012
- Tweets507
- Following2.5K
- Followers252
- Likes2.2K
Arnaud RΞNΛRD retweeted
In the 9.1 release, Elasticsearch now includes BBQ by default and ACORN for filtered vector search, delivering better results faster and at a lower cost. Learn more in our latest blog: go.es.io/471XRNq
Arnaud RΞNΛRD retweeted
We're now further taking advantage of it by amortizing the cost of some operations across many doc IDs github.com/apache/lucene/pul… or better vectorizing/pipelining other operations github.com/apache/lucene/pul…. Expect a new round of query evaluation speedups in 10.3.
Arnaud RΞNΛRD retweeted
Last month, Lucene changed query evaluation to work in a more term-at-a-time fashion within small-ish windows of doc IDs. This yielded a good speedup on its own (annotation IL benchmarks.mikemccandless.co…).
Arnaud RΞNΛRD retweeted
A nice optimization landed on the hash table that Lucene uses to build inverted indexes: github.com/apache/lucene/pul…. Some previously unused bits are now used to cache hash codes, effectively making collisions cheaper to resolve.
Arnaud RΞNΛRD retweeted
Lucene is getting faster at deep search by switching to a more efficient heap implementation to collect top hits. github.com/apache/lucene/pul…
Arnaud RΞNΛRD retweeted
I wanted to share what I learned from Tantivy's "Search Benchmark, the Game", so I set up GitHub pages and wrote two blogs, on general observations on the benchmark jpountz.github.io/2025/05/12… and how it helped drive performance improvements in Lucene jpountz.github.io/2025/04/12…
Arnaud RΞNΛRD retweeted
The search library benchmark from the Tantivy folks was just updated with Lucene 10.2 tantivy-search.github.io/ben…. Lucene now performs much better at the COUNT collection type, a bit better at TOP_K. Still somewhat slow at TOP_100_COUNT and phrase queries across all collection types.
Arnaud RΞNΛRD retweeted
I feel bad when I see users needing to shard their indexes only to not hit the 2B doc count limit. 2B was a lot when Lucene was created, not anymore. We should fix it. github.com/apache/lucene/iss…
full article: discord.com/blog/how-discord…
Arnaud RΞNΛRD retweeted
Finally took the time to read this great blog on BM25F by @julietibs sourcegraph.com/blog/keeping…. Great to see more usage of BM25F, lots of applications are still combining scores across fields via sum/max when BM25F would be a more robust choice.
Arnaud RΞNΛRD retweeted
This is now live on nightly benchmarks, with a 32% speedup on primary-key lookups benchmarks.mikemccandless.co… and a 5% speedup on fuzzy queries. The main benefit that Lucene users will notice is likely faster indexing with explicity document IDs
#Apache #Lucene will soon have a faster and smaller terms index!
This is a complex part of Lucene, and a major hotspot for terms heavy use cases like (primary) key/value store (~34% speedup, but results are preliminary!).
Lucene's pluggable Codec API makes experimentation like this possible.
github.com/apache/lucene/pul…
Arnaud RΞNΛRD retweeted
Now live! Elastic 9.0/8.18 includes faster quantization, Elastic Distributions of OTel and LLM observability, the GA of Attack Discovery and Automatic Import, major enhancements in ES|QL like JOIN, and more.
Learn more → go.es.io/42tjpyA
Arnaud RΞNΛRD retweeted
It's time to redo benchmarks! #Lucene 10.2 was just released, with
- huge speedups to non-scoring boolean queries, range queries and filtered vector search,
- better merging defaults for faster search,
- much faster merging of vectors
And more...
lucene.apache.org/core/coren…
Arnaud RΞNΛRD retweeted
Lucene will now intelligently merge HNSW graphs: elastic.co/search-labs/blog/… Now indexing and merging is much cheaper, reducing the compute required and improving indexing throughput:
Arnaud RΞNΛRD retweeted
Guo Feng contributed a 2.5x (!) speedup to #Lucene's numeric range queries by using vectorization. HZ sped up query evaluation, ID sped up decoding data from the index. Lots of great performance improvements coming in Lucene 10.2.
Arnaud RΞNΛRD retweeted
#Lucene had a ~15% speedup on conjunctive queries and ~20% on disjunctive queries recently by organizing the code in a way that is more friendly to the JIT compiler. See annotations HJ and HM at benchmarks.mikemccandless.co… and benchmarks.mikemccandless.co….
Arnaud RΞNΛRD retweeted
This was the second attempt to speed up Lucene's skipping. In the first attempt @jpountz tried branchless binary search, inspired by Tantivy (thank you @fulmicoton!).
Arnaud RΞNΛRD retweeted
Here's an exciting #Lucene change, using vectorized (SIMD) CPU instructions via Java's Panama Vector API to accelerate skipping (finding a specific integer in an int[128] array).
The iterations on this PR show how hard it is to safely tap into low level SIMD instructions from way up in javaland ("lanes", ARM vs X86, trueCount vs firstTrue)...
Thanks you @jpountz and @rcmuir!
github.com/apache/lucene/pul…
Arnaud RΞNΛRD retweeted
Oh, we have been cooking. In Elasticsearch 8.16 and Lucene 10, we are adding Better Binary Quantization (bbq) that beats Product Quantization for large scale vector search
✔️ Fast queries
✔️ Builds indices in lower memory environments
✔️ Built on solid research
🧵 [1/5]
Arnaud RΞNΛRD retweeted
The biggest problem of #Java is poor perception. It's technically super-solid, but too often folks discard it based on misconceptions or information outdated years ago.