5 postswith this tag
Fourth installment of the benchmark: we built the compressed corpus on 10,000 real documents. sqlite-zstd compresses 10.9x with random access, chunks are stored as 110-byte recipes instead of text, the word-search index ended up 15x smaller than feared, and TurboQuant quantization ties full-vector quality. Everything fits in ~11 GB.
Third installment of the benchmark: we merge the BM25 and embedding rankings and the result beats both individually. We also measure how to index the full corpus from the Mac M3: what works (GGUF/Metal), what does not (large batches, fp16), and why jina's binary quantization decides the storage architecture.
Second round of the benchmark: 499 documents from the full corpus (1999–2026), 3,023 LLM-generated queries across 6 types, a BM25 baseline with FTS5, and evaluation of late chunking for pplx-embed-context. BM25 and embeddings turn out to be complementary by query type.
We compare 10 embedding models on speed, memory, and retrieval quality over real DOF documents. We test int8, binary, and Matryoshka truncation quantization. Public leaderboards didn't predict the winner.
We evaluated 8 chunking strategies (5 from Chonkie, 2 pipelines, 1 custom) on 1,000 Official Journal of the Federation (DOF) documents. No off-the-shelf library option preserved both document structure and the token limit.
