13 postswith this tag
How we built a local application to test the agent with new questions, stream its verifiable process, and store answers and feedback without waiting for the vector index.
How we built a bounded tool loop, turned the parts of a question into verifiable requirements, and evaluated its answers on the DOF.
How we separated document search from passage retrieval and built five deterministic, traceable, and evaluable tools on top of the DOF.
We built a hand-crafted 42-question evaluation to measure complete lists, effective dates, legal references, multi-document queries, monitoring, and false premises. We ran it against full BM25 and a 38.2% vector index to set a baseline before finishing the embeddings.
While the embedding run advances over 6.7 million chunks, we built the full-corpus FTS5 index (2.7 GiB) and ran the first real-scale BM25 evaluation. The v2 query set produced an MRR of 0.170; after fixing fake titles and ambiguous queries, v3 reached 0.366 and the partial hybrid smoke test 0.402. Along the way: a COUNT(*) that lies, 32 documents almost invisible to the index, and a token-pruning step that turned 21 hours of queries into 34 minutes without changing the metrics.
We built the production foundations on the 657,867 documents of the Official Journal of the Federation (DOF): the full compressed corpus occupies 3.52 GiB, the chunk index holds 6.73 million 91-byte recipes, and the binary-vector pilot confirms the vector index will fit in less than 1 GiB. Along the way: a GROUP BY that consumed 35 GB of disk, a document that became 21 GB of chunks, and a lesson in result parity.
Fourth installment of the benchmark: we built the compressed corpus on 10,000 real documents. sqlite-zstd compresses 10.9x with random access, chunks are stored as 110-byte recipes instead of text, the word-search index ended up 15x smaller than feared, and TurboQuant quantization ties full-vector quality. Everything fits in ~11 GB.
Third installment of the benchmark: we merge the BM25 and embedding rankings and the result beats both individually. We also measure how to index the full corpus from the Mac M3: what works (GGUF/Metal), what does not (large batches, fp16), and why jina's binary quantization decides the storage architecture.
Second round of the benchmark: 499 documents from the full corpus (1999–2026), 3,023 LLM-generated queries across 6 types, a BM25 baseline with FTS5, and evaluation of late chunking for pplx-embed-context. BM25 and embeddings turn out to be complementary by query type.
We compare 10 embedding models on speed, memory, and retrieval quality over real DOF documents. We test int8, binary, and Matryoshka truncation quantization. Public leaderboards didn't predict the winner.
We evaluated 8 chunking strategies (5 from Chonkie, 2 pipelines, 1 custom) on 1,000 Official Journal of the Federation (DOF) documents. No off-the-shelf library option preserved both document structure and the token limit.
We present the RAG chunker: a classifier that detects 5 structural patterns in DOF Markdown before applying the right split strategy.
