4 postswith this tag
We built a hand-crafted 42-question evaluation to measure complete lists, effective dates, legal references, multi-document queries, monitoring, and false premises. We ran it against full BM25 and a 38.2% vector index to set a baseline before finishing the embeddings.
Second round of the benchmark: 499 documents from the full corpus (1999–2026), 3,023 LLM-generated queries across 6 types, a BM25 baseline with FTS5, and evaluation of late chunking for pplx-embed-context. BM25 and embeddings turn out to be complementary by query type.
We compare 10 embedding models on speed, memory, and retrieval quality over real DOF documents. We test int8, binary, and Matryoshka truncation quantization. Public leaderboards didn't predict the winner.
We present the RAG chunker: a classifier that detects 5 structural patterns in DOF Markdown before applying the right split strategy.
