From Automated Evaluation to Human Testing of the DOF Agent
How we built a local application to test the agent with new questions, stream its verifiable process, and store answers and feedback without waiting for the vector index.
Open research on Mexico's Official Journal
Making Mexico's regulatory memory searchable.
We document how to turn Mexico's Official Journal into a structured, verifiable corpus for artificial-intelligence retrieval systems.
Research notebook
Findings and technical decisions
An Agent That Knows When It Is Missing Evidence: First Runs on the DOF
How we built a bounded tool loop, turned the parts of a question into verifiable requirements, and evaluated its answers on the DOF.
From finding documents to finding evidence: five tools for querying the DOF
How we separated document search from passage retrieval and built five deterministic, traceable, and evaluable tools on top of the DOF.
Eval v4: finding the document is not enough; the evidence must be found too
We built a hand-crafted 42-question evaluation to measure complete lists, effective dates, legal references, multi-document queries, monitoring, and false premises. We ran it against full BM25 and a 38.2% vector index to set a baseline before finishing the embeddings.
The First Real-Scale Evaluation: BM25 Against 657,867 Documents, from 0.170 to 0.366 After Fixing the Set
While the embedding run advances over 6.7 million chunks, we built the full-corpus FTS5 index (2.7 GiB) and ran the first real-scale BM25 evaluation. The v2 query set produced an MRR of 0.170; after fixing fake titles and ambiguous queries, v3 reached 0.366 and the partial hybrid smoke test 0.402. Along the way: a COUNT(*) that lies, 32 documents almost invisible to the index, and a token-pruning step that turned 21 hours of queries into 34 minutes without changing the metrics.
657,867 documents later: the full corpus in 3.5 GB, 6.7 million chunks, and the bugs that only appear at real scale
We built the production foundations on the 657,867 documents of the Official Journal of the Federation (DOF): the full compressed corpus occupies 3.52 GiB, the chunk index holds 6.73 million 91-byte recipes, and the binary-vector pilot confirms the vector index will fit in less than 1 GiB. Along the way: a GROUP BY that consumed 35 GB of disk, a document that became 21 GB of chunks, and a lesson in result parity.
Storing 31 GB of Text in 3 GB (and Recovering Every Byte): the Storage Proof of Concept
Fourth installment of the benchmark: we built the compressed corpus on 10,000 real documents. sqlite-zstd compresses 10.9x with random access, chunks are stored as 110-byte recipes instead of text, the word-search index ended up 15x smaller than feared, and TurboQuant quantization ties full-vector quality. Everything fits in ~11 GB.
Hybrid search: how to combine BM25 and embeddings (and make them fit on a laptop)
Third installment of the benchmark: we merge the BM25 and embedding rankings and the result beats both individually. We also measure how to index the full corpus from the Mac M3: what works (GGUF/Metal), what does not (large batches, fp16), and why jina's binary quantization decides the storage architecture.
Embedding benchmark round 2: BM25, late chunking, and LLM-generated queries
Second round of the benchmark: 499 documents from the full corpus (1999–2026), 3,023 LLM-generated queries across 6 types, a BM25 baseline with FTS5, and evaluation of late chunking for pplx-embed-context. BM25 and embeddings turn out to be complementary by query type.
Embedding Benchmark: 10 Models Compete to Understand the DOF (and a 0.6B One Wins)
We compare 10 embedding models on speed, memory, and retrieval quality over real DOF documents. We test int8, binary, and Matryoshka truncation quantization. Public leaderboards didn't predict the winner.
Custom vs Chonkie: why we built our own chunker for the DOF
We evaluated 8 chunking strategies (5 from Chonkie, 2 pipelines, 1 custom) on 1,000 Official Journal of the Federation (DOF) documents. No off-the-shelf library option preserved both document structure and the token limit.
Pattern-based chunker: classifying 131,000 DOF documents before splitting them
We present the RAG chunker: a classifier that detects 5 structural patterns in DOF Markdown before applying the right split strategy.
DOF Markdown Structure: How Viable Is Heading-Based Chunking?
Analysis of the structure of 26,607 medium and large DOF documents (2020–2026) to determine the RAG chunking strategy.
Gemini 2.5 Flash Lite on 100 DOF images: a real-world scale test
We tested Gemini 2.5 Flash Lite with the v3 prompt on 100 random images from the Official Journal of the Federation. Zero errors, 2.3s average per image, and an estimated $41 USD to process the ~97,000 images in the full corpus.
Iterating the prompt: second round of VLM model comparison
Second iteration of the experiment: we changed the prompt, adjusted the test images, and replaced Qwen with Grok and Gemma. We compared 6 models on 14 images from the Official Journal of the Federation (DOF).
Who Describes DOF Images Best?
We compare 6 vision models (Gemini, GPT, Qwen, Claude) on the task of generating image descriptions for RAG indexing of Mexico's Official Journal of the Federation.
657 thousand documents later: how we converted ALL of the DOF to Markdown
From 657,867 .doc files from Mexico's Official Journal of the Federation to clean Markdown: tools, results, and what's left to do.
Four steps to process the DOF: conversion, cleanup, analysis and structure
From the downloaded WORD file to structured Markdown ready for embeddings: a walkthrough of our complete processing pipeline that includes LibreOffice conversion, custom LUA filters, Gemini image analysis, and a robust directory architecture.
From PDF to WORD: when computational cost dictates a change of strategy
How the reality of massive document processing led us to rethink our strategy: from downloading complete PDFs to obtaining segmented WORD files, dramatically reducing computational cost without sacrificing quality.
Storage Projections for DOF-RAG
A detailed analysis of storage projections for the DOF-RAG project, evaluating different embedding dimensions and their scalability implications over a 25-year horizon.
The Battle of Embeddings: When Three AI Models Compete to Understand Governmental Spanish
A comparative analysis of three embedding models (Nomic Embed, Gemini, Jina) evaluating speed, quality, and stability in vector search for Mexican official documents.
The Battle of the Converters: Our Experience Extracting Text from the DOF
A comparative analysis of different tools for converting PDFs to markdown and why we chose Marker for our DOF-RAG project.
Integrating Gemini Models in DOF RAG: Challenges and Solutions with Google's AI Libraries
An analysis of the challenges encountered during the integration of Google's AI models in the DOF RAG project, managing evolving libraries, and solving API-related issues.
Giving Context to Embeddings: Structured Headers
How we solved the problem of lack of context in text chunks to improve the accuracy of our RAG system.
When AI Models Describe Images: The Brilliant, The Absurd, and The Comical
A comparative analysis of different AI models in the task of describing images for the DOF-RAG project.
Introduction to the DOF-RAG Project
An initiative to improve the accessibility and understanding of information from the Official Journal of the Federation.
Explore by topic
RSS
Get new project notes in your preferred feed reader.