An Agent That Knows When It Is Missing Evidence: First Runs on the DOF
How we built a bounded tool loop, turned the parts of a question into verifiable requirements, and evaluated its answers on the DOF.
Joaquín Bravo ContrerasRead more
#evaluation
4 postswith this tag
From finding documents to finding evidence: five tools for querying the DOF
How we separated document search from passage retrieval and built five deterministic, traceable, and evaluable tools on top of the DOF.
Joaquín Bravo ContrerasRead more
Eval v4: finding the document is not enough; the evidence must be found too
We built a hand-crafted 42-question evaluation to measure complete lists, effective dates, legal references, multi-document queries, monitoring, and false premises. We ran it against full BM25 and a 38.2% vector index to set a baseline before finishing the embeddings.
Joaquín Bravo ContrerasRead more
The First Real-Scale Evaluation: BM25 Against 657,867 Documents, from 0.170 to 0.366 After Fixing the Set
While the embedding run advances over 6.7 million chunks, we built the full-corpus FTS5 index (2.7 GiB) and ran the first real-scale BM25 evaluation. The v2 query set produced an MRR of 0.170; after fixing fake titles and ambiguous queries, v3 reached 0.366 and the partial hybrid smoke test 0.402. Along the way: a COUNT(*) that lies, 32 documents almost invisible to the index, and a token-pruning step that turned 21 hours of queries into 34 minutes without changing the metrics.
Joaquín Bravo ContrerasRead more