Who Describes DOF Images Best?
We compare 6 vision models (Gemini, GPT, Qwen, Claude) on the task of generating image descriptions for RAG indexing of Mexico's Official Journal of the Federation.
DOF-RAG Team Read more
D O F - R A G
Tracking and documenting progress in the development of retrieval and augmented generation systems for the Official Journal of the Federation.
Tags
#DOF-RAG
#Gemini
#embeddings
#Markdown
#IA
#PDF
#LibreOffice
#Pandoc
#WORD
#descripción-imágenes
#Ollama
#proyecto
#introducción
#NLP
#chunks
#RAG
#Experiencia de Desarrollo
#Dificultades
#Google
#document conversion
#text extraction
#Nomic
#Jina
#DuckDB
#vector-search
#escalabilidad
#proyecciones
#almacenamiento
#pipeline
#web-scraping
#conversión de documentos
#catdoc
#visión
#modelos
#comparación
#OpenRouter
Recent Posts
657 thousand documents later: how we converted ALL of the DOF to Markdown
From 657,867 .doc files from Mexico's Official Journal of the Federation to clean Markdown: tools, results, and what's left to do.
DOF-RAG Team Read more
Four steps to process the DOF: conversion, cleanup, analysis and structure
From the downloaded WORD file to structured Markdown ready for embeddings: a walkthrough of our complete processing pipeline that includes LibreOffice conversion, custom LUA filters, Gemini image analysis, and a robust directory architecture.
DOF-RAG Team Read more
From PDF to WORD: when computational cost dictates a change of strategy
How the reality of massive document processing led us to rethink our strategy: from downloading complete PDFs to obtaining segmented WORD files, dramatically reducing computational cost without sacrificing quality.
DOF-RAG Team Read more
Storage Projections for DOF-RAG
A detailed analysis of storage projections for the DOF-RAG project, evaluating different embedding dimensions and their scalability implications over a 25-year horizon.
DOF-RAG Team Read more
The Battle of Embeddings: When Three AI Models Compete to Understand Governmental Spanish
A comparative analysis of three embedding models (Nomic Embed, Gemini, Jina) evaluating speed, quality, and stability in vector search for Mexican official documents.
DOF-RAG Team Read more
The Battle of the Converters: Our Experience Extracting Text from the DOF
A comparative analysis of different tools for converting PDFs to markdown and why we chose Marker for our DOF-RAG project.
DOF-RAG Team Read more
Integrating Gemini Models in DOF RAG: Challenges and Solutions with Google's AI Libraries
An analysis of the challenges encountered during the integration of Google's AI models in the DOF RAG project, managing evolving libraries, and solving API-related issues.
DOF-RAG Team Read more
Giving Context to Embeddings: Structured Headers
How we solved the problem of lack of context in text chunks to improve the accuracy of our RAG system.
DOF-RAG Team Read more
When AI Models Describe Images: The Brilliant, The Absurd, and The Comical
A comparative analysis of different AI models in the task of describing images for the DOF-RAG project.
DOF-RAG Team Read more
Introduction to the DOF-RAG Project
An initiative to improve the accessibility and understanding of information from the Official Journal of the Federation.
DOF-RAG Team Read more