feat: Add database migration service with Alembic for PostgreSQL, integrating job management and counting functionalities. Update Docker Compose to include db-migrations service, enhancing the microservices architecture. Revise documentation for clarity on database schema and migration processes.

This commit is contained in:
erik
2026-07-06 00:50:50 -05:00
parent d95f8f6ad8
commit 081da999d4
43 changed files with 2911 additions and 2271 deletions
+133 -39
View File
@@ -1,45 +1,139 @@
# IPSExtractor Extraction Service
# extraction
Microservicio independiente de extraccion. Esta carpeta puede vivir como repo
propio: contiene su API REST, dominio, infraestructura, dependencias y
Dockerfile sin importar codigo de otros servicios.
Servicio Python (FastAPI) de extracción de datos estructurados desde documentos médicos PDF.
Usa LLMs (compatible con OpenAI API) para descubrir campos, estimar costos y extraer datos por lotes.
## Estructura
También actúa como capa de persistencia para el flujo de conteo: guarda los resultados del `count-api` en PostgreSQL y orquesta la extracción de documentos validados.
```text
api/ Endpoints FastAPI
domain/ Logica de extraccion, prompts, estimacion y normalizacion
core/ Utilidades propias de PDF, texto y CSV
infrastructure/ Persistencia SQLAlchemy del servicio
requirements.txt
Dockerfile
```
## Desarrollo local
```bash
pip install -r requirements.txt
uvicorn api.main:app --host 0.0.0.0 --port 8001
```
## Poppler
El contenedor instala `poppler-utils` para que `pdf2image` encuentre
`pdfinfo` y `pdftoppm` desde el PATH del sistema, sin configurar variables de
entorno.
En desarrollo local Windows, el servicio usa automaticamente la carpeta incluida
en `dependencies/poppler-24.08.0/Library/bin`.
---
## Endpoints principales
- `GET /health`
- `GET /config`
- `GET /models`
- `POST /schemas`
- `POST /schemas/upload`
- `POST /estimate`
- `POST /batches`
- `POST /tasks/extraction`
- `POST /tasks/{task_id}/ingest`
- `GET /tasks`
Puerto: **8001**
### Salud y configuración
| Método | Ruta | Descripción |
|--------|------|-------------|
| `GET` | `/health` | Liveness |
| `GET` | `/config` | Configuración LLM activa |
| `GET` | `/models` | Modelos disponibles en el backend LLM |
### Descubrimiento de esquema
| Método | Ruta | Descripción |
|--------|------|-------------|
| `POST` | `/schemas` | Descubrir campos desde ruta en disco |
| `POST` | `/schemas/upload` | Descubrir campos desde PDF subido |
### Estimación de costo
| Método | Ruta | Descripción |
|--------|------|-------------|
| `POST` | `/estimate/upload` | Estimar tokens/costo desde PDF subido |
### Tareas de extracción
| Método | Ruta | Descripción |
|--------|------|-------------|
| `POST` | `/tasks/extraction` | Crear tarea de extracción |
| `POST` | `/tasks/{id}/ingest` | Subir un PDF a la tarea |
| `POST` | `/tasks/{id}/ingest-many` | Subir varios PDFs en bulk |
| `GET` | `/tasks` | Listar tareas |
| `GET` | `/tasks/{id}` | Detalle de tarea |
| `GET` | `/tasks/{id}/rows` | Filas extraídas de la tarea |
| `POST` | `/tasks/extraction/from-count` | Crear extracción desde conteo previo |
### Jobs (contenedor de tareas)
| Método | Ruta | Descripción |
|--------|------|-------------|
| `GET` | `/jobs` | Listar jobs |
| `GET` | `/jobs/{id}` | Detalle de job |
| `GET` | `/jobs/{id}/tasks` | Tareas del job |
### Conteo (persistencia desde count-api)
| Método | Ruta | Descripción |
|--------|------|-------------|
| `POST` | `/count-jobs` | Guardar resultados de conteo en BD |
| `GET` | `/count-jobs` | Listar jobs de conteo |
| `GET` | `/count-jobs/{task_id}` | Detalle de job de conteo |
| `GET` | `/count-jobs/{task_id}/files` | Archivos del conteo |
| `PATCH` | `/count-jobs/{task_id}/files/{file_id}` | Marcar archivo como válido/inválido |
---
## Flujo de datos
```
PDF local (desktop) → POST /tasks/{id}/ingest-many → procesamiento LLM
↓
filas en extraction_rows (PG)
count-api results → POST /count-jobs → count_job_files (PG)
→ POST /tasks/extraction/from-count → lee file_path desde /data
```
---
## Variables de entorno
| Variable | Descripción |
|----------|-------------|
| `DATABASE_URL` | PostgreSQL (`postgresql+psycopg://...`) |
| `REDIS_URL` | Redis para cola ARQ (`redis://...`) |
| `EXTRACTOR_BASE_URL` | URL base del LLM (compatible OpenAI) |
| `EXTRACTOR_API_KEY` | API key del LLM extractor |
| `EXTRACTOR_MODEL` | Modelo de extracción (ej. `gpt-5.4-mini`) |
| `NORMALIZER_BASE_URL` | URL base del LLM normalizador |
| `NORMALIZER_API_KEY` | API key del LLM normalizador |
| `NORMALIZER_MODEL` | Modelo normalizador (ej. `gpt-5.4-mini`) |
| `EXTRACTOR_TIMEOUT_S` | Timeout por documento (default `120`) |
| `PDF_DPI` | DPI para renderizado PDF (default `150`) |
| `PDF_THREAD_COUNT` | Hilos de renderizado PDF (default `4`) |
| `CONCURRENCY_LIMIT` | Máximo de extracciones paralelas (default `50`) |
| `ALLOWED_DOCUMENT_TYPES` | Tipos de documento permitidos |
| `ASYNC_QUEUE` | `true` para encolar en ARQ/Redis (default: sync) |
---
## Modelo de datos (PostgreSQL)
```
jobs — contenedor de tareas del usuario
tasks — una tarea por módulo (counting / extraction / cleaning)
count_job_files — resultados de conteo por PDF
extraction_batches — lotes de extracción
extraction_rows — filas extraídas por documento
```
---
## Estructura del código
```
api/
main.py — FastAPI app (~1400 líneas), todos los endpoints
desktop_ingest.py — lógica de ingesta desde desktop Electron
domain/ — lógica de negocio: LLM, prompts, extractor, batch
core/ — utilidades PDF, CSV, texto
infrastructure/ — SQLAlchemy, migraciones, uploads temporales
worker/ — workers ARQ (procesamiento async)
main.py — punto de entrada del worker
tasks.py — tareas ARQ
settings.py — configuración del worker
Dockerfile
requirements.txt
```
---
## Arrancar localmente
```bash
# Desde la raíz del repositorio:
docker compose up --build extraction worker
```
API disponible en `http://localhost:8001`.
+419
View File
@@ -28,8 +28,10 @@ from domain.estimator import Estimator
from domain.service import InvoiceProcessingService
from domain.strategies import OpenAICompatibleExtractor
from infrastructure.database import (
CountJobFile,
ExtractionBatch,
ExtractionRow,
Job,
Task,
init_db,
session_scope,
@@ -938,3 +940,420 @@ def get_rows(batch_id: str, limit: int = 500, offset: int = 0) -> dict[str, Any]
"limit": limit,
"offset": offset,
}
# ═══════════════════════════════════════════════════════════════════════════════
# Jobs (contenedores multimodulo)
# ═══════════════════════════════════════════════════════════════════════════════
class JobRead(BaseModel):
id: str
status: str
title: str | None
metadata: dict[str, Any]
created_at: str
updated_at: str
def serialize_job(job: Job) -> dict[str, Any]:
return {
"id": job.id,
"status": job.status,
"title": job.title,
"metadata": job.job_metadata or {},
"created_at": job.created_at.isoformat(),
"updated_at": job.updated_at.isoformat(),
}
@app.get("/jobs", response_model=list[JobRead])
def list_jobs() -> list[dict[str, Any]]:
with session_scope() as session:
jobs = session.scalars(select(Job).order_by(Job.created_at.desc())).all()
return [serialize_job(j) for j in jobs]
@app.get("/jobs/{job_id}", response_model=JobRead)
def get_job(job_id: str) -> dict[str, Any]:
with session_scope() as session:
job = session.get(Job, job_id)
if job is None:
raise HTTPException(status_code=404, detail="Job no encontrado")
return serialize_job(job)
@app.get("/jobs/{job_id}/tasks", response_model=list[TaskRead])
def get_job_tasks(job_id: str) -> list[dict[str, Any]]:
with session_scope() as session:
job = session.get(Job, job_id)
if job is None:
raise HTTPException(status_code=404, detail="Job no encontrado")
tasks = session.scalars(
select(Task).where(Task.job_id == job_id).order_by(Task.created_at)
).all()
return [serialize_task(t) for t in tasks]
# ═══════════════════════════════════════════════════════════════════════════════
# Count Jobs — persistir y gestionar resultados del servicio de conteo
# ═══════════════════════════════════════════════════════════════════════════════
class CountJobFileCreate(BaseModel):
radicado: str | None = None
filename: str
file_path: str | None = None
doc_type: str | None = None
service_count: int | None = None
services: list[dict[str, Any]] = Field(default_factory=list)
status: str = "ok"
error: str | None = None
class CreateCountJobRequest(BaseModel):
title: str | None = None
results: list[CountJobFileCreate]
class CountJobFileRead(BaseModel):
id: str
job_module_id: str
radicado: str | None
filename: str
file_path: str | None
doc_type: str | None
service_count: int | None
services: list[dict[str, Any]]
status: str
is_valid: bool | None
error: str | None
created_at: str
class UpdateCountJobFileRequest(BaseModel):
is_valid: bool | None
class ExtractionFromCountRequest(BaseModel):
counting_task_id: str
min_service_count: int = 0
fields: list[str] = Field(default_factory=list)
alias_map: dict[str, str] = Field(default_factory=dict)
doc_types: list[DocumentType] | None = None
structure_mode: str = "general"
instruction: str | None = None
tiff_page_range: tuple[int, int] | None = None
retry_on_empty: bool = False
model: str | None = None
def serialize_count_file(f: CountJobFile) -> dict[str, Any]:
return {
"id": f.id,
"job_module_id": f.job_module_id,
"radicado": f.radicado,
"filename": f.filename,
"file_path": f.file_path,
"doc_type": f.doc_type,
"service_count": f.service_count,
"services": f.services or [],
"status": f.status,
"is_valid": f.is_valid,
"error": f.error,
"created_at": f.created_at.isoformat(),
}
@app.post("/count-jobs", response_model=TaskRead)
def create_count_job(request: CreateCountJobRequest) -> dict[str, Any]:
"""Persiste los resultados del servicio de conteo de facturas en la BD compartida."""
total = len(request.results)
with session_scope() as session:
job = Job(
status="completed",
title=request.title or f"Conteo ({total} archivos)",
updated_at=utcnow(),
)
session.add(job)
session.flush()
task = Task(
job_id=job.id,
type="counting",
status="completed",
title=job.title,
request={},
result={
"total": total,
"processed": total,
"ok": sum(1 for r in request.results if r.status == "ok"),
"failed": sum(1 for r in request.results if r.status != "ok"),
},
started_at=utcnow(),
finished_at=utcnow(),
updated_at=utcnow(),
)
session.add(task)
session.flush()
for r in request.results:
session.add(
CountJobFile(
job_module_id=task.id,
radicado=r.radicado,
filename=r.filename,
file_path=r.file_path,
doc_type=r.doc_type,
service_count=r.service_count,
services=r.services,
status=r.status,
error=r.error,
)
)
session.flush()
return serialize_task(task)
@app.get("/count-jobs", response_model=list[TaskRead])
def list_count_jobs() -> list[dict[str, Any]]:
with session_scope() as session:
tasks = session.scalars(
select(Task)
.where(Task.type == "counting")
.order_by(Task.created_at.desc())
).all()
return [serialize_task(t) for t in tasks]
@app.get("/count-jobs/{task_id}", response_model=TaskRead)
def get_count_job(task_id: str) -> dict[str, Any]:
with session_scope() as session:
task = session.get(Task, task_id)
if task is None or task.type != "counting":
raise HTTPException(status_code=404, detail="Tarea de conteo no encontrada")
return serialize_task(task)
@app.get("/count-jobs/{task_id}/files", response_model=list[CountJobFileRead])
def list_count_job_files(task_id: str) -> list[dict[str, Any]]:
with session_scope() as session:
task = session.get(Task, task_id)
if task is None or task.type != "counting":
raise HTTPException(status_code=404, detail="Tarea de conteo no encontrada")
files = session.scalars(
select(CountJobFile)
.where(CountJobFile.job_module_id == task_id)
.order_by(CountJobFile.created_at)
).all()
return [serialize_count_file(f) for f in files]
@app.patch("/count-jobs/{task_id}/files/{file_id}", response_model=CountJobFileRead)
def update_count_job_file(
task_id: str, file_id: str, request: UpdateCountJobFileRequest
) -> dict[str, Any]:
with session_scope() as session:
f = session.get(CountJobFile, file_id)
if f is None or f.job_module_id != task_id:
raise HTTPException(status_code=404, detail="Archivo no encontrado")
f.is_valid = request.is_valid
session.flush()
return serialize_count_file(f)
# ═══════════════════════════════════════════════════════════════════════════════
# Extraccion desde resultados de conteo (sin re-subir archivos)
# ═══════════════════════════════════════════════════════════════════════════════
def run_from_count_task(task_id: str) -> None:
"""Procesa archivos validos de un job de conteo en background (lee desde el volumen /data)."""
import logging as _logging
_log = _logging.getLogger("extraction.from_count")
from api.desktop_ingest import _service_for_model, mark_task_completed_if_done
with session_scope() as session:
task = session.get(Task, task_id)
if task is None:
return
files_data: list[dict[str, Any]] = list((task.result or {}).get("files", []))
req = dict(task.request or {})
batch_id = str((task.result or {}).get("extraction_batch_id") or "")
model_name: str | None = req.get("model")
instruction: str | None = req.get("instruction")
tiff_page_range = req.get("tiff_page_range")
for file_data in files_data:
file_path: str | None = file_data.get("file_path")
filename = str(file_data.get("filename") or "documento.pdf")
radicado = str(file_data.get("radicado") or file_data.get("filename") or "")
doc_type: str | None = file_data.get("doc_type")
if not file_path or not __import__("os").path.exists(file_path):
_log.warning("Archivo no disponible: %s", file_path or filename)
with session_scope() as session:
task_obj = session.get(Task, task_id)
if task_obj is None:
return
result = dict(task_obj.result or {})
errors = list(result.get("errors") or [])
errors.append(f"Archivo no encontrado: {filename}")
result["errors"] = errors
result["processed"] = int(result.get("processed") or 0) + 1
task_obj.result = result
task_obj.updated_at = utcnow()
mark_task_completed_if_done(session, task_obj, batch_id)
continue
try:
svc = _service_for_model(processing_service, model_name)
with session_scope() as session:
task_obj = session.get(Task, task_id, with_for_update=True)
if task_obj is None:
return
from api.desktop_ingest import process_desktop_ingest
process_desktop_ingest(
svc, session,
task=task_obj,
batch_id=batch_id,
temp_path=file_path,
radicado=radicado,
filename=filename,
doc_type=doc_type,
instruction=instruction,
tiff_page_range=tiff_page_range,
model=model_name,
)
mark_task_completed_if_done(session, task_obj, batch_id)
except Exception as exc:
_log.exception("Error procesando %s: %s", filename, exc)
with session_scope() as session:
task_obj = session.get(Task, task_id)
if task_obj is None:
return
result = dict(task_obj.result or {})
errors = list(result.get("errors") or [])
errors.append(f"{filename}: {exc}")
result["errors"] = errors
result["processed"] = int(result.get("processed") or 0) + 1
task_obj.result = result
task_obj.updated_at = utcnow()
mark_task_completed_if_done(session, task_obj, batch_id)
@app.post("/tasks/extraction/from-count", response_model=TaskRead)
def create_extraction_from_count(
request: ExtractionFromCountRequest, background_tasks: BackgroundTasks
) -> dict[str, Any]:
"""Crea una tarea de extraccion usando archivos validos de un job de conteo previo.
No requiere subir archivos: usa las rutas almacenadas en count_job_files.
"""
if not request.fields:
raise HTTPException(
status_code=400,
detail="Envia fields. Usa POST /extraction/schemas/upload para descubrir campos.",
)
with session_scope() as session:
counting_task = session.get(Task, request.counting_task_id)
if counting_task is None or counting_task.type != "counting":
raise HTTPException(status_code=404, detail="Tarea de conteo no encontrada")
all_files = session.scalars(
select(CountJobFile).where(
CountJobFile.job_module_id == request.counting_task_id
)
).all()
valid_files = [
f for f in all_files
if f.status == "ok"
and f.is_valid is not False
and (f.service_count is None or f.service_count >= request.min_service_count)
]
if not valid_files:
raise HTTPException(
status_code=400,
detail="No hay archivos validos para extraer con los criterios indicados.",
)
files_data = [
{
"id": f.id,
"radicado": f.radicado or f.filename,
"filename": f.filename,
"file_path": f.file_path,
"doc_type": f.doc_type,
"service_count": f.service_count,
}
for f in valid_files
]
total = len(files_data)
job = Job(
status="running",
title=f"Extraccion desde conteo ({total} archivos)",
updated_at=utcnow(),
)
session.add(job)
session.flush()
batch = ExtractionBatch(
status="running",
root_path="count://",
request={
"source": "from-count",
"counting_task_id": request.counting_task_id,
"fields": request.fields,
"alias_map": request.alias_map,
"min_service_count": request.min_service_count,
"structure_mode": request.structure_mode,
},
updated_at=utcnow(),
)
session.add(batch)
session.flush()
task = Task(
job_id=job.id,
type="extraction",
status="running",
title=job.title,
request={
"source": "from-count",
"counting_task_id": request.counting_task_id,
"fields": request.fields,
"alias_map": request.alias_map,
"doc_types": [dt.model_dump() for dt in request.doc_types or []],
"structure_mode": request.structure_mode,
"instruction": request.instruction,
"tiff_page_range": list(request.tiff_page_range) if request.tiff_page_range else None,
"min_service_count": request.min_service_count,
"model": request.model,
},
result={
"source": "from-count",
"extraction_batch_id": batch.id,
"processed": 0,
"total": total,
"row_count": 0,
"errors": [],
"files": files_data,
"partials": {},
},
started_at=utcnow(),
updated_at=utcnow(),
)
session.add(task)
session.flush()
batch.job_module_id = task.id
session.flush()
serialized = serialize_task(task)
task_id = task.id
background_tasks.add_task(run_from_count_task, task_id)
return serialized
+48 -9
View File
@@ -9,6 +9,7 @@ from typing import Any, Iterator
from sqlalchemy import (
JSON,
Boolean,
DateTime,
ForeignKey,
Integer,
@@ -42,6 +43,20 @@ def new_id() -> str:
return str(uuid.uuid4())
class Job(Base):
"""Contenedor de trabajo del usuario; puede abarcar múltiples módulos/servicios."""
__tablename__ = "jobs"
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_id)
status: Mapped[str] = mapped_column(String(32), default="pending", index=True)
title: Mapped[str | None] = mapped_column(Text, nullable=True)
# "metadata" es reservado por DeclarativeBase; el atributo Python usa job_metadata
# pero la columna en la BD sigue llamándose "metadata".
job_metadata: Mapped[dict[str, Any]] = mapped_column("metadata", JSON, default=dict)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
class ExtractionBatch(Base):
__tablename__ = "extraction_batches"
@@ -50,14 +65,21 @@ class ExtractionBatch(Base):
root_path: Mapped[str | None] = mapped_column(Text, nullable=True)
request: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict)
error: Mapped[str | None] = mapped_column(Text, nullable=True)
job_module_id: Mapped[str | None] = mapped_column(
String(36), ForeignKey("tasks.id"), nullable=True, index=True
)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
class Task(Base):
"""Módulo/resultado dentro de un Job; un job puede tener N tasks (una por servicio)."""
__tablename__ = "tasks"
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_id)
job_id: Mapped[str | None] = mapped_column(
String(36), ForeignKey("jobs.id"), nullable=True, index=True
)
type: Mapped[str] = mapped_column(String(64), index=True)
status: Mapped[str] = mapped_column(String(32), default="pending", index=True)
title: Mapped[str | None] = mapped_column(Text, nullable=True)
@@ -71,6 +93,26 @@ class Task(Base):
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
class CountJobFile(Base):
"""Resultado del servicio de conteo para un PDF individual dentro de una tarea de conteo."""
__tablename__ = "count_job_files"
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_id)
job_module_id: Mapped[str] = mapped_column(
String(36), ForeignKey("tasks.id"), index=True
)
radicado: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
filename: Mapped[str] = mapped_column(String(512))
file_path: Mapped[str | None] = mapped_column(Text, nullable=True)
doc_type: Mapped[str | None] = mapped_column(String(128), nullable=True)
service_count: Mapped[int | None] = mapped_column(Integer, nullable=True)
services: Mapped[list[Any]] = mapped_column(JSON, default=list)
status: Mapped[str] = mapped_column(String(32), default="ok", index=True)
is_valid: Mapped[bool | None] = mapped_column(Boolean, nullable=True)
error: Mapped[str | None] = mapped_column(Text, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
class ExtractionRow(Base):
__tablename__ = "extraction_rows"
@@ -109,16 +151,13 @@ class CleanedRow(Base):
def init_db() -> None:
if DATABASE_URL.startswith("postgresql"):
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as connection:
connection.execute(text("SELECT pg_advisory_lock(74512001)"))
try:
Base.metadata.create_all(bind=connection)
finally:
connection.execute(text("SELECT pg_advisory_unlock(74512001)"))
return
"""Verifica conectividad con la BD.
Base.metadata.create_all(bind=engine)
Las migraciones son responsabilidad exclusiva de services/db (init container).
Este servicio asume que el esquema ya está actualizado al arrancar.
"""
with engine.connect() as conn:
conn.execute(text("SELECT 1"))
@contextmanager