feat: Add database migration service with Alembic for PostgreSQL, integrating job management and counting functionalities. Update Docker Compose to include db-migrations service, enhancing the microservices architecture. Revise documentation for clarity on database schema and migration processes.
This commit is contained in:
+133
-39
@@ -1,45 +1,139 @@
|
||||
# IPSExtractor Extraction Service
|
||||
# extraction
|
||||
|
||||
Microservicio independiente de extraccion. Esta carpeta puede vivir como repo
|
||||
propio: contiene su API REST, dominio, infraestructura, dependencias y
|
||||
Dockerfile sin importar codigo de otros servicios.
|
||||
Servicio Python (FastAPI) de extracción de datos estructurados desde documentos médicos PDF.
|
||||
Usa LLMs (compatible con OpenAI API) para descubrir campos, estimar costos y extraer datos por lotes.
|
||||
|
||||
## Estructura
|
||||
También actúa como capa de persistencia para el flujo de conteo: guarda los resultados del `count-api` en PostgreSQL y orquesta la extracción de documentos validados.
|
||||
|
||||
```text
|
||||
api/ Endpoints FastAPI
|
||||
domain/ Logica de extraccion, prompts, estimacion y normalizacion
|
||||
core/ Utilidades propias de PDF, texto y CSV
|
||||
infrastructure/ Persistencia SQLAlchemy del servicio
|
||||
requirements.txt
|
||||
Dockerfile
|
||||
```
|
||||
|
||||
## Desarrollo local
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
uvicorn api.main:app --host 0.0.0.0 --port 8001
|
||||
```
|
||||
|
||||
## Poppler
|
||||
|
||||
El contenedor instala `poppler-utils` para que `pdf2image` encuentre
|
||||
`pdfinfo` y `pdftoppm` desde el PATH del sistema, sin configurar variables de
|
||||
entorno.
|
||||
|
||||
En desarrollo local Windows, el servicio usa automaticamente la carpeta incluida
|
||||
en `dependencies/poppler-24.08.0/Library/bin`.
|
||||
---
|
||||
|
||||
## Endpoints principales
|
||||
|
||||
- `GET /health`
|
||||
- `GET /config`
|
||||
- `GET /models`
|
||||
- `POST /schemas`
|
||||
- `POST /schemas/upload`
|
||||
- `POST /estimate`
|
||||
- `POST /batches`
|
||||
- `POST /tasks/extraction`
|
||||
- `POST /tasks/{task_id}/ingest`
|
||||
- `GET /tasks`
|
||||
Puerto: **8001**
|
||||
|
||||
### Salud y configuración
|
||||
|
||||
| Método | Ruta | Descripción |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/health` | Liveness |
|
||||
| `GET` | `/config` | Configuración LLM activa |
|
||||
| `GET` | `/models` | Modelos disponibles en el backend LLM |
|
||||
|
||||
### Descubrimiento de esquema
|
||||
|
||||
| Método | Ruta | Descripción |
|
||||
|--------|------|-------------|
|
||||
| `POST` | `/schemas` | Descubrir campos desde ruta en disco |
|
||||
| `POST` | `/schemas/upload` | Descubrir campos desde PDF subido |
|
||||
|
||||
### Estimación de costo
|
||||
|
||||
| Método | Ruta | Descripción |
|
||||
|--------|------|-------------|
|
||||
| `POST` | `/estimate/upload` | Estimar tokens/costo desde PDF subido |
|
||||
|
||||
### Tareas de extracción
|
||||
|
||||
| Método | Ruta | Descripción |
|
||||
|--------|------|-------------|
|
||||
| `POST` | `/tasks/extraction` | Crear tarea de extracción |
|
||||
| `POST` | `/tasks/{id}/ingest` | Subir un PDF a la tarea |
|
||||
| `POST` | `/tasks/{id}/ingest-many` | Subir varios PDFs en bulk |
|
||||
| `GET` | `/tasks` | Listar tareas |
|
||||
| `GET` | `/tasks/{id}` | Detalle de tarea |
|
||||
| `GET` | `/tasks/{id}/rows` | Filas extraídas de la tarea |
|
||||
| `POST` | `/tasks/extraction/from-count` | Crear extracción desde conteo previo |
|
||||
|
||||
### Jobs (contenedor de tareas)
|
||||
|
||||
| Método | Ruta | Descripción |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/jobs` | Listar jobs |
|
||||
| `GET` | `/jobs/{id}` | Detalle de job |
|
||||
| `GET` | `/jobs/{id}/tasks` | Tareas del job |
|
||||
|
||||
### Conteo (persistencia desde count-api)
|
||||
|
||||
| Método | Ruta | Descripción |
|
||||
|--------|------|-------------|
|
||||
| `POST` | `/count-jobs` | Guardar resultados de conteo en BD |
|
||||
| `GET` | `/count-jobs` | Listar jobs de conteo |
|
||||
| `GET` | `/count-jobs/{task_id}` | Detalle de job de conteo |
|
||||
| `GET` | `/count-jobs/{task_id}/files` | Archivos del conteo |
|
||||
| `PATCH` | `/count-jobs/{task_id}/files/{file_id}` | Marcar archivo como válido/inválido |
|
||||
|
||||
---
|
||||
|
||||
## Flujo de datos
|
||||
|
||||
```
|
||||
PDF local (desktop) → POST /tasks/{id}/ingest-many → procesamiento LLM
|
||||
↓
|
||||
filas en extraction_rows (PG)
|
||||
|
||||
count-api results → POST /count-jobs → count_job_files (PG)
|
||||
→ POST /tasks/extraction/from-count → lee file_path desde /data
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Variables de entorno
|
||||
|
||||
| Variable | Descripción |
|
||||
|----------|-------------|
|
||||
| `DATABASE_URL` | PostgreSQL (`postgresql+psycopg://...`) |
|
||||
| `REDIS_URL` | Redis para cola ARQ (`redis://...`) |
|
||||
| `EXTRACTOR_BASE_URL` | URL base del LLM (compatible OpenAI) |
|
||||
| `EXTRACTOR_API_KEY` | API key del LLM extractor |
|
||||
| `EXTRACTOR_MODEL` | Modelo de extracción (ej. `gpt-5.4-mini`) |
|
||||
| `NORMALIZER_BASE_URL` | URL base del LLM normalizador |
|
||||
| `NORMALIZER_API_KEY` | API key del LLM normalizador |
|
||||
| `NORMALIZER_MODEL` | Modelo normalizador (ej. `gpt-5.4-mini`) |
|
||||
| `EXTRACTOR_TIMEOUT_S` | Timeout por documento (default `120`) |
|
||||
| `PDF_DPI` | DPI para renderizado PDF (default `150`) |
|
||||
| `PDF_THREAD_COUNT` | Hilos de renderizado PDF (default `4`) |
|
||||
| `CONCURRENCY_LIMIT` | Máximo de extracciones paralelas (default `50`) |
|
||||
| `ALLOWED_DOCUMENT_TYPES` | Tipos de documento permitidos |
|
||||
| `ASYNC_QUEUE` | `true` para encolar en ARQ/Redis (default: sync) |
|
||||
|
||||
---
|
||||
|
||||
## Modelo de datos (PostgreSQL)
|
||||
|
||||
```
|
||||
jobs — contenedor de tareas del usuario
|
||||
tasks — una tarea por módulo (counting / extraction / cleaning)
|
||||
count_job_files — resultados de conteo por PDF
|
||||
extraction_batches — lotes de extracción
|
||||
extraction_rows — filas extraídas por documento
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Estructura del código
|
||||
|
||||
```
|
||||
api/
|
||||
main.py — FastAPI app (~1400 líneas), todos los endpoints
|
||||
desktop_ingest.py — lógica de ingesta desde desktop Electron
|
||||
domain/ — lógica de negocio: LLM, prompts, extractor, batch
|
||||
core/ — utilidades PDF, CSV, texto
|
||||
infrastructure/ — SQLAlchemy, migraciones, uploads temporales
|
||||
worker/ — workers ARQ (procesamiento async)
|
||||
main.py — punto de entrada del worker
|
||||
tasks.py — tareas ARQ
|
||||
settings.py — configuración del worker
|
||||
Dockerfile
|
||||
requirements.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Arrancar localmente
|
||||
|
||||
```bash
|
||||
# Desde la raíz del repositorio:
|
||||
docker compose up --build extraction worker
|
||||
```
|
||||
|
||||
API disponible en `http://localhost:8001`.
|
||||
|
||||
@@ -28,8 +28,10 @@ from domain.estimator import Estimator
|
||||
from domain.service import InvoiceProcessingService
|
||||
from domain.strategies import OpenAICompatibleExtractor
|
||||
from infrastructure.database import (
|
||||
CountJobFile,
|
||||
ExtractionBatch,
|
||||
ExtractionRow,
|
||||
Job,
|
||||
Task,
|
||||
init_db,
|
||||
session_scope,
|
||||
@@ -938,3 +940,420 @@ def get_rows(batch_id: str, limit: int = 500, offset: int = 0) -> dict[str, Any]
|
||||
"limit": limit,
|
||||
"offset": offset,
|
||||
}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# Jobs (contenedores multimodulo)
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
class JobRead(BaseModel):
|
||||
id: str
|
||||
status: str
|
||||
title: str | None
|
||||
metadata: dict[str, Any]
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
def serialize_job(job: Job) -> dict[str, Any]:
|
||||
return {
|
||||
"id": job.id,
|
||||
"status": job.status,
|
||||
"title": job.title,
|
||||
"metadata": job.job_metadata or {},
|
||||
"created_at": job.created_at.isoformat(),
|
||||
"updated_at": job.updated_at.isoformat(),
|
||||
}
|
||||
|
||||
|
||||
@app.get("/jobs", response_model=list[JobRead])
|
||||
def list_jobs() -> list[dict[str, Any]]:
|
||||
with session_scope() as session:
|
||||
jobs = session.scalars(select(Job).order_by(Job.created_at.desc())).all()
|
||||
return [serialize_job(j) for j in jobs]
|
||||
|
||||
|
||||
@app.get("/jobs/{job_id}", response_model=JobRead)
|
||||
def get_job(job_id: str) -> dict[str, Any]:
|
||||
with session_scope() as session:
|
||||
job = session.get(Job, job_id)
|
||||
if job is None:
|
||||
raise HTTPException(status_code=404, detail="Job no encontrado")
|
||||
return serialize_job(job)
|
||||
|
||||
|
||||
@app.get("/jobs/{job_id}/tasks", response_model=list[TaskRead])
|
||||
def get_job_tasks(job_id: str) -> list[dict[str, Any]]:
|
||||
with session_scope() as session:
|
||||
job = session.get(Job, job_id)
|
||||
if job is None:
|
||||
raise HTTPException(status_code=404, detail="Job no encontrado")
|
||||
tasks = session.scalars(
|
||||
select(Task).where(Task.job_id == job_id).order_by(Task.created_at)
|
||||
).all()
|
||||
return [serialize_task(t) for t in tasks]
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# Count Jobs — persistir y gestionar resultados del servicio de conteo
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
class CountJobFileCreate(BaseModel):
|
||||
radicado: str | None = None
|
||||
filename: str
|
||||
file_path: str | None = None
|
||||
doc_type: str | None = None
|
||||
service_count: int | None = None
|
||||
services: list[dict[str, Any]] = Field(default_factory=list)
|
||||
status: str = "ok"
|
||||
error: str | None = None
|
||||
|
||||
|
||||
class CreateCountJobRequest(BaseModel):
|
||||
title: str | None = None
|
||||
results: list[CountJobFileCreate]
|
||||
|
||||
|
||||
class CountJobFileRead(BaseModel):
|
||||
id: str
|
||||
job_module_id: str
|
||||
radicado: str | None
|
||||
filename: str
|
||||
file_path: str | None
|
||||
doc_type: str | None
|
||||
service_count: int | None
|
||||
services: list[dict[str, Any]]
|
||||
status: str
|
||||
is_valid: bool | None
|
||||
error: str | None
|
||||
created_at: str
|
||||
|
||||
|
||||
class UpdateCountJobFileRequest(BaseModel):
|
||||
is_valid: bool | None
|
||||
|
||||
|
||||
class ExtractionFromCountRequest(BaseModel):
|
||||
counting_task_id: str
|
||||
min_service_count: int = 0
|
||||
fields: list[str] = Field(default_factory=list)
|
||||
alias_map: dict[str, str] = Field(default_factory=dict)
|
||||
doc_types: list[DocumentType] | None = None
|
||||
structure_mode: str = "general"
|
||||
instruction: str | None = None
|
||||
tiff_page_range: tuple[int, int] | None = None
|
||||
retry_on_empty: bool = False
|
||||
model: str | None = None
|
||||
|
||||
|
||||
def serialize_count_file(f: CountJobFile) -> dict[str, Any]:
|
||||
return {
|
||||
"id": f.id,
|
||||
"job_module_id": f.job_module_id,
|
||||
"radicado": f.radicado,
|
||||
"filename": f.filename,
|
||||
"file_path": f.file_path,
|
||||
"doc_type": f.doc_type,
|
||||
"service_count": f.service_count,
|
||||
"services": f.services or [],
|
||||
"status": f.status,
|
||||
"is_valid": f.is_valid,
|
||||
"error": f.error,
|
||||
"created_at": f.created_at.isoformat(),
|
||||
}
|
||||
|
||||
|
||||
@app.post("/count-jobs", response_model=TaskRead)
|
||||
def create_count_job(request: CreateCountJobRequest) -> dict[str, Any]:
|
||||
"""Persiste los resultados del servicio de conteo de facturas en la BD compartida."""
|
||||
total = len(request.results)
|
||||
with session_scope() as session:
|
||||
job = Job(
|
||||
status="completed",
|
||||
title=request.title or f"Conteo ({total} archivos)",
|
||||
updated_at=utcnow(),
|
||||
)
|
||||
session.add(job)
|
||||
session.flush()
|
||||
|
||||
task = Task(
|
||||
job_id=job.id,
|
||||
type="counting",
|
||||
status="completed",
|
||||
title=job.title,
|
||||
request={},
|
||||
result={
|
||||
"total": total,
|
||||
"processed": total,
|
||||
"ok": sum(1 for r in request.results if r.status == "ok"),
|
||||
"failed": sum(1 for r in request.results if r.status != "ok"),
|
||||
},
|
||||
started_at=utcnow(),
|
||||
finished_at=utcnow(),
|
||||
updated_at=utcnow(),
|
||||
)
|
||||
session.add(task)
|
||||
session.flush()
|
||||
|
||||
for r in request.results:
|
||||
session.add(
|
||||
CountJobFile(
|
||||
job_module_id=task.id,
|
||||
radicado=r.radicado,
|
||||
filename=r.filename,
|
||||
file_path=r.file_path,
|
||||
doc_type=r.doc_type,
|
||||
service_count=r.service_count,
|
||||
services=r.services,
|
||||
status=r.status,
|
||||
error=r.error,
|
||||
)
|
||||
)
|
||||
session.flush()
|
||||
return serialize_task(task)
|
||||
|
||||
|
||||
@app.get("/count-jobs", response_model=list[TaskRead])
|
||||
def list_count_jobs() -> list[dict[str, Any]]:
|
||||
with session_scope() as session:
|
||||
tasks = session.scalars(
|
||||
select(Task)
|
||||
.where(Task.type == "counting")
|
||||
.order_by(Task.created_at.desc())
|
||||
).all()
|
||||
return [serialize_task(t) for t in tasks]
|
||||
|
||||
|
||||
@app.get("/count-jobs/{task_id}", response_model=TaskRead)
|
||||
def get_count_job(task_id: str) -> dict[str, Any]:
|
||||
with session_scope() as session:
|
||||
task = session.get(Task, task_id)
|
||||
if task is None or task.type != "counting":
|
||||
raise HTTPException(status_code=404, detail="Tarea de conteo no encontrada")
|
||||
return serialize_task(task)
|
||||
|
||||
|
||||
@app.get("/count-jobs/{task_id}/files", response_model=list[CountJobFileRead])
|
||||
def list_count_job_files(task_id: str) -> list[dict[str, Any]]:
|
||||
with session_scope() as session:
|
||||
task = session.get(Task, task_id)
|
||||
if task is None or task.type != "counting":
|
||||
raise HTTPException(status_code=404, detail="Tarea de conteo no encontrada")
|
||||
files = session.scalars(
|
||||
select(CountJobFile)
|
||||
.where(CountJobFile.job_module_id == task_id)
|
||||
.order_by(CountJobFile.created_at)
|
||||
).all()
|
||||
return [serialize_count_file(f) for f in files]
|
||||
|
||||
|
||||
@app.patch("/count-jobs/{task_id}/files/{file_id}", response_model=CountJobFileRead)
|
||||
def update_count_job_file(
|
||||
task_id: str, file_id: str, request: UpdateCountJobFileRequest
|
||||
) -> dict[str, Any]:
|
||||
with session_scope() as session:
|
||||
f = session.get(CountJobFile, file_id)
|
||||
if f is None or f.job_module_id != task_id:
|
||||
raise HTTPException(status_code=404, detail="Archivo no encontrado")
|
||||
f.is_valid = request.is_valid
|
||||
session.flush()
|
||||
return serialize_count_file(f)
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# Extraccion desde resultados de conteo (sin re-subir archivos)
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def run_from_count_task(task_id: str) -> None:
|
||||
"""Procesa archivos validos de un job de conteo en background (lee desde el volumen /data)."""
|
||||
import logging as _logging
|
||||
_log = _logging.getLogger("extraction.from_count")
|
||||
|
||||
from api.desktop_ingest import _service_for_model, mark_task_completed_if_done
|
||||
|
||||
with session_scope() as session:
|
||||
task = session.get(Task, task_id)
|
||||
if task is None:
|
||||
return
|
||||
files_data: list[dict[str, Any]] = list((task.result or {}).get("files", []))
|
||||
req = dict(task.request or {})
|
||||
batch_id = str((task.result or {}).get("extraction_batch_id") or "")
|
||||
|
||||
model_name: str | None = req.get("model")
|
||||
instruction: str | None = req.get("instruction")
|
||||
tiff_page_range = req.get("tiff_page_range")
|
||||
|
||||
for file_data in files_data:
|
||||
file_path: str | None = file_data.get("file_path")
|
||||
filename = str(file_data.get("filename") or "documento.pdf")
|
||||
radicado = str(file_data.get("radicado") or file_data.get("filename") or "")
|
||||
doc_type: str | None = file_data.get("doc_type")
|
||||
|
||||
if not file_path or not __import__("os").path.exists(file_path):
|
||||
_log.warning("Archivo no disponible: %s", file_path or filename)
|
||||
with session_scope() as session:
|
||||
task_obj = session.get(Task, task_id)
|
||||
if task_obj is None:
|
||||
return
|
||||
result = dict(task_obj.result or {})
|
||||
errors = list(result.get("errors") or [])
|
||||
errors.append(f"Archivo no encontrado: {filename}")
|
||||
result["errors"] = errors
|
||||
result["processed"] = int(result.get("processed") or 0) + 1
|
||||
task_obj.result = result
|
||||
task_obj.updated_at = utcnow()
|
||||
mark_task_completed_if_done(session, task_obj, batch_id)
|
||||
continue
|
||||
|
||||
try:
|
||||
svc = _service_for_model(processing_service, model_name)
|
||||
with session_scope() as session:
|
||||
task_obj = session.get(Task, task_id, with_for_update=True)
|
||||
if task_obj is None:
|
||||
return
|
||||
from api.desktop_ingest import process_desktop_ingest
|
||||
process_desktop_ingest(
|
||||
svc, session,
|
||||
task=task_obj,
|
||||
batch_id=batch_id,
|
||||
temp_path=file_path,
|
||||
radicado=radicado,
|
||||
filename=filename,
|
||||
doc_type=doc_type,
|
||||
instruction=instruction,
|
||||
tiff_page_range=tiff_page_range,
|
||||
model=model_name,
|
||||
)
|
||||
mark_task_completed_if_done(session, task_obj, batch_id)
|
||||
except Exception as exc:
|
||||
_log.exception("Error procesando %s: %s", filename, exc)
|
||||
with session_scope() as session:
|
||||
task_obj = session.get(Task, task_id)
|
||||
if task_obj is None:
|
||||
return
|
||||
result = dict(task_obj.result or {})
|
||||
errors = list(result.get("errors") or [])
|
||||
errors.append(f"{filename}: {exc}")
|
||||
result["errors"] = errors
|
||||
result["processed"] = int(result.get("processed") or 0) + 1
|
||||
task_obj.result = result
|
||||
task_obj.updated_at = utcnow()
|
||||
mark_task_completed_if_done(session, task_obj, batch_id)
|
||||
|
||||
|
||||
@app.post("/tasks/extraction/from-count", response_model=TaskRead)
|
||||
def create_extraction_from_count(
|
||||
request: ExtractionFromCountRequest, background_tasks: BackgroundTasks
|
||||
) -> dict[str, Any]:
|
||||
"""Crea una tarea de extraccion usando archivos validos de un job de conteo previo.
|
||||
|
||||
No requiere subir archivos: usa las rutas almacenadas en count_job_files.
|
||||
"""
|
||||
if not request.fields:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Envia fields. Usa POST /extraction/schemas/upload para descubrir campos.",
|
||||
)
|
||||
|
||||
with session_scope() as session:
|
||||
counting_task = session.get(Task, request.counting_task_id)
|
||||
if counting_task is None or counting_task.type != "counting":
|
||||
raise HTTPException(status_code=404, detail="Tarea de conteo no encontrada")
|
||||
|
||||
all_files = session.scalars(
|
||||
select(CountJobFile).where(
|
||||
CountJobFile.job_module_id == request.counting_task_id
|
||||
)
|
||||
).all()
|
||||
|
||||
valid_files = [
|
||||
f for f in all_files
|
||||
if f.status == "ok"
|
||||
and f.is_valid is not False
|
||||
and (f.service_count is None or f.service_count >= request.min_service_count)
|
||||
]
|
||||
|
||||
if not valid_files:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="No hay archivos validos para extraer con los criterios indicados.",
|
||||
)
|
||||
|
||||
files_data = [
|
||||
{
|
||||
"id": f.id,
|
||||
"radicado": f.radicado or f.filename,
|
||||
"filename": f.filename,
|
||||
"file_path": f.file_path,
|
||||
"doc_type": f.doc_type,
|
||||
"service_count": f.service_count,
|
||||
}
|
||||
for f in valid_files
|
||||
]
|
||||
total = len(files_data)
|
||||
|
||||
job = Job(
|
||||
status="running",
|
||||
title=f"Extraccion desde conteo ({total} archivos)",
|
||||
updated_at=utcnow(),
|
||||
)
|
||||
session.add(job)
|
||||
session.flush()
|
||||
|
||||
batch = ExtractionBatch(
|
||||
status="running",
|
||||
root_path="count://",
|
||||
request={
|
||||
"source": "from-count",
|
||||
"counting_task_id": request.counting_task_id,
|
||||
"fields": request.fields,
|
||||
"alias_map": request.alias_map,
|
||||
"min_service_count": request.min_service_count,
|
||||
"structure_mode": request.structure_mode,
|
||||
},
|
||||
updated_at=utcnow(),
|
||||
)
|
||||
session.add(batch)
|
||||
session.flush()
|
||||
|
||||
task = Task(
|
||||
job_id=job.id,
|
||||
type="extraction",
|
||||
status="running",
|
||||
title=job.title,
|
||||
request={
|
||||
"source": "from-count",
|
||||
"counting_task_id": request.counting_task_id,
|
||||
"fields": request.fields,
|
||||
"alias_map": request.alias_map,
|
||||
"doc_types": [dt.model_dump() for dt in request.doc_types or []],
|
||||
"structure_mode": request.structure_mode,
|
||||
"instruction": request.instruction,
|
||||
"tiff_page_range": list(request.tiff_page_range) if request.tiff_page_range else None,
|
||||
"min_service_count": request.min_service_count,
|
||||
"model": request.model,
|
||||
},
|
||||
result={
|
||||
"source": "from-count",
|
||||
"extraction_batch_id": batch.id,
|
||||
"processed": 0,
|
||||
"total": total,
|
||||
"row_count": 0,
|
||||
"errors": [],
|
||||
"files": files_data,
|
||||
"partials": {},
|
||||
},
|
||||
started_at=utcnow(),
|
||||
updated_at=utcnow(),
|
||||
)
|
||||
session.add(task)
|
||||
session.flush()
|
||||
|
||||
batch.job_module_id = task.id
|
||||
session.flush()
|
||||
|
||||
serialized = serialize_task(task)
|
||||
task_id = task.id
|
||||
|
||||
background_tasks.add_task(run_from_count_task, task_id)
|
||||
return serialized
|
||||
|
||||
@@ -9,6 +9,7 @@ from typing import Any, Iterator
|
||||
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
Boolean,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Integer,
|
||||
@@ -42,6 +43,20 @@ def new_id() -> str:
|
||||
return str(uuid.uuid4())
|
||||
|
||||
|
||||
class Job(Base):
|
||||
"""Contenedor de trabajo del usuario; puede abarcar múltiples módulos/servicios."""
|
||||
__tablename__ = "jobs"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_id)
|
||||
status: Mapped[str] = mapped_column(String(32), default="pending", index=True)
|
||||
title: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
# "metadata" es reservado por DeclarativeBase; el atributo Python usa job_metadata
|
||||
# pero la columna en la BD sigue llamándose "metadata".
|
||||
job_metadata: Mapped[dict[str, Any]] = mapped_column("metadata", JSON, default=dict)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
|
||||
|
||||
|
||||
class ExtractionBatch(Base):
|
||||
__tablename__ = "extraction_batches"
|
||||
|
||||
@@ -50,14 +65,21 @@ class ExtractionBatch(Base):
|
||||
root_path: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
request: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict)
|
||||
error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
job_module_id: Mapped[str | None] = mapped_column(
|
||||
String(36), ForeignKey("tasks.id"), nullable=True, index=True
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
|
||||
|
||||
|
||||
class Task(Base):
|
||||
"""Módulo/resultado dentro de un Job; un job puede tener N tasks (una por servicio)."""
|
||||
__tablename__ = "tasks"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_id)
|
||||
job_id: Mapped[str | None] = mapped_column(
|
||||
String(36), ForeignKey("jobs.id"), nullable=True, index=True
|
||||
)
|
||||
type: Mapped[str] = mapped_column(String(64), index=True)
|
||||
status: Mapped[str] = mapped_column(String(32), default="pending", index=True)
|
||||
title: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
@@ -71,6 +93,26 @@ class Task(Base):
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class CountJobFile(Base):
|
||||
"""Resultado del servicio de conteo para un PDF individual dentro de una tarea de conteo."""
|
||||
__tablename__ = "count_job_files"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_id)
|
||||
job_module_id: Mapped[str] = mapped_column(
|
||||
String(36), ForeignKey("tasks.id"), index=True
|
||||
)
|
||||
radicado: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
||||
filename: Mapped[str] = mapped_column(String(512))
|
||||
file_path: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
doc_type: Mapped[str | None] = mapped_column(String(128), nullable=True)
|
||||
service_count: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
services: Mapped[list[Any]] = mapped_column(JSON, default=list)
|
||||
status: Mapped[str] = mapped_column(String(32), default="ok", index=True)
|
||||
is_valid: Mapped[bool | None] = mapped_column(Boolean, nullable=True)
|
||||
error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
|
||||
|
||||
|
||||
class ExtractionRow(Base):
|
||||
__tablename__ = "extraction_rows"
|
||||
|
||||
@@ -109,16 +151,13 @@ class CleanedRow(Base):
|
||||
|
||||
|
||||
def init_db() -> None:
|
||||
if DATABASE_URL.startswith("postgresql"):
|
||||
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as connection:
|
||||
connection.execute(text("SELECT pg_advisory_lock(74512001)"))
|
||||
try:
|
||||
Base.metadata.create_all(bind=connection)
|
||||
finally:
|
||||
connection.execute(text("SELECT pg_advisory_unlock(74512001)"))
|
||||
return
|
||||
"""Verifica conectividad con la BD.
|
||||
|
||||
Base.metadata.create_all(bind=engine)
|
||||
Las migraciones son responsabilidad exclusiva de services/db (init container).
|
||||
Este servicio asume que el esquema ya está actualizado al arrancar.
|
||||
"""
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text("SELECT 1"))
|
||||
|
||||
|
||||
@contextmanager
|
||||
|
||||
Reference in New Issue
Block a user