feat: Add .env.example configuration file and integrate Redis service into Docker Compose for improved task management. Update desktop application to support document indexing and extraction workflows, enhancing user experience with a new extraction wizard and stepper components. Revise README and documentation for clarity on setup and usage.
This commit is contained in:
erik
@@ -0,0 +1,15 @@
|
||||
# Go
|
||||
bin/
|
||||
vendor/
|
||||
|
||||
# Modelos ONNX (regenerables con export_onnx.py)
|
||||
models/latest/*
|
||||
!models/latest/.gitkeep
|
||||
|
||||
# Datos de paridad
|
||||
data/
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
.cursor/
|
||||
@@ -0,0 +1,56 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
|
||||
FROM golang:1.23-bookworm AS builder
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
gcc g++ pkg-config \
|
||||
libtesseract-dev libleptonica-dev \
|
||||
tesseract-ocr tesseract-ocr-spa \
|
||||
wget ca-certificates \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# ONNX Runtime
|
||||
ARG ONNX_VERSION=1.22.0
|
||||
RUN wget -q https://github.com/microsoft/onnxruntime/releases/download/v${ONNX_VERSION}/onnxruntime-linux-x64-${ONNX_VERSION}.tgz \
|
||||
&& tar -xzf onnxruntime-linux-x64-${ONNX_VERSION}.tgz \
|
||||
&& cp onnxruntime-linux-x64-${ONNX_VERSION}/lib/libonnxruntime.so* /usr/local/lib/ \
|
||||
&& ldconfig
|
||||
|
||||
ENV CGO_ENABLED=1
|
||||
ENV ONNXRUNTIME_SHARED_LIBRARY=/usr/local/lib/libonnxruntime.so
|
||||
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
COPY . .
|
||||
|
||||
RUN go build -o /out/api ./cmd/api && go build -o /out/worker ./cmd/worker
|
||||
|
||||
FROM debian:bookworm-slim AS runtime
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
ca-certificates curl wget \
|
||||
tesseract-ocr tesseract-ocr-spa \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
ARG ONNX_VERSION=1.22.0
|
||||
RUN wget -q https://github.com/microsoft/onnxruntime/releases/download/v${ONNX_VERSION}/onnxruntime-linux-x64-${ONNX_VERSION}.tgz \
|
||||
&& tar -xzf onnxruntime-linux-x64-${ONNX_VERSION}.tgz \
|
||||
&& cp onnxruntime-linux-x64-${ONNX_VERSION}/lib/libonnxruntime.so* /usr/local/lib/ \
|
||||
&& rm -rf onnxruntime-linux-x64-${ONNX_VERSION}* \
|
||||
&& ldconfig
|
||||
|
||||
ENV ONNXRUNTIME_SHARED_LIBRARY=/usr/local/lib/libonnxruntime.so
|
||||
|
||||
WORKDIR /app
|
||||
COPY --from=builder /out/api /app/api
|
||||
COPY --from=builder /out/worker /app/worker
|
||||
COPY api/openapi.yaml /app/openapi.yaml
|
||||
|
||||
ENV OPENAPI_PATH=/app/openapi.yaml
|
||||
ENV MODEL_PATH=/app/models/model.onnx
|
||||
ENV INFERENCE_CONFIG=/app/models/inference.json
|
||||
ENV PORT=8080
|
||||
|
||||
EXPOSE 8080
|
||||
CMD ["/app/api"]
|
||||
@@ -0,0 +1,14 @@
|
||||
FROM golang:1.23-bookworm AS builder
|
||||
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
COPY . .
|
||||
RUN go build -o /out/autoscaler ./cmd/autoscaler
|
||||
|
||||
FROM docker:27-cli
|
||||
|
||||
COPY --from=builder /out/autoscaler /usr/local/bin/autoscaler
|
||||
|
||||
WORKDIR /project
|
||||
ENTRYPOINT ["/usr/local/bin/autoscaler"]
|
||||
@@ -0,0 +1,339 @@
|
||||
# Services Count — API de Facturas Médicas
|
||||
|
||||
|
||||
|
||||
Microservicio de inferencia en Go que cuenta servicios facturados en facturas médicas PDF. El entrenamiento del modelo vive en el repositorio hermano [services-count-facturas-medicas-ml](https://github.com/builker-col/services-count-facturas-medicas-ml); este repositorio despliega la API HTTP y los workers de procesamiento.
|
||||
|
||||
## Características
|
||||
|
||||
- Conteo **síncrono** (1–50 PDFs por petición) y **asíncrono** (hasta 50 000 PDFs por job).
|
||||
- Pipeline de inferencia: OCR → clustering DBSCAN → Random Forest vía ONNX.
|
||||
- Escalado horizontal de workers con Docker Compose.
|
||||
- Especificación OpenAPI y descriptor de herramientas para agentes AI.
|
||||
|
||||
## Arquitectura
|
||||
|
||||
El sistema se compone de una API HTTP, workers consumidores y tres servicios de infraestructura (Redis, NATS JetStream y MinIO), definidos en [`docker-compose.yml`](docker-compose.yml).
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
Client[Cliente_o_Agente]
|
||||
API[API_Go_8080]
|
||||
Worker[Worker_Go_xN]
|
||||
Redis[(Redis)]
|
||||
NATS[NATS_JetStream]
|
||||
MinIO[(MinIO)]
|
||||
Pipeline[OCR_Tesseract_ONNX]
|
||||
|
||||
Client -->|sync_POST_v1_count| API
|
||||
Client -->|async_POST_v1_jobs| API
|
||||
API --> Pipeline
|
||||
API --> Redis
|
||||
API --> MinIO
|
||||
API -->|publish| NATS
|
||||
NATS -->|consume| Worker
|
||||
Worker --> MinIO
|
||||
Worker --> Pipeline
|
||||
Worker --> Redis
|
||||
```
|
||||
|
||||
### Pipeline de inferencia
|
||||
|
||||
Por cada PDF se procesa la primera página siguiendo este flujo (`internal/pipeline`):
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
PDF[PDF_primera_pagina] --> Render[Render_MuPDF]
|
||||
Render --> OCR[Tesseract_spa]
|
||||
OCR --> Cluster[DBSCAN_filas]
|
||||
Cluster --> Features[8_features]
|
||||
Features --> Infer[ONNX_RandomForest]
|
||||
Infer --> Result[service_count_y_services]
|
||||
```
|
||||
|
||||
1. **Render** — La primera página se rasteriza con MuPDF (go-fitz) a 300 DPI.
|
||||
2. **OCR** — Tesseract extrae cajas de texto en español.
|
||||
3. **Clustering** — DBSCAN agrupa palabras en filas candidatas.
|
||||
4. **Features** — Se calculan 8 atributos por fila (códigos, moneda, posición, etc.).
|
||||
5. **Inferencia** — Un Random Forest exportado a ONNX clasifica cada fila como servicio o no.
|
||||
|
||||
## Componentes
|
||||
|
||||
| Componente | Binario / servicio | Paquete | Responsabilidad |
|
||||
|------------|-------------------|---------|-----------------|
|
||||
| API HTTP | `cmd/api` | `internal/api` | Endpoints REST, gestión de jobs, health probes |
|
||||
| Worker | `cmd/worker` | `internal/worker` | Consume NATS, descarga PDFs de MinIO, persiste resultados |
|
||||
| Pipeline | — | `internal/pipeline` | Orquestación OCR → clustering → features → inferencia |
|
||||
| Modelo ONNX | — | `internal/model` | Carga `model.onnx` e `inference.json` |
|
||||
| OCR | — | `internal/ocr` | Render PDF (go-fitz) + Tesseract (gosseract) |
|
||||
| Estado de jobs | Redis 7 | `internal/store` | Metadatos, resultados y progreso (TTL 72 h) |
|
||||
| Cola | NATS JetStream | `internal/queue` | Stream `PDF_JOBS`, subject `pdf.process.*` |
|
||||
| Object storage | MinIO | `internal/storage` | Almacenamiento de PDFs en jobs asíncronos |
|
||||
|
||||
### Estructura del repositorio
|
||||
|
||||
```
|
||||
├── cmd/
|
||||
│ ├── api/ # Punto de entrada HTTP
|
||||
│ └── worker/ # Punto de entrada worker
|
||||
├── internal/
|
||||
│ ├── api/ # Servidor HTTP (chi)
|
||||
│ ├── config/ # Variables de entorno
|
||||
│ ├── domain/ # Tipos de dominio
|
||||
│ ├── model/ # Inferencia ONNX
|
||||
│ ├── ocr/ # Extracción OCR
|
||||
│ ├── pipeline/ # Procesamiento por PDF
|
||||
│ ├── queue/ # NATS JetStream
|
||||
│ ├── storage/ # Cliente MinIO
|
||||
│ ├── store/ # Estado en Redis
|
||||
│ └── worker/ # Runner del worker
|
||||
├── api/openapi.yaml # Especificación OpenAPI 3.1
|
||||
├── mcp/tool.json # Descriptor de herramientas para agentes
|
||||
├── web/ # Cliente demo (embebido en la API en /demo/)
|
||||
├── scripts/ # Export ONNX, paridad con Python
|
||||
└── models/latest/ # model.onnx + inference.json (montado en Docker)
|
||||
```
|
||||
|
||||
## Modos de operación
|
||||
|
||||
| Modo | Endpoint principal | Límite | Cuándo usarlo |
|
||||
|------|-------------------|--------|---------------|
|
||||
| Síncrono | `POST /v1/count` | 50 PDFs | Pruebas, integraciones en tiempo real, lotes pequeños |
|
||||
| Asíncrono | `POST /v1/jobs` + poll | 50 000 PDFs/job | MinIO + NATS + workers (path producción) |
|
||||
|
||||
**Flujo asíncrono (producción):**
|
||||
|
||||
1. Crear job (`POST /v1/jobs`) — shell vacío, prefijo MinIO o lista de claves.
|
||||
2. Subir PDFs por chunks (`POST /v1/jobs/{jobID}/upload`, máx. 500 por chunk) → **MinIO Put** → **NATS Publish**.
|
||||
3. Workers consumen la cola, descargan de MinIO, OCR, guardan en Redis.
|
||||
4. Consultar progreso (`GET /v1/jobs/{jobID}?wait_ms=30000` para long-polling).
|
||||
5. Obtener resultados paginados (`GET /v1/jobs/{jobID}/results?cursor=&limit=`).
|
||||
|
||||
## API REST
|
||||
|
||||
Especificación completa: `GET /openapi.json` o [`api/openapi.yaml`](api/openapi.yaml).
|
||||
|
||||
| Método | Ruta | Descripción |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/health` | Liveness probe |
|
||||
| `GET` | `/ready` | Readiness probe |
|
||||
| `POST` | `/v1/count` | Conteo síncrono (multipart `files`) |
|
||||
| `POST` | `/v1/jobs` | Crear job asíncrono |
|
||||
| `POST` | `/v1/jobs/{jobID}/upload` | Subir chunk de PDFs |
|
||||
| `GET` | `/v1/jobs/{jobID}` | Estado y progreso del job |
|
||||
| `GET` | `/v1/jobs/{jobID}/results` | Resultados paginados |
|
||||
| `DELETE` | `/v1/jobs/{jobID}` | Cancelar job |
|
||||
|
||||
**Parámetros relevantes:**
|
||||
|
||||
- `include_rejected` (query) — Si es `true`, incluye todas las filas OCR con el campo `is_service`.
|
||||
- Jobs asíncronos devuelven `202 Accepted`; la cola rechaza con `429` si supera `QUEUE_MAX_PENDING`.
|
||||
|
||||
**Respuesta por PDF:**
|
||||
|
||||
```json
|
||||
{
|
||||
"filename": "factura.pdf",
|
||||
"service_count": 2,
|
||||
"status": "ok",
|
||||
"services": [
|
||||
{ "text": "890201 Consulta ...", "probability": 0.94 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Stack tecnológico
|
||||
|
||||
| Área | Tecnología |
|
||||
|------|------------|
|
||||
| Lenguaje | Go 1.23 |
|
||||
| HTTP router | [go-chi/chi](https://github.com/go-chi/chi) v5 |
|
||||
| OCR | go-fitz (MuPDF) + gosseract (Tesseract) |
|
||||
| ML | ONNX Runtime + Random Forest exportado |
|
||||
| Cola | NATS JetStream |
|
||||
| Estado | Redis 7 |
|
||||
| Storage | MinIO |
|
||||
| Contenedores | Docker multi-stage ([`Dockerfile`](Dockerfile)) |
|
||||
|
||||
## Inicio rápido
|
||||
|
||||
### 1. Exportar el modelo ONNX
|
||||
|
||||
Desde el repositorio de entrenamiento, generar los artefactos de inferencia:
|
||||
|
||||
```bash
|
||||
pip install -r requirements-export.txt
|
||||
python scripts/export_onnx.py \
|
||||
--joblib ../services-count/models/latest/model.joblib \
|
||||
--metadata ../services-count/models/v1.0.0/model_metadata.json
|
||||
```
|
||||
|
||||
Esto produce `models/latest/model.onnx` e `inference.json`.
|
||||
|
||||
### 2. Levantar con Docker Compose
|
||||
|
||||
```bash
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
| Servicio | Puerto / acceso |
|
||||
|----------|-----------------|
|
||||
| API | `http://localhost:8080` |
|
||||
| MinIO Console | `http://localhost:9002` |
|
||||
| Workers | 1 réplica inicial; autoscaler sube hasta 8 según cola |
|
||||
| Autoscaler | Escala `worker` vía Docker socket según `PendingCount` en Redis |
|
||||
|
||||
### 3. Probar
|
||||
|
||||
```bash
|
||||
# Síncrono
|
||||
curl -F "files=@factura.pdf" http://localhost:8080/v1/count
|
||||
|
||||
# Asíncrono (PDFs ya en MinIO bajo un prefijo)
|
||||
curl -X POST http://localhost:8080/v1/jobs \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"prefix":"jobs/batch-1/"}'
|
||||
```
|
||||
|
||||
## Cliente web de ejemplo
|
||||
|
||||
Con la API en ejecución, abre **[http://localhost:8080/demo/](http://localhost:8080/demo/)** para probar la integración desde el navegador. Los archivos estáticos se sirven desde la propia API (mismo origen, sin CORS).
|
||||
|
||||
| Pestaña | Demuestra |
|
||||
|---------|-----------|
|
||||
| **Síncrono** | `POST /v1/count` con subida multipart de hasta 50 PDFs |
|
||||
| **Asíncrono** | `POST /v1/jobs` → upload → polling de estado → resultados paginados |
|
||||
|
||||
El modo asíncrono requiere que workers, Redis, NATS y MinIO estén levantados (igual que con `docker compose up`). Código fuente en [`web/`](web/).
|
||||
|
||||
## Desarrollo local
|
||||
|
||||
Requisitos adicionales a Go 1.23+:
|
||||
|
||||
- Tesseract con datos de idioma `spa`
|
||||
- ONNX Runtime (la variable `ONNXRUNTIME_SHARED_LIBRARY` apunta a la librería compartida)
|
||||
- Redis, NATS y MinIO en ejecución
|
||||
|
||||
```bash
|
||||
go build -o api ./cmd/api
|
||||
go build -o worker ./cmd/worker
|
||||
```
|
||||
|
||||
Configurar las variables de entorno para apuntar a los servicios locales (ver tabla siguiente).
|
||||
|
||||
## Configuración
|
||||
|
||||
Variables definidas en [`internal/config/config.go`](internal/config/config.go):
|
||||
|
||||
| Variable | Default | Descripción |
|
||||
|----------|---------|-------------|
|
||||
| `PORT` | `8080` | Puerto de la API |
|
||||
| `REDIS_URL` | `redis://redis:6379/0` | Conexión Redis |
|
||||
| `NATS_URL` | `nats://nats:4222` | Conexión NATS |
|
||||
| `NATS_STREAM` | `PDF_JOBS` | Nombre del stream JetStream |
|
||||
| `NATS_SUBJECT_PREFIX` | `pdf.process` | Prefijo de subjects |
|
||||
| `MINIO_ENDPOINT` | `minio:9000` | Endpoint MinIO |
|
||||
| `MINIO_ACCESS_KEY` | `minioadmin` | Access key |
|
||||
| `MINIO_SECRET_KEY` | `minioadmin` | Secret key |
|
||||
| `MINIO_BUCKET` | `pdf-count` | Bucket de PDFs |
|
||||
| `MINIO_USE_SSL` | `false` | Usar SSL con MinIO |
|
||||
| `MODEL_PATH` | `/app/models/model.onnx` | Ruta al modelo ONNX |
|
||||
| `INFERENCE_CONFIG` | `/app/models/inference.json` | Metadata de inferencia |
|
||||
| `OCR_DPI` | `300` | DPI de renderizado PDF (150 reduce latencia ~2–4×) |
|
||||
| `OCR_LANG` | `spa` | Idioma Tesseract |
|
||||
| `DBSCAN_EPS` | `10` | Parámetro epsilon de DBSCAN |
|
||||
| `DBSCAN_MIN_SAMPLES` | `1` | Mínimo de muestras DBSCAN |
|
||||
| `SYNC_MAX_FILES` | `50` | Máximo de PDFs en modo síncrono |
|
||||
| `JOB_MAX_FILES` | `50000` | Máximo de PDFs por job |
|
||||
| `QUEUE_MAX_PENDING` | `100000` | Límite de mensajes pendientes en cola |
|
||||
| `WORKER_CONCURRENCY` | `4` | Goroutines paralelas por worker |
|
||||
| `MAX_PDF_SIZE_MB` | `20` | Tamaño máximo por PDF |
|
||||
| `PDF_PROCESS_TIMEOUT_SEC` | `120` | Timeout de procesamiento por PDF |
|
||||
| `JOB_RESULT_TTL_HOURS` | `72` | TTL de resultados en Redis |
|
||||
| `OCR_POOL_SIZE` | `4` (API) / `WORKER_CONCURRENCY` (worker) | Clientes Tesseract reutilizables en pool |
|
||||
| `SYNC_PROCESS_CONCURRENCY` | `4` (o `OCR_POOL_SIZE`) | PDFs procesados en paralelo en `/v1/count` |
|
||||
| `JOB_INLINE_PROCESSING` | `false` | `true` procesa uploads en la API sin MinIO (solo dev local) |
|
||||
| `LOG_PIPELINE_TIMING` | `false` | Logs de duración por etapa del pipeline |
|
||||
| `NATS_FETCH_MAX_WAIT_MS` | `300` | Tiempo máximo de espera del worker al leer NATS |
|
||||
|
||||
## Rendimiento
|
||||
|
||||
El cuello de botella de `/v1/count` es el **OCR con Tesseract**, no la inferencia ONNX. Los clientes Tesseract se crean una vez al arranque y se reutilizan desde un pool (`internal/ocr/pool.go`).
|
||||
|
||||
| Etapa | Tiempo típico (1 PDF) |
|
||||
|-------|------------------------|
|
||||
| Render PDF (MuPDF) | ~100–300 ms |
|
||||
| OCR Tesseract | ~3–8 s |
|
||||
| Clustering + ONNX | ~10–50 ms |
|
||||
|
||||
**Latencia esperada (síncrono):** ~3–10 s por PDF. Varios PDFs se procesan en paralelo (`SYNC_PROCESS_CONCURRENCY`, default 4). Ejemplo: 20 PDFs ≈ `ceil(20/4) × 2.85s ≈ 15s`.
|
||||
|
||||
**Latencia esperada (asíncrono por upload):** path producción MinIO → NATS → worker. Con `WORKER_CONCURRENCY=4` y autoscaler activo, 16 PDFs ≈ `ceil(16/4) × 2.85s` más overhead Put/Get (~15–25 s). 100 PDFs escala hasta 8 workers (32 slots OCR).
|
||||
|
||||
**Asíncrono por prefix MinIO** (`POST /v1/jobs {"prefix":"..."}`): mismos workers; los PDFs ya están en el bucket.
|
||||
|
||||
**Ajustes para reducir latencia:**
|
||||
|
||||
- `OCR_DPI=150` — reduce tiempo de render y OCR (~2–4× más rápido) con posible ligera pérdida de precisión. Validar con `scripts/compare_parity.py` antes de usar en producción.
|
||||
- `OCR_POOL_SIZE=4` y `WORKER_CONCURRENCY=4` — paralelismo por réplica worker.
|
||||
- Upload pipeline publica en NATS tras cada Put (no al final del lote).
|
||||
- `LOG_PIPELINE_TIMING=true` — logs por etapa: `ocr_render`, `ocr_tesseract`, `cluster`, `infer`, `total`.
|
||||
- `NATS_FETCH_MAX_WAIT_MS=300` — el worker recoge mensajes con menor espera.
|
||||
- `GET /v1/jobs/{id}?wait_ms=30000` — long-polling en lugar de poll cada 500 ms.
|
||||
|
||||
## Escalado
|
||||
|
||||
### Autoscaler (docker compose)
|
||||
|
||||
El servicio `autoscaler` lee `PendingCount` en Redis cada 5 s y ajusta réplicas `worker`:
|
||||
|
||||
```
|
||||
desired = clamp(ceil(pending / WORKER_SLOTS), MIN_WORKERS, MAX_WORKERS)
|
||||
```
|
||||
|
||||
| Variable | Default | Descripción |
|
||||
|----------|---------|-------------|
|
||||
| `MIN_WORKERS` | `1` | Réplicas mínimas |
|
||||
| `MAX_WORKERS` | `8` | Réplicas máximas |
|
||||
| `WORKER_SLOTS` | `4` | PDFs en paralelo por réplica (`WORKER_CONCURRENCY`) |
|
||||
| `SCALE_DOWN_IDLE_SEC` | `60` | Segundos sin cola antes de bajar 1 réplica |
|
||||
|
||||
Requiere montar `/var/run/docker.sock` (solo entorno local). En Kubernetes usar HPA sobre lag de NATS o métricas custom.
|
||||
|
||||
### Escalado manual
|
||||
|
||||
```bash
|
||||
docker compose up --scale worker=8 -d
|
||||
```
|
||||
|
||||
Cada worker procesa hasta `WORKER_CONCURRENCY` PDFs en paralelo. NATS JetStream distribuye los mensajes entre réplicas con política work-queue.
|
||||
|
||||
## Integración con agentes (MCP)
|
||||
|
||||
[`mcp/tool.json`](mcp/tool.json) es un **descriptor declarativo** de herramientas para agentes AI (Cursor, etc.). No incluye un servidor MCP embebido.
|
||||
|
||||
| Herramienta | Modo | Flujo |
|
||||
|-------------|------|-------|
|
||||
| `count_pdf_services` | Síncrono | `POST /v1/count` con multipart |
|
||||
| `count_pdf_services_batch` | Asíncrono | Crear job → poll estado → resultados paginados |
|
||||
|
||||
## Paridad con Python
|
||||
|
||||
Scripts para validar que la inferencia Go reproduce los resultados del pipeline Python original:
|
||||
|
||||
```bash
|
||||
set SERVICES_COUNT_ROOT=../services-count
|
||||
python scripts/parity_baseline.py
|
||||
python scripts/compare_parity.py
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Síntoma | Causa probable | Acción |
|
||||
|---------|------------------|--------|
|
||||
| API no arranca | Modelo ONNX ausente | Ejecutar `scripts/export_onnx.py` y verificar `models/latest/` |
|
||||
| `/ready` falla | Redis o NATS no disponibles | Comprobar healthchecks en `docker compose ps` |
|
||||
| Jobs en `pending` sin avance | Workers no consumen cola | Verificar logs del worker y conectividad a NATS/MinIO |
|
||||
| Job atascado en `running` con N/N procesados | Race en metadata del job (versiones antiguas) | Reconstruir con `docker compose up --build`; `GetJob` ahora finaliza jobs completados |
|
||||
| `429` al crear job | Cola saturada | Escalar workers o esperar a que baje `QUEUE_MAX_PENDING` |
|
||||
| Resultados vacíos o erróneos | OCR o modelo desactualizado | Revisar `OCR_LANG=spa`, reexportar ONNX desde el repo ML |
|
||||
@@ -0,0 +1,230 @@
|
||||
openapi: 3.1.0
|
||||
info:
|
||||
title: PDF Count API
|
||||
description: Conteo de servicios facturados en facturas médicas PDF.
|
||||
version: 1.0.0
|
||||
servers:
|
||||
- url: http://localhost:8080
|
||||
paths:
|
||||
/health:
|
||||
get:
|
||||
summary: Liveness probe
|
||||
responses:
|
||||
"200":
|
||||
description: OK
|
||||
/ready:
|
||||
get:
|
||||
summary: Readiness probe
|
||||
responses:
|
||||
"200":
|
||||
description: Ready
|
||||
/v1/count:
|
||||
post:
|
||||
summary: Conteo síncrono (hasta 50 PDFs)
|
||||
parameters:
|
||||
- name: include_rejected
|
||||
in: query
|
||||
schema:
|
||||
type: boolean
|
||||
default: false
|
||||
requestBody:
|
||||
required: true
|
||||
content:
|
||||
multipart/form-data:
|
||||
schema:
|
||||
type: object
|
||||
properties:
|
||||
files:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
format: binary
|
||||
responses:
|
||||
"200":
|
||||
description: Resultados por PDF
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/CountResponse"
|
||||
/v1/jobs:
|
||||
post:
|
||||
summary: Crear job asíncrono
|
||||
requestBody:
|
||||
required: true
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/CreateJobRequest"
|
||||
responses:
|
||||
"202":
|
||||
description: Job aceptado
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/JobStatus"
|
||||
/v1/jobs/{jobID}:
|
||||
get:
|
||||
summary: Estado del job
|
||||
parameters:
|
||||
- name: jobID
|
||||
in: path
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: wait_ms
|
||||
in: query
|
||||
description: Long-polling; espera hasta wait_ms (máx. 30000) a que el job cambie de estado
|
||||
schema:
|
||||
type: integer
|
||||
minimum: 0
|
||||
maximum: 30000
|
||||
responses:
|
||||
"200":
|
||||
description: Progreso
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/JobStatus"
|
||||
delete:
|
||||
summary: Cancelar job
|
||||
parameters:
|
||||
- name: jobID
|
||||
in: path
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: Cancelado
|
||||
/v1/jobs/{jobID}/upload:
|
||||
post:
|
||||
summary: Subir chunk de PDFs a un job
|
||||
parameters:
|
||||
- name: jobID
|
||||
in: path
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: include_rejected
|
||||
in: query
|
||||
schema:
|
||||
type: boolean
|
||||
requestBody:
|
||||
content:
|
||||
multipart/form-data:
|
||||
schema:
|
||||
type: object
|
||||
properties:
|
||||
files:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
format: binary
|
||||
responses:
|
||||
"202":
|
||||
description: Chunk aceptado
|
||||
/v1/jobs/{jobID}/results:
|
||||
get:
|
||||
summary: Resultados paginados
|
||||
parameters:
|
||||
- name: jobID
|
||||
in: path
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: cursor
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: limit
|
||||
in: query
|
||||
schema:
|
||||
type: integer
|
||||
default: 500
|
||||
responses:
|
||||
"200":
|
||||
description: Página de resultados
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/JobResultsPage"
|
||||
components:
|
||||
schemas:
|
||||
ServiceRow:
|
||||
type: object
|
||||
properties:
|
||||
text:
|
||||
type: string
|
||||
probability:
|
||||
type: number
|
||||
format: float
|
||||
is_service:
|
||||
type: boolean
|
||||
PDFResult:
|
||||
type: object
|
||||
properties:
|
||||
filename:
|
||||
type: string
|
||||
service_count:
|
||||
type: integer
|
||||
status:
|
||||
type: string
|
||||
enum: [ok, error, pending]
|
||||
services:
|
||||
type: array
|
||||
items:
|
||||
$ref: "#/components/schemas/ServiceRow"
|
||||
error:
|
||||
type: string
|
||||
CountResponse:
|
||||
type: object
|
||||
properties:
|
||||
results:
|
||||
type: array
|
||||
items:
|
||||
$ref: "#/components/schemas/PDFResult"
|
||||
processed:
|
||||
type: integer
|
||||
failed:
|
||||
type: integer
|
||||
JobStatus:
|
||||
type: object
|
||||
properties:
|
||||
job_id:
|
||||
type: string
|
||||
status:
|
||||
type: string
|
||||
total:
|
||||
type: integer
|
||||
processed:
|
||||
type: integer
|
||||
failed:
|
||||
type: integer
|
||||
JobResultsPage:
|
||||
type: object
|
||||
properties:
|
||||
job_id:
|
||||
type: string
|
||||
results:
|
||||
type: array
|
||||
items:
|
||||
$ref: "#/components/schemas/PDFResult"
|
||||
next_cursor:
|
||||
type: string
|
||||
has_more:
|
||||
type: boolean
|
||||
CreateJobRequest:
|
||||
type: object
|
||||
properties:
|
||||
files:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
prefix:
|
||||
type: string
|
||||
description: Prefijo MinIO con PDFs ya subidos
|
||||
include_rejected:
|
||||
type: boolean
|
||||
default: false
|
||||
callback_url:
|
||||
type: string
|
||||
@@ -0,0 +1,74 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/pys/pdf-count/internal/api"
|
||||
"github.com/pys/pdf-count/internal/config"
|
||||
"github.com/pys/pdf-count/internal/model"
|
||||
"github.com/pys/pdf-count/internal/ocr"
|
||||
"github.com/pys/pdf-count/internal/queue"
|
||||
"github.com/pys/pdf-count/internal/storage"
|
||||
"github.com/pys/pdf-count/internal/store"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cfg := config.Load()
|
||||
ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer cancel()
|
||||
|
||||
predictor, err := model.NewPredictor(cfg.ModelPath, cfg.InferencePath)
|
||||
if err != nil {
|
||||
log.Fatalf("model: %v", err)
|
||||
}
|
||||
defer predictor.Close()
|
||||
|
||||
poolSize := cfg.OCRPoolSize
|
||||
if poolSize <= 0 {
|
||||
poolSize = 4
|
||||
}
|
||||
ocrEngine, err := ocr.NewEngine(cfg.OCRLang, poolSize)
|
||||
if err != nil {
|
||||
log.Fatalf("ocr: %v", err)
|
||||
}
|
||||
defer ocrEngine.Close()
|
||||
|
||||
st, err := store.New(cfg.RedisURL, cfg.JobResultTTL)
|
||||
if err != nil {
|
||||
log.Fatalf("redis: %v", err)
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
stg, err := storage.New(cfg.MinIOEndpoint, cfg.MinIOAccessKey, cfg.MinIOSecretKey, cfg.MinIOBucket, cfg.MinIOUseSSL)
|
||||
if err != nil {
|
||||
log.Fatalf("minio: %v", err)
|
||||
}
|
||||
initCtx, initCancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
if err := stg.EnsureBucket(initCtx); err != nil {
|
||||
initCancel()
|
||||
log.Fatalf("minio bucket: %v", err)
|
||||
}
|
||||
initCancel()
|
||||
|
||||
pub, err := queue.NewPublisher(cfg.NATSURL, cfg.StreamName, cfg.SubjectPrefix)
|
||||
if err != nil {
|
||||
log.Fatalf("nats: %v", err)
|
||||
}
|
||||
|
||||
openAPIPath := os.Getenv("OPENAPI_PATH")
|
||||
if openAPIPath == "" {
|
||||
openAPIPath = "api/openapi.json"
|
||||
}
|
||||
|
||||
srv := api.NewServer(cfg, predictor, ocrEngine, st, stg, pub, api.LoadOpenAPI(openAPIPath))
|
||||
addr := ":" + cfg.Port
|
||||
log.Printf("api listening on %s", addr)
|
||||
if err := api.ListenAndServe(ctx, addr, srv.Router()); err != nil {
|
||||
log.Fatalf("api: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"math"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/pys/pdf-count/internal/store"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cfg := loadConfig()
|
||||
ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer cancel()
|
||||
|
||||
st, err := store.New(cfg.redisURL, 72*time.Hour)
|
||||
if err != nil {
|
||||
log.Fatalf("redis: %v", err)
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
log.Printf("autoscaler started min=%d max=%d slots=%d service=%s",
|
||||
cfg.minWorkers, cfg.maxWorkers, cfg.workerSlots, cfg.composeService)
|
||||
|
||||
var idleSince time.Time
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
log.Println("autoscaler stopped")
|
||||
return
|
||||
default:
|
||||
}
|
||||
|
||||
pending, err := st.PendingCount(ctx)
|
||||
if err != nil {
|
||||
log.Printf("autoscaler pending count: %v", err)
|
||||
time.Sleep(cfg.pollInterval)
|
||||
continue
|
||||
}
|
||||
|
||||
current, err := countWorkers(cfg)
|
||||
if err != nil {
|
||||
log.Printf("autoscaler count workers: %v", err)
|
||||
time.Sleep(cfg.pollInterval)
|
||||
continue
|
||||
}
|
||||
|
||||
desired := cfg.minWorkers
|
||||
if pending > 0 {
|
||||
desired = int(math.Ceil(float64(pending) / float64(cfg.workerSlots)))
|
||||
if desired < cfg.minWorkers {
|
||||
desired = cfg.minWorkers
|
||||
}
|
||||
if desired > cfg.maxWorkers {
|
||||
desired = cfg.maxWorkers
|
||||
}
|
||||
idleSince = time.Time{}
|
||||
} else if current > cfg.minWorkers {
|
||||
if idleSince.IsZero() {
|
||||
idleSince = time.Now()
|
||||
} else if time.Since(idleSince) >= cfg.scaleDownIdle {
|
||||
desired = current - 1
|
||||
idleSince = time.Time{}
|
||||
} else {
|
||||
desired = current
|
||||
}
|
||||
}
|
||||
|
||||
if desired != current {
|
||||
log.Printf("autoscaler pending=%d current=%d desired=%d scaling", pending, current, desired)
|
||||
if err := scaleWorkers(cfg, desired); err != nil {
|
||||
log.Printf("autoscaler scale: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
time.Sleep(cfg.pollInterval)
|
||||
}
|
||||
}
|
||||
|
||||
type autoscalerConfig struct {
|
||||
redisURL string
|
||||
composeFile string
|
||||
composeService string
|
||||
minWorkers int
|
||||
maxWorkers int
|
||||
workerSlots int
|
||||
pollInterval time.Duration
|
||||
scaleDownIdle time.Duration
|
||||
}
|
||||
|
||||
func loadConfig() autoscalerConfig {
|
||||
return autoscalerConfig{
|
||||
redisURL: getenv("REDIS_URL", "redis://redis:6379/0"),
|
||||
composeFile: getenv("COMPOSE_FILE", "/project/docker-compose.yml"),
|
||||
composeService: getenv("COMPOSE_SERVICE", "worker"),
|
||||
minWorkers: getenvInt("MIN_WORKERS", 1),
|
||||
maxWorkers: getenvInt("MAX_WORKERS", 8),
|
||||
workerSlots: getenvInt("WORKER_SLOTS", 4),
|
||||
pollInterval: time.Duration(getenvInt("POLL_INTERVAL_SEC", 5)) * time.Second,
|
||||
scaleDownIdle: time.Duration(getenvInt("SCALE_DOWN_IDLE_SEC", 60)) * time.Second,
|
||||
}
|
||||
}
|
||||
|
||||
func countWorkers(cfg autoscalerConfig) (int, error) {
|
||||
cmd := exec.Command("docker", "compose",
|
||||
"-f", cfg.composeFile,
|
||||
"ps", cfg.composeService,
|
||||
"--status", "running",
|
||||
"-q",
|
||||
)
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
trimmed := strings.TrimSpace(string(out))
|
||||
if trimmed == "" {
|
||||
return 0, nil
|
||||
}
|
||||
return len(strings.Split(trimmed, "\n")), nil
|
||||
}
|
||||
|
||||
func scaleWorkers(cfg autoscalerConfig, n int) error {
|
||||
scaleArg := cfg.composeService + "=" + strconv.Itoa(n)
|
||||
cmd := exec.Command("docker", "compose",
|
||||
"-f", cfg.composeFile,
|
||||
"up", "-d", "--no-recreate",
|
||||
"--scale", scaleArg,
|
||||
cfg.composeService,
|
||||
)
|
||||
cmd.Stdout = os.Stdout
|
||||
cmd.Stderr = os.Stderr
|
||||
return cmd.Run()
|
||||
}
|
||||
|
||||
func getenv(key, fallback string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
func getenvInt(key string, fallback int) int {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
if n, err := strconv.Atoi(v); err == nil {
|
||||
return n
|
||||
}
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
package main
|
||||
|
||||
import "github.com/pys/pdf-count/internal/worker"
|
||||
|
||||
func main() {
|
||||
worker.RunMain()
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
services:␍
|
||||
redis:␍
|
||||
image: redis:7-alpine␍
|
||||
command: redis-server --maxmemory 1gb --maxmemory-policy allkeys-lru␍
|
||||
healthcheck:␍
|
||||
test: ["CMD", "redis-cli", "ping"]␍
|
||||
interval: 5s␍
|
||||
timeout: 3s␍
|
||||
retries: 5␍
|
||||
␍
|
||||
nats:␍
|
||||
image: nats:2-alpine␍
|
||||
command: ["-js", "-m", "8222"]␍
|
||||
healthcheck:␍
|
||||
test: ["CMD", "wget", "-qO-", "http://localhost:8222/healthz"]␍
|
||||
interval: 5s␍
|
||||
timeout: 3s␍
|
||||
retries: 5␍
|
||||
␍
|
||||
minio:␍
|
||||
image: minio/minio:latest␍
|
||||
command: server /data --console-address ":9001"␍
|
||||
environment:␍
|
||||
MINIO_ROOT_USER: minioadmin␍
|
||||
MINIO_ROOT_PASSWORD: minioadmin␍
|
||||
volumes:␍
|
||||
- minio_data:/data␍
|
||||
ports:␍
|
||||
- "9002:9001"␍
|
||||
healthcheck:␍
|
||||
test: ["CMD-SHELL", "curl -f http://localhost:9000/minio/health/live || exit 1"]␍
|
||||
interval: 10s␍
|
||||
timeout: 5s␍
|
||||
retries: 5␍
|
||||
start_period: 10s␍
|
||||
␍
|
||||
api:␍
|
||||
build: .␍
|
||||
command: ["/app/api"]␍
|
||||
ports:␍
|
||||
- "8080:8080"␍
|
||||
depends_on:␍
|
||||
redis:␍
|
||||
condition: service_healthy␍
|
||||
nats:␍
|
||||
condition: service_healthy␍
|
||||
minio:␍
|
||||
condition: service_started␍
|
||||
environment:␍
|
||||
REDIS_URL: redis://redis:6379/0␍
|
||||
NATS_URL: nats://nats:4222␍
|
||||
MINIO_ENDPOINT: minio:9000␍
|
||||
MINIO_ACCESS_KEY: minioadmin␍
|
||||
MINIO_SECRET_KEY: minioadmin␍
|
||||
MINIO_BUCKET: pdf-count␍
|
||||
MODEL_PATH: /app/models/model.onnx␍
|
||||
INFERENCE_CONFIG: /app/models/inference.json␍
|
||||
SYNC_MAX_FILES: 50␍
|
||||
JOB_MAX_FILES: 50000␍
|
||||
QUEUE_MAX_PENDING: 100000␍
|
||||
OPENAPI_PATH: /app/openapi.yaml␍
|
||||
OCR_POOL_SIZE: 4␍
|
||||
SYNC_PROCESS_CONCURRENCY: 4␍
|
||||
JOB_INLINE_PROCESSING: "false"␍
|
||||
LOG_PIPELINE_TIMING: "true"␍
|
||||
volumes:␍
|
||||
- ./models/latest:/app/models:ro␍
|
||||
healthcheck:␍
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]␍
|
||||
interval: 10s␍
|
||||
timeout: 5s␍
|
||||
retries: 5␍
|
||||
␍
|
||||
worker:␍
|
||||
build: .␍
|
||||
command: ["/app/worker"]␍
|
||||
depends_on:␍
|
||||
redis:␍
|
||||
condition: service_healthy␍
|
||||
nats:␍
|
||||
condition: service_healthy␍
|
||||
minio:␍
|
||||
condition: service_started␍
|
||||
environment:␍
|
||||
REDIS_URL: redis://redis:6379/0␍
|
||||
NATS_URL: nats://nats:4222␍
|
||||
MINIO_ENDPOINT: minio:9000␍
|
||||
MINIO_ACCESS_KEY: minioadmin␍
|
||||
MINIO_SECRET_KEY: minioadmin␍
|
||||
MINIO_BUCKET: pdf-count␍
|
||||
MODEL_PATH: /app/models/model.onnx␍
|
||||
INFERENCE_CONFIG: /app/models/inference.json␍
|
||||
WORKER_CONCURRENCY: 4␍
|
||||
PDF_PROCESS_TIMEOUT_SEC: 120␍
|
||||
OCR_POOL_SIZE: 4␍
|
||||
NATS_FETCH_MAX_WAIT_MS: 300␍
|
||||
LOG_PIPELINE_TIMING: "true"␍
|
||||
JOB_INLINE_PROCESSING: "false"␍
|
||||
volumes:␍
|
||||
- ./models/latest:/app/models:ro␍
|
||||
␍
|
||||
autoscaler:␍
|
||||
build:␍
|
||||
context: .␍
|
||||
dockerfile: Dockerfile.autoscaler␍
|
||||
depends_on:␍
|
||||
redis:␍
|
||||
condition: service_healthy␍
|
||||
volumes:␍
|
||||
- /var/run/docker.sock:/var/run/docker.sock␍
|
||||
- .:/project:ro␍
|
||||
working_dir: /project␍
|
||||
environment:␍
|
||||
REDIS_URL: redis://redis:6379/0␍
|
||||
COMPOSE_FILE: /project/docker-compose.yml␍
|
||||
COMPOSE_SERVICE: worker␍
|
||||
MIN_WORKERS: 1␍
|
||||
MAX_WORKERS: 8␍
|
||||
WORKER_SLOTS: 4␍
|
||||
POLL_INTERVAL_SEC: 5␍
|
||||
SCALE_DOWN_IDLE_SEC: 60␍
|
||||
␍
|
||||
volumes:␍
|
||||
minio_data:␍
|
||||
␍
|
||||
@@ -0,0 +1,40 @@
|
||||
module github.com/pys/pdf-count
|
||||
|
||||
go 1.23.0
|
||||
|
||||
toolchain go1.23.4
|
||||
|
||||
require (
|
||||
github.com/gen2brain/go-fitz v1.24.14
|
||||
github.com/go-chi/chi/v5 v5.2.1
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/minio/minio-go/v7 v7.0.90
|
||||
github.com/nats-io/nats.go v1.39.1
|
||||
github.com/otiai10/gosseract/v2 v2.4.1
|
||||
github.com/redis/go-redis/v9 v9.7.3
|
||||
github.com/yalue/onnxruntime_go v1.21.0
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.2.0 // indirect
|
||||
github.com/davecgh/go-spew v1.1.1 // indirect
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/ebitengine/purego v0.8.0 // indirect
|
||||
github.com/go-ini/ini v1.67.0 // indirect
|
||||
github.com/goccy/go-json v0.10.5 // indirect
|
||||
github.com/jupiterrider/ffi v0.2.0 // indirect
|
||||
github.com/klauspost/compress v1.18.0 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.2.10 // indirect
|
||||
github.com/minio/crc64nvme v1.0.1 // indirect
|
||||
github.com/minio/md5-simd v1.1.2 // indirect
|
||||
github.com/nats-io/nkeys v0.4.9 // indirect
|
||||
github.com/nats-io/nuid v1.0.1 // indirect
|
||||
github.com/pmezard/go-difflib v1.0.0 // indirect
|
||||
github.com/rs/xid v1.6.0 // indirect
|
||||
golang.org/x/crypto v0.36.0 // indirect
|
||||
golang.org/x/net v0.38.0 // indirect
|
||||
golang.org/x/sys v0.31.0 // indirect
|
||||
golang.org/x/text v0.23.0 // indirect
|
||||
)
|
||||
@@ -0,0 +1,69 @@
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
|
||||
github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
|
||||
github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/ebitengine/purego v0.8.0 h1:JbqvnEzRvPpxhCJzJJ2y0RbiZ8nyjccVUrSM3q+GvvE=
|
||||
github.com/ebitengine/purego v0.8.0/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
|
||||
github.com/gen2brain/go-fitz v1.24.14 h1:09weRkjVtLYNGo7l0J7DyOwBExbwi8SJ9h8YPhw9WEo=
|
||||
github.com/gen2brain/go-fitz v1.24.14/go.mod h1:0KaZeQgASc20Yp5R/pFzyy7SmP01XcoHKNF842U2/S4=
|
||||
github.com/go-chi/chi/v5 v5.2.1 h1:KOIHODQj58PmL80G2Eak4WdvUzjSJSm0vG72crDCqb8=
|
||||
github.com/go-chi/chi/v5 v5.2.1/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
|
||||
github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
|
||||
github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
|
||||
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
|
||||
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/jupiterrider/ffi v0.2.0 h1:tMM70PexgYNmV+WyaYhJgCvQAvtTCs3wXeILPutihnA=
|
||||
github.com/jupiterrider/ffi v0.2.0/go.mod h1:yqYqX5DdEccAsHeMn+6owkoI2llBLySVAF8dwCDZPVs=
|
||||
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
|
||||
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
|
||||
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||
github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE=
|
||||
github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
||||
github.com/minio/crc64nvme v1.0.1 h1:DHQPrYPdqK7jQG/Ls5CTBZWeex/2FMS3G5XGkycuFrY=
|
||||
github.com/minio/crc64nvme v1.0.1/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg=
|
||||
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
|
||||
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
|
||||
github.com/minio/minio-go/v7 v7.0.90 h1:TmSj1083wtAD0kEYTx7a5pFsv3iRYMsOJ6A4crjA1lE=
|
||||
github.com/minio/minio-go/v7 v7.0.90/go.mod h1:uvMUcGrpgeSAAI6+sD3818508nUyMULw94j2Nxku/Go=
|
||||
github.com/nats-io/nats.go v1.39.1 h1:oTkfKBmz7W047vRxV762M67ZdXeOtUgvbBaNoQ+3PPk=
|
||||
github.com/nats-io/nats.go v1.39.1/go.mod h1:MgRb8oOdigA6cYpEPhXJuRVH6UE/V4jblJ2jQ27IXYM=
|
||||
github.com/nats-io/nkeys v0.4.9 h1:qe9Faq2Gxwi6RZnZMXfmGMZkg3afLLOtrU+gDZJ35b0=
|
||||
github.com/nats-io/nkeys v0.4.9/go.mod h1:jcMqs+FLG+W5YO36OX6wFIFcmpdAns+w1Wm6D3I/evE=
|
||||
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
|
||||
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
|
||||
github.com/otiai10/gosseract/v2 v2.4.1 h1:G8AyBpXEeSlcq8TI85LH/pM5SXk8Djy2GEXisgyblRw=
|
||||
github.com/otiai10/gosseract/v2 v2.4.1/go.mod h1:1gNWP4Hgr2o7yqWfs6r5bZxAatjOIdqWxJLWsTsembk=
|
||||
github.com/otiai10/mint v1.6.3 h1:87qsV/aw1F5as1eH1zS/yqHY85ANKVMgkDrf9rcxbQs=
|
||||
github.com/otiai10/mint v1.6.3/go.mod h1:MJm72SBthJjz8qhefc4z1PYEieWmy8Bku7CjcAqyUSM=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/redis/go-redis/v9 v9.7.3 h1:YpPyAayJV+XErNsatSElgRZZVCwXX9QzkKYNvO7x0wM=
|
||||
github.com/redis/go-redis/v9 v9.7.3/go.mod h1:bGUrSggJ9X9GUmZpZNEOQKaANxSGgOEBRltRTZHSvrA=
|
||||
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
|
||||
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
|
||||
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
|
||||
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
|
||||
github.com/yalue/onnxruntime_go v1.21.0 h1:DdtvfY7OP5gR8mwPDqAOAQckf+KcI30hPNJL8hQaYWI=
|
||||
github.com/yalue/onnxruntime_go v1.21.0/go.mod h1:b4X26A8pekNb1ACJ58wAXgNKeUCGEAQ9dmACut9Sm/4=
|
||||
golang.org/x/crypto v0.36.0 h1:AnAEvhDddvBdpY+uR+MyHmuZzzNqXSe/GvuDeob5L34=
|
||||
golang.org/x/crypto v0.36.0/go.mod h1:Y4J0ReaxCR1IMaabaSMugxJES1EpwhBHhv2bDHklZvc=
|
||||
golang.org/x/net v0.38.0 h1:vRMAPTMaeGqVhG5QyLJHqNDwecKTomGeqbnfZyKlBI8=
|
||||
golang.org/x/net v0.38.0/go.mod h1:ivrbrMbzFq5J41QOQh0siUuly180yBYtLp+CKbEaFx8=
|
||||
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
|
||||
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
||||
golang.org/x/text v0.23.0 h1:D71I7dUrlY+VX0gQShAThNGHFxZ13dGLBHQLVl1mJlY=
|
||||
golang.org/x/text v0.23.0/go.mod h1:/BLNzu4aZCJ1+kcD0DNRotWKage4q2rGVAg4o22unh4=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,53 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/pys/pdf-count/internal/pipeline"
|
||||
"github.com/pys/pdf-count/internal/store"
|
||||
)
|
||||
|
||||
// FilePayload holds a PDF ready for inline job processing.
|
||||
type FilePayload struct {
|
||||
Filename string
|
||||
Data []byte
|
||||
}
|
||||
|
||||
// JobRunner processes uploaded job files in-process using the API OCR pool.
|
||||
type JobRunner struct {
|
||||
store *store.Store
|
||||
newProc func(includeRejected bool) *pipeline.Processor
|
||||
sem chan struct{}
|
||||
}
|
||||
|
||||
func NewJobRunner(st *store.Store, newProc func(includeRejected bool) *pipeline.Processor, concurrency int) *JobRunner {
|
||||
if concurrency < 1 {
|
||||
concurrency = 1
|
||||
}
|
||||
return &JobRunner{
|
||||
store: st,
|
||||
newProc: newProc,
|
||||
sem: make(chan struct{}, concurrency),
|
||||
}
|
||||
}
|
||||
|
||||
// ProcessOne schedules a single PDF for inline OCR (non-blocking).
|
||||
func (jr *JobRunner) ProcessOne(jobID string, file FilePayload, includeRejected bool) {
|
||||
go func() {
|
||||
jr.sem <- struct{}{}
|
||||
defer func() { <-jr.sem }()
|
||||
|
||||
ctx := context.Background()
|
||||
proc := jr.newProc(includeRejected)
|
||||
result := proc.ProcessPDF(file.Filename, file.Data)
|
||||
_ = jr.store.SaveResult(ctx, jobID, result)
|
||||
_ = jr.store.TryCompleteJob(ctx, jobID)
|
||||
}()
|
||||
}
|
||||
|
||||
// ProcessFiles runs OCR in background goroutines and updates job progress in Redis.
|
||||
func (jr *JobRunner) ProcessFiles(jobID string, files []FilePayload, includeRejected bool) {
|
||||
for _, file := range files {
|
||||
jr.ProcessOne(jobID, file, includeRejected)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,591 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/go-chi/chi/v5/middleware"
|
||||
"github.com/google/uuid"
|
||||
"github.com/pys/pdf-count/internal/config"
|
||||
"github.com/pys/pdf-count/internal/domain"
|
||||
"github.com/pys/pdf-count/internal/model"
|
||||
"github.com/pys/pdf-count/internal/ocr"
|
||||
"github.com/pys/pdf-count/internal/pipeline"
|
||||
"github.com/pys/pdf-count/internal/queue"
|
||||
"github.com/pys/pdf-count/internal/storage"
|
||||
"github.com/pys/pdf-count/internal/store"
|
||||
"github.com/pys/pdf-count/web"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
type Server struct {
|
||||
cfg config.Config
|
||||
predictor *model.Predictor
|
||||
ocrEngine *ocr.Engine
|
||||
store *store.Store
|
||||
storage *storage.Client
|
||||
publisher *queue.Publisher
|
||||
jobRunner *JobRunner
|
||||
openAPI []byte
|
||||
ready bool
|
||||
readyMu sync.RWMutex
|
||||
}
|
||||
|
||||
func NewServer(cfg config.Config, predictor *model.Predictor, ocrEngine *ocr.Engine, st *store.Store, stg *storage.Client, pub *queue.Publisher, openAPI []byte) *Server {
|
||||
srv := &Server{
|
||||
cfg: cfg,
|
||||
predictor: predictor,
|
||||
ocrEngine: ocrEngine,
|
||||
store: st,
|
||||
storage: stg,
|
||||
publisher: pub,
|
||||
openAPI: openAPI,
|
||||
ready: true,
|
||||
}
|
||||
srv.jobRunner = NewJobRunner(st, srv.newProcessor, srv.ocrConcurrency())
|
||||
return srv
|
||||
}
|
||||
|
||||
func (s *Server) processConcurrency() int {
|
||||
c := s.cfg.SyncProcessConcurrency
|
||||
if c <= 0 {
|
||||
c = s.cfg.OCRPoolSize
|
||||
}
|
||||
if c <= 0 {
|
||||
c = 4
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func (s *Server) ocrConcurrency() int {
|
||||
c := s.processConcurrency()
|
||||
pool := s.cfg.OCRPoolSize
|
||||
if pool <= 0 {
|
||||
pool = 4
|
||||
}
|
||||
if c > pool {
|
||||
c = pool
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func (s *Server) SetReady(v bool) {
|
||||
s.readyMu.Lock()
|
||||
s.ready = v
|
||||
s.readyMu.Unlock()
|
||||
}
|
||||
|
||||
func (s *Server) Router() http.Handler {
|
||||
r := chi.NewRouter()
|
||||
r.Use(middleware.RequestID)
|
||||
r.Use(middleware.RealIP)
|
||||
r.Use(middleware.Recoverer)
|
||||
|
||||
r.Get("/health", s.health)
|
||||
r.Get("/ready", s.readyHandler)
|
||||
r.Get("/openapi.json", s.openAPIHandler)
|
||||
|
||||
r.Get("/demo", func(w http.ResponseWriter, r *http.Request) {
|
||||
http.Redirect(w, r, "/demo/", http.StatusMovedPermanently)
|
||||
})
|
||||
r.Handle("/demo/*", http.StripPrefix("/demo/", http.FileServer(http.FS(web.FS))))
|
||||
|
||||
r.Route("/v1", func(r chi.Router) {
|
||||
r.Post("/count", s.countSync)
|
||||
r.Post("/jobs", s.createJob)
|
||||
r.Post("/jobs/{jobID}/upload", s.uploadChunk)
|
||||
r.Get("/jobs/{jobID}", s.getJob)
|
||||
r.Get("/jobs/{jobID}/results", s.getJobResults)
|
||||
r.Delete("/jobs/{jobID}", s.cancelJob)
|
||||
})
|
||||
return r
|
||||
}
|
||||
|
||||
func (s *Server) health(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
|
||||
}
|
||||
|
||||
func (s *Server) readyHandler(w http.ResponseWriter, r *http.Request) {
|
||||
s.readyMu.RLock()
|
||||
ok := s.ready
|
||||
s.readyMu.RUnlock()
|
||||
if !ok {
|
||||
writeJSON(w, http.StatusServiceUnavailable, map[string]string{"status": "not_ready"})
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"status": "ready"})
|
||||
}
|
||||
|
||||
func (s *Server) openAPIHandler(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write(s.openAPI)
|
||||
}
|
||||
|
||||
func (s *Server) countSync(w http.ResponseWriter, r *http.Request) {
|
||||
includeRejected := r.URL.Query().Get("include_rejected") == "true"
|
||||
maxBytes := int64(s.cfg.MaxPDFSizeMB) * 1024 * 1024
|
||||
|
||||
if err := r.ParseMultipartForm(maxBytes * int64(s.cfg.SyncMaxFiles)); err != nil {
|
||||
httpError(w, http.StatusBadRequest, "invalid multipart form")
|
||||
return
|
||||
}
|
||||
|
||||
files := r.MultipartForm.File["files"]
|
||||
if len(files) == 0 {
|
||||
httpError(w, http.StatusBadRequest, "field 'files' required")
|
||||
return
|
||||
}
|
||||
if len(files) > s.cfg.SyncMaxFiles {
|
||||
httpError(w, http.StatusBadRequest, "too many files")
|
||||
return
|
||||
}
|
||||
|
||||
proc := s.newProcessor(includeRejected)
|
||||
type fileInput struct {
|
||||
filename string
|
||||
data []byte
|
||||
errRes *domain.PDFResult
|
||||
}
|
||||
inputs := make([]fileInput, 0, len(files))
|
||||
|
||||
for _, fh := range files {
|
||||
f, err := fh.Open()
|
||||
if err != nil {
|
||||
errRes := domain.PDFResult{
|
||||
Filename: filepath.Base(fh.Filename),
|
||||
Status: domain.StatusError,
|
||||
Error: err.Error(),
|
||||
}
|
||||
inputs = append(inputs, fileInput{errRes: &errRes})
|
||||
continue
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(f, maxBytes+1))
|
||||
_ = f.Close()
|
||||
if err != nil {
|
||||
errRes := domain.PDFResult{
|
||||
Filename: filepath.Base(fh.Filename),
|
||||
Status: domain.StatusError,
|
||||
Error: err.Error(),
|
||||
}
|
||||
inputs = append(inputs, fileInput{errRes: &errRes})
|
||||
continue
|
||||
}
|
||||
if int64(len(data)) > maxBytes {
|
||||
errRes := domain.PDFResult{
|
||||
Filename: filepath.Base(fh.Filename),
|
||||
Status: domain.StatusError,
|
||||
Error: "file too large",
|
||||
}
|
||||
inputs = append(inputs, fileInput{errRes: &errRes})
|
||||
continue
|
||||
}
|
||||
inputs = append(inputs, fileInput{
|
||||
filename: filepath.Base(fh.Filename),
|
||||
data: data,
|
||||
})
|
||||
}
|
||||
|
||||
results := make([]domain.PDFResult, len(inputs))
|
||||
failed := 0
|
||||
sem := make(chan struct{}, s.ocrConcurrency())
|
||||
var wg sync.WaitGroup
|
||||
var mu sync.Mutex
|
||||
|
||||
for i, in := range inputs {
|
||||
if in.errRes != nil {
|
||||
results[i] = *in.errRes
|
||||
failed++
|
||||
continue
|
||||
}
|
||||
wg.Add(1)
|
||||
go func(idx int, in fileInput) {
|
||||
defer wg.Done()
|
||||
sem <- struct{}{}
|
||||
defer func() { <-sem }()
|
||||
res := proc.ProcessPDF(in.filename, in.data)
|
||||
mu.Lock()
|
||||
results[idx] = res
|
||||
if res.Status == domain.StatusError {
|
||||
failed++
|
||||
}
|
||||
mu.Unlock()
|
||||
}(i, in)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
writeJSON(w, http.StatusOK, domain.CountResponse{
|
||||
Results: results,
|
||||
Processed: len(results) - failed,
|
||||
Failed: failed,
|
||||
})
|
||||
}
|
||||
|
||||
type createJobRequest struct {
|
||||
Files []string `json:"files"`
|
||||
Prefix string `json:"prefix"`
|
||||
IncludeRejected bool `json:"include_rejected"`
|
||||
CallbackURL string `json:"callback_url"`
|
||||
}
|
||||
|
||||
func (s *Server) createJob(w http.ResponseWriter, r *http.Request) {
|
||||
var req createJobRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
httpError(w, http.StatusBadRequest, "invalid json body")
|
||||
return
|
||||
}
|
||||
|
||||
ctx := r.Context()
|
||||
pending, err := s.store.PendingCount(ctx)
|
||||
if err == nil && int(pending) > s.cfg.QueueMaxPending {
|
||||
httpError(w, http.StatusTooManyRequests, "queue full")
|
||||
return
|
||||
}
|
||||
|
||||
jobID := uuid.NewString()
|
||||
var objectKeys []string
|
||||
var filenames []string
|
||||
|
||||
if req.Prefix != "" {
|
||||
prefix := strings.TrimSuffix(req.Prefix, "/")
|
||||
keys, err := s.storage.ListByPrefix(ctx, prefix)
|
||||
if err != nil {
|
||||
httpError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
objectKeys = keys
|
||||
for _, k := range keys {
|
||||
filenames = append(filenames, filepath.Base(k))
|
||||
}
|
||||
} else if len(req.Files) > 0 {
|
||||
for _, f := range req.Files {
|
||||
name := filepath.Base(f)
|
||||
objectKeys = append(objectKeys, storage.JobObjectKey(jobID, name))
|
||||
filenames = append(filenames, name)
|
||||
}
|
||||
} else {
|
||||
// Empty job shell for chunked upload flow.
|
||||
if err := s.store.CreateJob(ctx, jobID, 0, req.IncludeRejected); err != nil {
|
||||
httpError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, domain.JobStatus{
|
||||
JobID: jobID,
|
||||
Status: domain.JobQueued,
|
||||
Total: 0,
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
if len(objectKeys) > s.cfg.JobMaxFiles {
|
||||
httpError(w, http.StatusBadRequest, "too many files in job")
|
||||
return
|
||||
}
|
||||
|
||||
if err := s.store.CreateJob(ctx, jobID, len(objectKeys), req.IncludeRejected); err != nil {
|
||||
httpError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.store.AddJobFiles(ctx, jobID, filenames); err != nil {
|
||||
httpError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
_ = s.store.MarkRunning(ctx, jobID)
|
||||
|
||||
for i, key := range objectKeys {
|
||||
msg := domain.QueueMessage{
|
||||
JobID: jobID,
|
||||
ObjectKey: key,
|
||||
Filename: filenames[i],
|
||||
IncludeRejected: req.IncludeRejected,
|
||||
}
|
||||
if err := s.publisher.Publish(ctx, msg); err != nil {
|
||||
httpError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusAccepted, domain.JobStatus{
|
||||
JobID: jobID,
|
||||
Status: domain.JobRunning,
|
||||
Total: len(objectKeys),
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) uploadChunk(w http.ResponseWriter, r *http.Request) {
|
||||
jobID := chi.URLParam(r, "jobID")
|
||||
includeRejected := r.URL.Query().Get("include_rejected") == "true"
|
||||
maxBytes := int64(s.cfg.MaxPDFSizeMB) * 1024 * 1024
|
||||
|
||||
if err := r.ParseMultipartForm(maxBytes * 500); err != nil {
|
||||
httpError(w, http.StatusBadRequest, "invalid multipart form")
|
||||
return
|
||||
}
|
||||
files := r.MultipartForm.File["files"]
|
||||
if len(files) == 0 {
|
||||
httpError(w, http.StatusBadRequest, "field 'files' required")
|
||||
return
|
||||
}
|
||||
if len(files) > 500 {
|
||||
httpError(w, http.StatusBadRequest, "max 500 files per chunk")
|
||||
return
|
||||
}
|
||||
|
||||
ctx := r.Context()
|
||||
status, err := s.store.GetJob(ctx, jobID)
|
||||
if err != nil {
|
||||
_ = s.store.CreateJob(ctx, jobID, 0, includeRejected)
|
||||
status = domain.JobStatus{JobID: jobID, Status: domain.JobQueued}
|
||||
}
|
||||
if status.Status == domain.JobCancelled {
|
||||
httpError(w, http.StatusConflict, "job cancelled")
|
||||
return
|
||||
}
|
||||
|
||||
var filenames []string
|
||||
|
||||
if s.cfg.JobInlineProcessing {
|
||||
_ = s.store.MarkRunning(ctx, jobID)
|
||||
for _, fh := range files {
|
||||
f, err := fh.Open()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(f, maxBytes+1))
|
||||
_ = f.Close()
|
||||
if err != nil || int64(len(data)) > maxBytes {
|
||||
continue
|
||||
}
|
||||
name := filepath.Base(fh.Filename)
|
||||
if err := s.store.AddJobFiles(ctx, jobID, []string{name}); err != nil {
|
||||
httpError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
if err := s.store.IncrementTotal(ctx, jobID, 1); err != nil {
|
||||
httpError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
filenames = append(filenames, name)
|
||||
s.jobRunner.ProcessOne(jobID, FilePayload{Filename: name, Data: data}, includeRejected)
|
||||
}
|
||||
} else {
|
||||
var err error
|
||||
filenames, err = s.uploadChunkMinIO(ctx, jobID, files, includeRejected, maxBytes)
|
||||
if err != nil {
|
||||
httpError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if len(filenames) == 0 {
|
||||
httpError(w, http.StatusBadRequest, "no valid files uploaded")
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusAccepted, map[string]interface{}{
|
||||
"job_id": jobID,
|
||||
"uploaded": len(filenames),
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) uploadChunkMinIO(ctx context.Context, jobID string, files []*multipart.FileHeader, includeRejected bool, maxBytes int64) ([]string, error) {
|
||||
filenames := make([]string, 0, len(files))
|
||||
for _, fh := range files {
|
||||
filenames = append(filenames, filepath.Base(fh.Filename))
|
||||
}
|
||||
if len(filenames) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
if err := s.store.AddJobFiles(ctx, jobID, filenames); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := s.store.IncrementTotal(ctx, jobID, len(filenames)); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := s.store.MarkRunning(ctx, jobID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
ioLimit := 8
|
||||
if len(files) < ioLimit {
|
||||
ioLimit = len(files)
|
||||
}
|
||||
ioSem := make(chan struct{}, ioLimit)
|
||||
|
||||
var wg sync.WaitGroup
|
||||
var mu sync.Mutex
|
||||
uploadedNames := make([]string, 0, len(files))
|
||||
|
||||
for _, fh := range files {
|
||||
wg.Add(1)
|
||||
go func(fh *multipart.FileHeader) {
|
||||
defer wg.Done()
|
||||
ioSem <- struct{}{}
|
||||
defer func() { <-ioSem }()
|
||||
|
||||
name := filepath.Base(fh.Filename)
|
||||
f, err := fh.Open()
|
||||
if err != nil {
|
||||
s.recordUploadError(ctx, jobID, name, err.Error())
|
||||
return
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(f, maxBytes+1))
|
||||
_ = f.Close()
|
||||
if err != nil {
|
||||
s.recordUploadError(ctx, jobID, name, err.Error())
|
||||
return
|
||||
}
|
||||
if int64(len(data)) > maxBytes {
|
||||
s.recordUploadError(ctx, jobID, name, "file too large")
|
||||
return
|
||||
}
|
||||
|
||||
key := storage.JobObjectKey(jobID, name)
|
||||
if err := s.storage.Put(ctx, key, data, "application/pdf"); err != nil {
|
||||
s.recordUploadError(ctx, jobID, name, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
msg := domain.QueueMessage{
|
||||
JobID: jobID,
|
||||
ObjectKey: key,
|
||||
Filename: name,
|
||||
IncludeRejected: includeRejected,
|
||||
}
|
||||
if err := s.publisher.Publish(ctx, msg); err != nil {
|
||||
s.recordUploadError(ctx, jobID, name, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
mu.Lock()
|
||||
uploadedNames = append(uploadedNames, name)
|
||||
mu.Unlock()
|
||||
}(fh)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
if len(uploadedNames) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
return uploadedNames, nil
|
||||
}
|
||||
|
||||
func (s *Server) recordUploadError(ctx context.Context, jobID, filename, errMsg string) {
|
||||
_ = s.store.SaveResult(ctx, jobID, domain.PDFResult{
|
||||
Filename: filename,
|
||||
Status: domain.StatusError,
|
||||
Error: errMsg,
|
||||
})
|
||||
_ = s.store.TryCompleteJob(ctx, jobID)
|
||||
}
|
||||
|
||||
func (s *Server) getJob(w http.ResponseWriter, r *http.Request) {
|
||||
jobID := chi.URLParam(r, "jobID")
|
||||
waitMs, _ := strconv.Atoi(r.URL.Query().Get("wait_ms"))
|
||||
if waitMs > 30000 {
|
||||
waitMs = 30000
|
||||
}
|
||||
|
||||
deadline := time.Now()
|
||||
if waitMs > 0 {
|
||||
deadline = deadline.Add(time.Duration(waitMs) * time.Millisecond)
|
||||
}
|
||||
|
||||
for {
|
||||
status, err := s.store.GetJob(r.Context(), jobID)
|
||||
if err != nil {
|
||||
httpError(w, http.StatusNotFound, "job not found")
|
||||
return
|
||||
}
|
||||
if isTerminalJobStatus(status.Status) || waitMs <= 0 || !time.Now().Before(deadline) {
|
||||
writeJSON(w, http.StatusOK, status)
|
||||
return
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func isTerminalJobStatus(status string) bool {
|
||||
return status == domain.JobCompleted || status == domain.JobCancelled
|
||||
}
|
||||
|
||||
func (s *Server) getJobResults(w http.ResponseWriter, r *http.Request) {
|
||||
jobID := chi.URLParam(r, "jobID")
|
||||
cursor := r.URL.Query().Get("cursor")
|
||||
limit, _ := strconv.Atoi(r.URL.Query().Get("limit"))
|
||||
page, err := s.store.GetResultsPage(r.Context(), jobID, cursor, limit)
|
||||
if err != nil {
|
||||
httpError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, page)
|
||||
}
|
||||
|
||||
func (s *Server) cancelJob(w http.ResponseWriter, r *http.Request) {
|
||||
jobID := chi.URLParam(r, "jobID")
|
||||
if err := s.store.CancelJob(r.Context(), jobID); err != nil {
|
||||
httpError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"job_id": jobID, "status": domain.JobCancelled})
|
||||
}
|
||||
|
||||
func (s *Server) newProcessor(includeRejected bool) *pipeline.Processor {
|
||||
return pipeline.NewProcessor(s.predictor, s.ocrEngine, pipeline.Options{
|
||||
DPI: s.cfg.OCRDPI,
|
||||
Lang: s.cfg.OCRLang,
|
||||
DBSCANEps: s.cfg.DBSCANEps,
|
||||
DBSCANMinSamples: s.cfg.DBSCANMinSamples,
|
||||
IncludeRejected: includeRejected,
|
||||
LogPipelineTiming: s.cfg.LogPipelineTiming,
|
||||
})
|
||||
}
|
||||
|
||||
func writeJSON(w http.ResponseWriter, status int, v interface{}) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
|
||||
func httpError(w http.ResponseWriter, status int, msg string) {
|
||||
writeJSON(w, status, map[string]string{"error": msg})
|
||||
}
|
||||
|
||||
func LoadOpenAPI(path string) []byte {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return []byte(`{"openapi":"3.1.0","info":{"title":"pdf-count","version":"1.0.0"}}`)
|
||||
}
|
||||
if strings.HasSuffix(strings.ToLower(path), ".yaml") || strings.HasSuffix(strings.ToLower(path), ".yml") {
|
||||
var doc interface{}
|
||||
if err := yaml.Unmarshal(data, &doc); err == nil {
|
||||
if j, err := json.Marshal(doc); err == nil {
|
||||
return j
|
||||
}
|
||||
}
|
||||
}
|
||||
return data
|
||||
}
|
||||
|
||||
func ListenAndServe(ctx context.Context, addr string, handler http.Handler) error {
|
||||
srv := &http.Server{Addr: addr, Handler: handler}
|
||||
go func() {
|
||||
<-ctx.Done()
|
||||
_ = srv.Shutdown(context.Background())
|
||||
}()
|
||||
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
Port string
|
||||
RedisURL string
|
||||
NATSURL string
|
||||
MinIOEndpoint string
|
||||
MinIOAccessKey string
|
||||
MinIOSecretKey string
|
||||
MinIOBucket string
|
||||
MinIOUseSSL bool
|
||||
ModelPath string
|
||||
InferencePath string
|
||||
OCRDPI int
|
||||
OCRLang string
|
||||
DBSCANEps float64
|
||||
DBSCANMinSamples int
|
||||
SyncMaxFiles int
|
||||
JobMaxFiles int
|
||||
QueueMaxPending int
|
||||
WorkerConcurrency int
|
||||
MaxPDFSizeMB int
|
||||
PDFProcessTimeout time.Duration
|
||||
JobResultTTL time.Duration
|
||||
StreamName string
|
||||
SubjectPrefix string
|
||||
OCRPoolSize int
|
||||
LogPipelineTiming bool
|
||||
NATSFetchMaxWait time.Duration
|
||||
JobInlineProcessing bool
|
||||
SyncProcessConcurrency int
|
||||
}
|
||||
|
||||
func Load() Config {
|
||||
return Config{
|
||||
Port: getenv("PORT", "8080"),
|
||||
RedisURL: getenv("REDIS_URL", "redis://redis:6379/0"),
|
||||
NATSURL: getenv("NATS_URL", "nats://nats:4222"),
|
||||
MinIOEndpoint: getenv("MINIO_ENDPOINT", "minio:9000"),
|
||||
MinIOAccessKey: getenv("MINIO_ACCESS_KEY", "minioadmin"),
|
||||
MinIOSecretKey: getenv("MINIO_SECRET_KEY", "minioadmin"),
|
||||
MinIOBucket: getenv("MINIO_BUCKET", "pdf-count"),
|
||||
MinIOUseSSL: getenvBool("MINIO_USE_SSL", false),
|
||||
ModelPath: getenv("MODEL_PATH", "/app/models/model.onnx"),
|
||||
InferencePath: getenv("INFERENCE_CONFIG", "/app/models/inference.json"),
|
||||
OCRDPI: getenvInt("OCR_DPI", 300),
|
||||
OCRLang: getenv("OCR_LANG", "spa"),
|
||||
DBSCANEps: getenvFloat("DBSCAN_EPS", 10),
|
||||
DBSCANMinSamples: getenvInt("DBSCAN_MIN_SAMPLES", 1),
|
||||
SyncMaxFiles: getenvInt("SYNC_MAX_FILES", 50),
|
||||
JobMaxFiles: getenvInt("JOB_MAX_FILES", 50000),
|
||||
QueueMaxPending: getenvInt("QUEUE_MAX_PENDING", 100000),
|
||||
WorkerConcurrency: getenvInt("WORKER_CONCURRENCY", 4),
|
||||
MaxPDFSizeMB: getenvInt("MAX_PDF_SIZE_MB", 20),
|
||||
PDFProcessTimeout: time.Duration(getenvInt("PDF_PROCESS_TIMEOUT_SEC", 120)) * time.Second,
|
||||
JobResultTTL: time.Duration(getenvInt("JOB_RESULT_TTL_HOURS", 72)) * time.Hour,
|
||||
StreamName: getenv("NATS_STREAM", "PDF_JOBS"),
|
||||
SubjectPrefix: getenv("NATS_SUBJECT_PREFIX", "pdf.process"),
|
||||
OCRPoolSize: getenvInt("OCR_POOL_SIZE", 0),
|
||||
LogPipelineTiming: getenvBool("LOG_PIPELINE_TIMING", false),
|
||||
NATSFetchMaxWait: time.Duration(getenvInt("NATS_FETCH_MAX_WAIT_MS", 300)) * time.Millisecond,
|
||||
JobInlineProcessing: getenvBool("JOB_INLINE_PROCESSING", false),
|
||||
SyncProcessConcurrency: getenvInt("SYNC_PROCESS_CONCURRENCY", 0),
|
||||
}
|
||||
}
|
||||
|
||||
func getenv(key, fallback string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
func getenvInt(key string, fallback int) int {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
if n, err := strconv.Atoi(v); err == nil {
|
||||
return n
|
||||
}
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
func getenvFloat(key string, fallback float64) float64 {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
if n, err := strconv.ParseFloat(v, 64); err == nil {
|
||||
return n
|
||||
}
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
func getenvBool(key string, fallback bool) bool {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
if b, err := strconv.ParseBool(v); err == nil {
|
||||
return b
|
||||
}
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package domain
|
||||
|
||||
// ServiceRow is one detected invoice line.
|
||||
type ServiceRow struct {
|
||||
Text string `json:"text"`
|
||||
Probability float64 `json:"probability"`
|
||||
IsService *bool `json:"is_service,omitempty"`
|
||||
}
|
||||
|
||||
// PDFResult is the processing outcome for a single PDF.
|
||||
type PDFResult struct {
|
||||
Filename string `json:"filename"`
|
||||
ServiceCount int `json:"service_count"`
|
||||
Status string `json:"status"`
|
||||
Services []ServiceRow `json:"services,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// CountResponse wraps sync batch results.
|
||||
type CountResponse struct {
|
||||
Results []PDFResult `json:"results"`
|
||||
Processed int `json:"processed"`
|
||||
Failed int `json:"failed"`
|
||||
}
|
||||
|
||||
// JobStatus tracks async job progress.
|
||||
type JobStatus struct {
|
||||
JobID string `json:"job_id"`
|
||||
Status string `json:"status"`
|
||||
Total int `json:"total"`
|
||||
Processed int `json:"processed"`
|
||||
Failed int `json:"failed"`
|
||||
}
|
||||
|
||||
// JobResultsPage is a paginated slice of PDF results.
|
||||
type JobResultsPage struct {
|
||||
JobID string `json:"job_id"`
|
||||
Results []PDFResult `json:"results"`
|
||||
NextCursor string `json:"next_cursor,omitempty"`
|
||||
HasMore bool `json:"has_more"`
|
||||
}
|
||||
|
||||
// QueueMessage is published per PDF for workers.
|
||||
type QueueMessage struct {
|
||||
JobID string `json:"job_id"`
|
||||
ObjectKey string `json:"object_key"`
|
||||
Filename string `json:"filename"`
|
||||
IncludeRejected bool `json:"include_rejected"`
|
||||
}
|
||||
|
||||
const (
|
||||
StatusOK = "ok"
|
||||
StatusError = "error"
|
||||
|
||||
JobQueued = "queued"
|
||||
JobRunning = "running"
|
||||
JobCompleted = "completed"
|
||||
JobCancelled = "cancelled"
|
||||
)
|
||||
@@ -0,0 +1,110 @@
|
||||
package model
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"math"
|
||||
"os"
|
||||
"sync"
|
||||
|
||||
ort "github.com/yalue/onnxruntime_go"
|
||||
)
|
||||
|
||||
type InferenceMeta struct {
|
||||
Threshold float64 `json:"threshold"`
|
||||
FeatureColumns []string `json:"feature_columns"`
|
||||
}
|
||||
|
||||
type Predictor struct {
|
||||
session *ort.DynamicAdvancedSession
|
||||
threshold float64
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
func NewPredictor(modelPath, metaPath string) (*Predictor, error) {
|
||||
if lib := os.Getenv("ONNXRUNTIME_SHARED_LIBRARY"); lib != "" {
|
||||
ort.SetSharedLibraryPath(lib)
|
||||
}
|
||||
if err := ort.InitializeEnvironment(); err != nil {
|
||||
return nil, fmt.Errorf("onnx init: %w", err)
|
||||
}
|
||||
|
||||
metaBytes, err := os.ReadFile(metaPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read inference meta: %w", err)
|
||||
}
|
||||
var meta InferenceMeta
|
||||
if err := json.Unmarshal(metaBytes, &meta); err != nil {
|
||||
return nil, fmt.Errorf("parse inference meta: %w", err)
|
||||
}
|
||||
if meta.Threshold == 0 {
|
||||
meta.Threshold = 0.5
|
||||
}
|
||||
|
||||
session, err := ort.NewDynamicAdvancedSession(
|
||||
modelPath,
|
||||
[]string{"float_input"},
|
||||
[]string{"probabilities"},
|
||||
nil,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("onnx session: %w", err)
|
||||
}
|
||||
|
||||
return &Predictor{session: session, threshold: meta.Threshold}, nil
|
||||
}
|
||||
|
||||
func (p *Predictor) Threshold() float64 {
|
||||
return p.threshold
|
||||
}
|
||||
|
||||
func (p *Predictor) Close() error {
|
||||
if p.session != nil {
|
||||
p.session.Destroy()
|
||||
}
|
||||
return ort.DestroyEnvironment()
|
||||
}
|
||||
|
||||
func (p *Predictor) PredictProba(features [][8]float32) ([]float64, error) {
|
||||
if len(features) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
n := len(features)
|
||||
flat := make([]float32, 0, n*8)
|
||||
for _, row := range features {
|
||||
flat = append(flat, row[:]...)
|
||||
}
|
||||
|
||||
inputShape := ort.NewShape(int64(n), 8)
|
||||
inputTensor, err := ort.NewTensor(inputShape, flat)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer inputTensor.Destroy()
|
||||
|
||||
outputShape := ort.NewShape(int64(n), 2)
|
||||
outputTensor, err := ort.NewEmptyTensor[float32](outputShape)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer outputTensor.Destroy()
|
||||
|
||||
p.mu.Lock()
|
||||
err = p.session.Run([]ort.Value{inputTensor}, []ort.Value{outputTensor})
|
||||
p.mu.Unlock()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
data := outputTensor.GetData()
|
||||
probs := make([]float64, n)
|
||||
for i := 0; i < n; i++ {
|
||||
probs[i] = float64(data[i*2+1])
|
||||
}
|
||||
return probs, nil
|
||||
}
|
||||
|
||||
func RoundProb(p float64) float64 {
|
||||
return math.Round(p*10000) / 10000
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
package ocr
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"image"
|
||||
"image/png"
|
||||
|
||||
"github.com/gen2brain/go-fitz"
|
||||
"github.com/otiai10/gosseract/v2"
|
||||
)
|
||||
|
||||
type Box struct {
|
||||
Text string
|
||||
X int
|
||||
Y int
|
||||
W int
|
||||
H int
|
||||
}
|
||||
|
||||
// RenderPage rasterizes the first page of a PDF.
|
||||
func RenderPage(pdfData []byte, dpi int) (image.Image, int, int, error) {
|
||||
doc, err := fitz.NewFromMemory(pdfData)
|
||||
if err != nil {
|
||||
return nil, 0, 0, fmt.Errorf("open pdf: %w", err)
|
||||
}
|
||||
defer doc.Close()
|
||||
|
||||
if doc.NumPage() == 0 {
|
||||
return nil, 0, 0, fmt.Errorf("pdf has no pages")
|
||||
}
|
||||
|
||||
img, err := doc.ImageDPI(0, float64(dpi))
|
||||
if err != nil {
|
||||
return nil, 0, 0, fmt.Errorf("render page: %w", err)
|
||||
}
|
||||
bounds := img.Bounds()
|
||||
return img, bounds.Dx(), bounds.Dy(), nil
|
||||
}
|
||||
|
||||
// OCRImage runs Tesseract word-level OCR on a rendered page image.
|
||||
func (e *Engine) OCRImage(img image.Image) ([]Box, error) {
|
||||
var buf bytes.Buffer
|
||||
if err := png.Encode(&buf, img); err != nil {
|
||||
return nil, fmt.Errorf("encode png: %w", err)
|
||||
}
|
||||
|
||||
client := e.Acquire()
|
||||
defer e.Release(client)
|
||||
|
||||
if err := client.SetImageFromBytes(buf.Bytes()); err != nil {
|
||||
return nil, fmt.Errorf("tesseract image: %w", err)
|
||||
}
|
||||
|
||||
boxes, err := client.GetBoundingBoxes(gosseract.RIL_WORD)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("tesseract ocr: %w", err)
|
||||
}
|
||||
|
||||
out := make([]Box, 0, len(boxes))
|
||||
for _, b := range boxes {
|
||||
text := trim(b.Word)
|
||||
if text == "" || b.Confidence <= 0 {
|
||||
continue
|
||||
}
|
||||
out = append(out, Box{
|
||||
Text: text,
|
||||
X: b.Box.Min.X,
|
||||
Y: b.Box.Min.Y,
|
||||
W: b.Box.Dx(),
|
||||
H: b.Box.Dy(),
|
||||
})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// ExtractBoxes renders the first PDF page and runs OCR with a pooled Tesseract client.
|
||||
func (e *Engine) ExtractBoxes(pdfData []byte, dpi int) ([]Box, int, int, error) {
|
||||
img, pageW, pageH, err := RenderPage(pdfData, dpi)
|
||||
if err != nil {
|
||||
return nil, 0, 0, err
|
||||
}
|
||||
boxes, err := e.OCRImage(img)
|
||||
if err != nil {
|
||||
return nil, 0, 0, err
|
||||
}
|
||||
return boxes, pageW, pageH, nil
|
||||
}
|
||||
|
||||
func trim(s string) string {
|
||||
start, end := 0, len(s)
|
||||
for start < end && (s[start] == ' ' || s[start] == '\t' || s[start] == '\n') {
|
||||
start++
|
||||
}
|
||||
for end > start && (s[end-1] == ' ' || s[end-1] == '\t' || s[end-1] == '\n') {
|
||||
end--
|
||||
}
|
||||
return s[start:end]
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
package ocr
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/otiai10/gosseract/v2"
|
||||
)
|
||||
|
||||
// Engine holds a pool of reusable Tesseract clients.
|
||||
type Engine struct {
|
||||
pool chan *gosseract.Client
|
||||
clients []*gosseract.Client
|
||||
lang string
|
||||
}
|
||||
|
||||
// NewEngine warms up poolSize Tesseract clients for the given language.
|
||||
func NewEngine(lang string, poolSize int) (*Engine, error) {
|
||||
if poolSize < 1 {
|
||||
poolSize = 1
|
||||
}
|
||||
|
||||
e := &Engine{
|
||||
pool: make(chan *gosseract.Client, poolSize),
|
||||
clients: make([]*gosseract.Client, 0, poolSize),
|
||||
lang: lang,
|
||||
}
|
||||
|
||||
for i := 0; i < poolSize; i++ {
|
||||
client, err := newPooledClient(lang)
|
||||
if err != nil {
|
||||
e.Close()
|
||||
return nil, fmt.Errorf("tesseract client %d: %w", i, err)
|
||||
}
|
||||
e.clients = append(e.clients, client)
|
||||
e.pool <- client
|
||||
}
|
||||
|
||||
return e, nil
|
||||
}
|
||||
|
||||
func newPooledClient(lang string) (*gosseract.Client, error) {
|
||||
client := gosseract.NewClient()
|
||||
if err := client.SetLanguage(lang); err != nil {
|
||||
if err2 := client.SetLanguage("eng"); err2 != nil {
|
||||
_ = client.Close()
|
||||
return nil, fmt.Errorf("tesseract lang: %w", err)
|
||||
}
|
||||
}
|
||||
return client, nil
|
||||
}
|
||||
|
||||
// Acquire returns a Tesseract client from the pool.
|
||||
func (e *Engine) Acquire() *gosseract.Client {
|
||||
return <-e.pool
|
||||
}
|
||||
|
||||
// Release returns a client to the pool.
|
||||
func (e *Engine) Release(client *gosseract.Client) {
|
||||
e.pool <- client
|
||||
}
|
||||
|
||||
// Close shuts down all pooled clients.
|
||||
func (e *Engine) Close() error {
|
||||
for _, client := range e.clients {
|
||||
_ = client.Close()
|
||||
}
|
||||
e.clients = nil
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"math"
|
||||
"sort"
|
||||
)
|
||||
|
||||
// ClusterRows runs 1D DBSCAN on Y coordinates (parity with sklearn DBSCAN).
|
||||
func ClusterRows(boxes []indexedBox, eps float64, minSamples int) {
|
||||
if len(boxes) == 0 {
|
||||
return
|
||||
}
|
||||
ys := make([]float64, len(boxes))
|
||||
for i, b := range boxes {
|
||||
ys[i] = float64(b.Y)
|
||||
}
|
||||
labels := dbscan1D(ys, eps, minSamples)
|
||||
for i := range boxes {
|
||||
boxes[i].Cluster = labels[i]
|
||||
}
|
||||
}
|
||||
|
||||
func dbscan1D(points []float64, eps float64, minSamples int) []int {
|
||||
n := len(points)
|
||||
labels := make([]int, n)
|
||||
for i := range labels {
|
||||
labels[i] = -1
|
||||
}
|
||||
clusterID := 0
|
||||
visited := make([]bool, n)
|
||||
|
||||
for i := 0; i < n; i++ {
|
||||
if visited[i] {
|
||||
continue
|
||||
}
|
||||
visited[i] = true
|
||||
neighbors := regionQuery(points, i, eps)
|
||||
if len(neighbors) < minSamples {
|
||||
continue
|
||||
}
|
||||
labels[i] = clusterID
|
||||
seed := append([]int(nil), neighbors...)
|
||||
for j := 0; j < len(seed); j++ {
|
||||
q := seed[j]
|
||||
if !visited[q] {
|
||||
visited[q] = true
|
||||
neighborsQ := regionQuery(points, q, eps)
|
||||
if len(neighborsQ) >= minSamples {
|
||||
seed = appendUnique(seed, neighborsQ)
|
||||
}
|
||||
}
|
||||
if labels[q] == -1 {
|
||||
labels[q] = clusterID
|
||||
}
|
||||
}
|
||||
clusterID++
|
||||
}
|
||||
return labels
|
||||
}
|
||||
|
||||
func regionQuery(points []float64, idx int, eps float64) []int {
|
||||
var neighbors []int
|
||||
for i, p := range points {
|
||||
if math.Abs(p-points[idx]) <= eps {
|
||||
neighbors = append(neighbors, i)
|
||||
}
|
||||
}
|
||||
return neighbors
|
||||
}
|
||||
|
||||
func appendUnique(seed, add []int) []int {
|
||||
inSeed := make(map[int]bool, len(seed))
|
||||
for _, v := range seed {
|
||||
inSeed[v] = true
|
||||
}
|
||||
for _, v := range add {
|
||||
if !inSeed[v] {
|
||||
seed = append(seed, v)
|
||||
inSeed[v] = true
|
||||
}
|
||||
}
|
||||
return seed
|
||||
}
|
||||
|
||||
type indexedBox struct {
|
||||
Text string
|
||||
X int
|
||||
Y int
|
||||
W int
|
||||
H int
|
||||
Cluster int
|
||||
}
|
||||
|
||||
func groupByCluster(boxes []indexedBox) map[int][]indexedBox {
|
||||
groups := make(map[int][]indexedBox)
|
||||
for _, b := range boxes {
|
||||
if b.Cluster < 0 {
|
||||
continue
|
||||
}
|
||||
groups[b.Cluster] = append(groups[b.Cluster], b)
|
||||
}
|
||||
return groups
|
||||
}
|
||||
|
||||
func sortClusters(groups map[int][]indexedBox) []int {
|
||||
keys := make([]int, 0, len(groups))
|
||||
for k := range groups {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Ints(keys)
|
||||
return keys
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
var (
|
||||
codePattern = regexp.MustCompile(`^\d{4,6}$`)
|
||||
currencyPattern = regexp.MustCompile(`^\d{1,3}(,\d{3})+(\.\d+)?$`)
|
||||
)
|
||||
|
||||
type rowFeatures struct {
|
||||
Features [8]float32
|
||||
TextoFila string
|
||||
Cluster int
|
||||
}
|
||||
|
||||
func extractRowFeatures(group []indexedBox, pageW, pageH int) rowFeatures {
|
||||
sort.Slice(group, func(i, j int) bool { return group[i].X < group[j].X })
|
||||
|
||||
texts := make([]string, len(group))
|
||||
for i, b := range group {
|
||||
texts[i] = b.Text
|
||||
}
|
||||
textoFila := strings.Join(texts, " ")
|
||||
firstToken := ""
|
||||
if len(texts) > 0 {
|
||||
firstToken = texts[0]
|
||||
}
|
||||
|
||||
currencyCount := 0
|
||||
integerCount := 0
|
||||
for _, t := range texts {
|
||||
if currencyPattern.MatchString(t) {
|
||||
currencyCount++
|
||||
}
|
||||
if isDigits(t) {
|
||||
integerCount++
|
||||
}
|
||||
}
|
||||
|
||||
xMin, xMax := group[0].X, group[0].X
|
||||
ySum := 0
|
||||
for _, b := range group {
|
||||
if b.X < xMin {
|
||||
xMin = b.X
|
||||
}
|
||||
if b.X > xMax {
|
||||
xMax = b.X
|
||||
}
|
||||
ySum += b.Y
|
||||
}
|
||||
|
||||
xSpanNorm := 0.0
|
||||
if pageW > 0 {
|
||||
xSpanNorm = float64(xMax-xMin) / float64(pageW)
|
||||
}
|
||||
yNorm := 0.0
|
||||
if pageH > 0 {
|
||||
yNorm = float64(ySum/len(group)) / float64(pageH)
|
||||
}
|
||||
|
||||
totalChars := len(textoFila)
|
||||
if totalChars == 0 {
|
||||
totalChars = 1
|
||||
}
|
||||
special := 0
|
||||
digits := 0
|
||||
for _, c := range textoFila {
|
||||
if !unicode.IsLetter(c) && !unicode.IsNumber(c) && !unicode.IsSpace(c) {
|
||||
special++
|
||||
}
|
||||
if unicode.IsDigit(c) {
|
||||
digits++
|
||||
}
|
||||
}
|
||||
|
||||
firstIsCode := 0
|
||||
if codePattern.MatchString(firstToken) {
|
||||
firstIsCode = 1
|
||||
}
|
||||
|
||||
return rowFeatures{
|
||||
Features: [8]float32{
|
||||
float32(firstIsCode),
|
||||
float32(currencyCount),
|
||||
float32(integerCount),
|
||||
float32(xSpanNorm),
|
||||
float32(yNorm),
|
||||
float32(len(texts)),
|
||||
float32(special) / float32(totalChars),
|
||||
float32(digits) / float32(totalChars),
|
||||
},
|
||||
TextoFila: textoFila,
|
||||
Cluster: group[0].Cluster,
|
||||
}
|
||||
}
|
||||
|
||||
func isDigits(s string) bool {
|
||||
if s == "" {
|
||||
return false
|
||||
}
|
||||
for _, c := range s {
|
||||
if c < '0' || c > '9' {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func buildFeatureRows(boxes []indexedBox, pageW, pageH int) []rowFeatures {
|
||||
groups := groupByCluster(boxes)
|
||||
order := sortClusters(groups)
|
||||
rows := make([]rowFeatures, 0, len(groups))
|
||||
for _, cid := range order {
|
||||
rows = append(rows, extractRowFeatures(groups[cid], pageW, pageH))
|
||||
}
|
||||
return rows
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"github.com/pys/pdf-count/internal/domain"
|
||||
"github.com/pys/pdf-count/internal/model"
|
||||
"github.com/pys/pdf-count/internal/ocr"
|
||||
)
|
||||
|
||||
type Options struct {
|
||||
DPI int
|
||||
Lang string
|
||||
DBSCANEps float64
|
||||
DBSCANMinSamples int
|
||||
IncludeRejected bool
|
||||
LogPipelineTiming bool
|
||||
}
|
||||
|
||||
type Processor struct {
|
||||
predictor *model.Predictor
|
||||
ocrEngine *ocr.Engine
|
||||
opts Options
|
||||
}
|
||||
|
||||
func NewProcessor(predictor *model.Predictor, ocrEngine *ocr.Engine, opts Options) *Processor {
|
||||
return &Processor{predictor: predictor, ocrEngine: ocrEngine, opts: opts}
|
||||
}
|
||||
|
||||
func (p *Processor) ProcessPDF(filename string, pdfData []byte) domain.PDFResult {
|
||||
start := time.Now()
|
||||
result := domain.PDFResult{Filename: filename, Status: domain.StatusOK}
|
||||
|
||||
tRender := time.Now()
|
||||
img, pageW, pageH, err := ocr.RenderPage(pdfData, p.opts.DPI)
|
||||
renderDur := time.Since(tRender)
|
||||
if err != nil {
|
||||
result.Status = domain.StatusError
|
||||
result.Error = err.Error()
|
||||
return result
|
||||
}
|
||||
|
||||
tOCR := time.Now()
|
||||
boxes, err := p.ocrEngine.OCRImage(img)
|
||||
ocrDur := time.Since(tOCR)
|
||||
if err != nil {
|
||||
result.Status = domain.StatusError
|
||||
result.Error = err.Error()
|
||||
return result
|
||||
}
|
||||
if len(boxes) == 0 {
|
||||
result.ServiceCount = 0
|
||||
result.Services = []domain.ServiceRow{}
|
||||
p.logTiming(filename, renderDur, ocrDur, 0, 0, time.Since(start))
|
||||
return result
|
||||
}
|
||||
|
||||
tCluster := time.Now()
|
||||
indexed := make([]indexedBox, len(boxes))
|
||||
for i, b := range boxes {
|
||||
indexed[i] = indexedBox{Text: b.Text, X: b.X, Y: b.Y, W: b.W, H: b.H}
|
||||
}
|
||||
ClusterRows(indexed, p.opts.DBSCANEps, p.opts.DBSCANMinSamples)
|
||||
|
||||
rows := buildFeatureRows(indexed, pageW, pageH)
|
||||
clusterDur := time.Since(tCluster)
|
||||
if len(rows) == 0 {
|
||||
result.ServiceCount = 0
|
||||
result.Services = []domain.ServiceRow{}
|
||||
p.logTiming(filename, renderDur, ocrDur, clusterDur, 0, time.Since(start))
|
||||
return result
|
||||
}
|
||||
|
||||
features := make([][8]float32, len(rows))
|
||||
for i, r := range rows {
|
||||
features[i] = r.Features
|
||||
}
|
||||
|
||||
tInfer := time.Now()
|
||||
probs, err := p.predictor.PredictProba(features)
|
||||
inferDur := time.Since(tInfer)
|
||||
if err != nil {
|
||||
result.Status = domain.StatusError
|
||||
result.Error = fmt.Sprintf("inference: %v", err)
|
||||
return result
|
||||
}
|
||||
|
||||
threshold := p.predictor.Threshold()
|
||||
services := make([]domain.ServiceRow, 0)
|
||||
serviceCount := 0
|
||||
for i, row := range rows {
|
||||
prob := model.RoundProb(probs[i])
|
||||
isService := prob >= threshold
|
||||
if isService {
|
||||
serviceCount++
|
||||
}
|
||||
if p.opts.IncludeRejected {
|
||||
v := isService
|
||||
services = append(services, domain.ServiceRow{
|
||||
Text: row.TextoFila,
|
||||
Probability: prob,
|
||||
IsService: &v,
|
||||
})
|
||||
} else if isService {
|
||||
services = append(services, domain.ServiceRow{
|
||||
Text: row.TextoFila,
|
||||
Probability: prob,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
result.Services = services
|
||||
result.ServiceCount = serviceCount
|
||||
p.logTiming(filename, renderDur, ocrDur, clusterDur, inferDur, time.Since(start))
|
||||
return result
|
||||
}
|
||||
|
||||
func (p *Processor) logTiming(filename string, render, ocr, cluster, infer, total time.Duration) {
|
||||
if !p.opts.LogPipelineTiming {
|
||||
return
|
||||
}
|
||||
log.Printf(
|
||||
"pipeline filename=%s ocr_render=%s ocr_tesseract=%s cluster=%s infer=%s total=%s",
|
||||
filename, render, ocr, cluster, infer, total,
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
package queue
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
"github.com/pys/pdf-count/internal/domain"
|
||||
)
|
||||
|
||||
type Publisher struct {
|
||||
js jetstream.JetStream
|
||||
stream string
|
||||
subject string
|
||||
}
|
||||
|
||||
func NewPublisher(natsURL, streamName, subjectPrefix string) (*Publisher, error) {
|
||||
nc, err := nats.Connect(natsURL)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("nats connect: %w", err)
|
||||
}
|
||||
js, err := jetstream.New(nc)
|
||||
if err != nil {
|
||||
nc.Close()
|
||||
return nil, err
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
|
||||
_, err = js.CreateOrUpdateStream(ctx, jetstream.StreamConfig{
|
||||
Name: streamName,
|
||||
Subjects: []string{subjectPrefix + ".>"},
|
||||
Retention: jetstream.WorkQueuePolicy,
|
||||
MaxMsgs: 200000,
|
||||
})
|
||||
if err != nil {
|
||||
nc.Close()
|
||||
return nil, fmt.Errorf("create stream: %w", err)
|
||||
}
|
||||
|
||||
return &Publisher{js: js, stream: streamName, subject: subjectPrefix}, nil
|
||||
}
|
||||
|
||||
func (p *Publisher) Publish(ctx context.Context, msg domain.QueueMessage) error {
|
||||
data, err := json.Marshal(msg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
subject := fmt.Sprintf("%s.%s", p.subject, msg.JobID)
|
||||
_, err = p.js.Publish(ctx, subject, data)
|
||||
return err
|
||||
}
|
||||
|
||||
type Consumer struct {
|
||||
js jetstream.JetStream
|
||||
subject string
|
||||
consumerName string
|
||||
}
|
||||
|
||||
func NewConsumer(natsURL, streamName, subjectPrefix, consumerName string) (*Consumer, jetstream.Consumer, *nats.Conn, error) {
|
||||
nc, err := nats.Connect(natsURL)
|
||||
if err != nil {
|
||||
return nil, nil, nil, err
|
||||
}
|
||||
js, err := jetstream.New(nc)
|
||||
if err != nil {
|
||||
nc.Close()
|
||||
return nil, nil, nil, err
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
|
||||
_, err = js.CreateOrUpdateStream(ctx, jetstream.StreamConfig{
|
||||
Name: streamName,
|
||||
Subjects: []string{subjectPrefix + ".>"},
|
||||
Retention: jetstream.WorkQueuePolicy,
|
||||
MaxMsgs: 200000,
|
||||
})
|
||||
if err != nil {
|
||||
nc.Close()
|
||||
return nil, nil, nil, err
|
||||
}
|
||||
|
||||
cons, err := js.CreateOrUpdateConsumer(ctx, streamName, jetstream.ConsumerConfig{
|
||||
Name: consumerName,
|
||||
FilterSubject: subjectPrefix + ".>",
|
||||
AckPolicy: jetstream.AckExplicitPolicy,
|
||||
MaxDeliver: 3,
|
||||
AckWait: 2 * time.Minute,
|
||||
MaxAckPending: 256,
|
||||
})
|
||||
if err != nil {
|
||||
nc.Close()
|
||||
return nil, nil, nil, err
|
||||
}
|
||||
|
||||
return &Consumer{js: js, subject: subjectPrefix, consumerName: consumerName}, cons, nc, nil
|
||||
}
|
||||
|
||||
func DecodeMessage(data []byte) (domain.QueueMessage, error) {
|
||||
var msg domain.QueueMessage
|
||||
err := json.Unmarshal(data, &msg)
|
||||
return msg, err
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
|
||||
"github.com/minio/minio-go/v7"
|
||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||
)
|
||||
|
||||
type Client struct {
|
||||
client *minio.Client
|
||||
bucket string
|
||||
}
|
||||
|
||||
func New(endpoint, accessKey, secretKey, bucket string, useSSL bool) (*Client, error) {
|
||||
client, err := minio.New(endpoint, &minio.Options{
|
||||
Creds: credentials.NewStaticV4(accessKey, secretKey, ""),
|
||||
Secure: useSSL,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Client{client: client, bucket: bucket}, nil
|
||||
}
|
||||
|
||||
func (c *Client) EnsureBucket(ctx context.Context) error {
|
||||
exists, err := c.client.BucketExists(ctx, c.bucket)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if exists {
|
||||
return nil
|
||||
}
|
||||
err = c.client.MakeBucket(ctx, c.bucket, minio.MakeBucketOptions{})
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
resp := minio.ToErrorResponse(err)
|
||||
if resp.Code == "BucketAlreadyOwnedByYou" || resp.Code == "BucketAlreadyExists" {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func (c *Client) Put(ctx context.Context, objectKey string, data []byte, contentType string) error {
|
||||
if contentType == "" {
|
||||
contentType = "application/pdf"
|
||||
}
|
||||
_, err := c.client.PutObject(ctx, c.bucket, objectKey, bytes.NewReader(data), int64(len(data)), minio.PutObjectOptions{
|
||||
ContentType: contentType,
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
func (c *Client) Get(ctx context.Context, objectKey string) ([]byte, error) {
|
||||
obj, err := c.client.GetObject(ctx, c.bucket, objectKey, minio.GetObjectOptions{})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer obj.Close()
|
||||
return io.ReadAll(obj)
|
||||
}
|
||||
|
||||
func (c *Client) ListByPrefix(ctx context.Context, prefix string) ([]string, error) {
|
||||
var keys []string
|
||||
for obj := range c.client.ListObjects(ctx, c.bucket, minio.ListObjectsOptions{
|
||||
Prefix: prefix,
|
||||
Recursive: true,
|
||||
}) {
|
||||
if obj.Err != nil {
|
||||
return nil, obj.Err
|
||||
}
|
||||
if strings.HasSuffix(strings.ToLower(obj.Key), ".pdf") {
|
||||
keys = append(keys, obj.Key)
|
||||
}
|
||||
}
|
||||
return keys, nil
|
||||
}
|
||||
|
||||
func JobObjectKey(jobID, filename string) string {
|
||||
return fmt.Sprintf("jobs/%s/%s", jobID, filename)
|
||||
}
|
||||
@@ -0,0 +1,276 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/pys/pdf-count/internal/domain"
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
type Store struct {
|
||||
client *redis.Client
|
||||
ttl time.Duration
|
||||
}
|
||||
|
||||
func New(redisURL string, ttl time.Duration) (*Store, error) {
|
||||
opts, err := redis.ParseURL(redisURL)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
client := redis.NewClient(opts)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
if err := client.Ping(ctx).Err(); err != nil {
|
||||
return nil, fmt.Errorf("redis ping: %w", err)
|
||||
}
|
||||
return &Store{client: client, ttl: ttl}, nil
|
||||
}
|
||||
|
||||
func (s *Store) Close() error {
|
||||
return s.client.Close()
|
||||
}
|
||||
|
||||
func metaKey(jobID string) string { return fmt.Sprintf("job:%s:meta", jobID) }
|
||||
func filesKey(jobID string) string { return fmt.Sprintf("job:%s:files", jobID) }
|
||||
func resultsKey(jobID string) string {
|
||||
return fmt.Sprintf("job:%s:results", jobID)
|
||||
}
|
||||
|
||||
func (s *Store) CreateJob(ctx context.Context, jobID string, total int, includeRejected bool) error {
|
||||
pipe := s.client.Pipeline()
|
||||
pipe.HSet(ctx, metaKey(jobID), map[string]interface{}{
|
||||
"status": domain.JobQueued,
|
||||
"total": total,
|
||||
"processed": 0,
|
||||
"failed": 0,
|
||||
"include_rejected": boolStr(includeRejected),
|
||||
})
|
||||
pipe.Expire(ctx, metaKey(jobID), s.ttl)
|
||||
_, err := pipe.Exec(ctx)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) AddJobFiles(ctx context.Context, jobID string, filenames []string) error {
|
||||
if len(filenames) == 0 {
|
||||
return nil
|
||||
}
|
||||
members := make([]interface{}, len(filenames))
|
||||
for i, f := range filenames {
|
||||
members[i] = f
|
||||
}
|
||||
pipe := s.client.Pipeline()
|
||||
pipe.RPush(ctx, filesKey(jobID), members...)
|
||||
pipe.Expire(ctx, filesKey(jobID), s.ttl)
|
||||
_, err := pipe.Exec(ctx)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) MarkRunning(ctx context.Context, jobID string) error {
|
||||
return s.client.HSet(ctx, metaKey(jobID), "status", domain.JobRunning).Err()
|
||||
}
|
||||
|
||||
var saveResultScript = redis.NewScript(`
|
||||
local meta = KEYS[1]
|
||||
local results = KEYS[2]
|
||||
redis.call('HSET', results, ARGV[1], ARGV[2])
|
||||
local processed = redis.call('HINCRBY', meta, 'processed', 1)
|
||||
if ARGV[3] == '1' then
|
||||
redis.call('HINCRBY', meta, 'failed', 1)
|
||||
end
|
||||
redis.call('EXPIRE', results, ARGV[4])
|
||||
local total = tonumber(redis.call('HGET', meta, 'total') or '0')
|
||||
local status = redis.call('HGET', meta, 'status')
|
||||
if status ~= 'cancelled' and status ~= 'completed' and total > 0 and processed >= total then
|
||||
redis.call('HSET', meta, 'status', 'completed')
|
||||
end
|
||||
return processed
|
||||
`)
|
||||
|
||||
func (s *Store) SaveResult(ctx context.Context, jobID string, result domain.PDFResult) error {
|
||||
data, err := json.Marshal(result)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
failFlag := "0"
|
||||
if result.Status == domain.StatusError {
|
||||
failFlag = "1"
|
||||
}
|
||||
ttlSec := int64(s.ttl.Seconds())
|
||||
if ttlSec <= 0 {
|
||||
ttlSec = 72 * 3600
|
||||
}
|
||||
_, err = saveResultScript.Run(ctx, s.client,
|
||||
[]string{metaKey(jobID), resultsKey(jobID)},
|
||||
result.Filename, string(data), failFlag, ttlSec,
|
||||
).Result()
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) TryCompleteJob(ctx context.Context, jobID string) error {
|
||||
meta, err := s.client.HGetAll(ctx, metaKey(jobID)).Result()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = s.maybeCompleteJob(ctx, jobID, meta)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) maybeCompleteJob(ctx context.Context, jobID string, meta map[string]string) (string, error) {
|
||||
status := meta["status"]
|
||||
if status == domain.JobCancelled || status == domain.JobCompleted {
|
||||
return status, nil
|
||||
}
|
||||
total, _ := strconv.Atoi(meta["total"])
|
||||
processed, _ := strconv.Atoi(meta["processed"])
|
||||
if processed >= total && total > 0 {
|
||||
if err := s.client.HSet(ctx, metaKey(jobID), "status", domain.JobCompleted).Err(); err != nil {
|
||||
return status, err
|
||||
}
|
||||
return domain.JobCompleted, nil
|
||||
}
|
||||
return status, nil
|
||||
}
|
||||
|
||||
func (s *Store) GetJob(ctx context.Context, jobID string) (domain.JobStatus, error) {
|
||||
meta, err := s.client.HGetAll(ctx, metaKey(jobID)).Result()
|
||||
if err != nil {
|
||||
return domain.JobStatus{}, err
|
||||
}
|
||||
if len(meta) == 0 {
|
||||
return domain.JobStatus{}, fmt.Errorf("job not found")
|
||||
}
|
||||
status, err := s.maybeCompleteJob(ctx, jobID, meta)
|
||||
if err != nil {
|
||||
return domain.JobStatus{}, err
|
||||
}
|
||||
if status == domain.JobCompleted {
|
||||
meta["status"] = domain.JobCompleted
|
||||
}
|
||||
total, _ := strconv.Atoi(meta["total"])
|
||||
processed, _ := strconv.Atoi(meta["processed"])
|
||||
failed, _ := strconv.Atoi(meta["failed"])
|
||||
return domain.JobStatus{
|
||||
JobID: jobID,
|
||||
Status: status,
|
||||
Total: total,
|
||||
Processed: processed,
|
||||
Failed: failed,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (s *Store) GetResultsPage(ctx context.Context, jobID, cursor string, limit int) (domain.JobResultsPage, error) {
|
||||
if limit <= 0 || limit > 1000 {
|
||||
limit = 500
|
||||
}
|
||||
allFiles, err := s.client.LRange(ctx, filesKey(jobID), 0, -1).Result()
|
||||
if err != nil {
|
||||
return domain.JobResultsPage{}, err
|
||||
}
|
||||
|
||||
start := 0
|
||||
if cursor != "" {
|
||||
for i, f := range allFiles {
|
||||
if f == cursor {
|
||||
start = i + 1
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
end := start + limit
|
||||
if end > len(allFiles) {
|
||||
end = len(allFiles)
|
||||
}
|
||||
pageFiles := allFiles[start:end]
|
||||
|
||||
results := make([]domain.PDFResult, 0, len(pageFiles))
|
||||
if len(pageFiles) > 0 {
|
||||
raw, err := s.client.HMGet(ctx, resultsKey(jobID), pageFiles...).Result()
|
||||
if err != nil {
|
||||
return domain.JobResultsPage{}, err
|
||||
}
|
||||
for i, v := range raw {
|
||||
if v == nil {
|
||||
results = append(results, domain.PDFResult{
|
||||
Filename: pageFiles[i],
|
||||
Status: "pending",
|
||||
})
|
||||
continue
|
||||
}
|
||||
var r domain.PDFResult
|
||||
if err := json.Unmarshal([]byte(v.(string)), &r); err != nil {
|
||||
continue
|
||||
}
|
||||
results = append(results, r)
|
||||
}
|
||||
}
|
||||
|
||||
page := domain.JobResultsPage{
|
||||
JobID: jobID,
|
||||
Results: results,
|
||||
HasMore: end < len(allFiles),
|
||||
}
|
||||
if page.HasMore && len(pageFiles) > 0 {
|
||||
page.NextCursor = pageFiles[len(pageFiles)-1]
|
||||
}
|
||||
return page, nil
|
||||
}
|
||||
|
||||
func (s *Store) CancelJob(ctx context.Context, jobID string) error {
|
||||
return s.client.HSet(ctx, metaKey(jobID), "status", domain.JobCancelled).Err()
|
||||
}
|
||||
|
||||
func (s *Store) IsCancelled(ctx context.Context, jobID string) (bool, error) {
|
||||
status, err := s.client.HGet(ctx, metaKey(jobID), "status").Result()
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return status == domain.JobCancelled, nil
|
||||
}
|
||||
|
||||
func (s *Store) PendingCount(ctx context.Context) (int64, error) {
|
||||
// Approximate: sum of (total - processed) for running jobs
|
||||
var cursor uint64
|
||||
var pending int64
|
||||
for {
|
||||
keys, next, err := s.client.Scan(ctx, cursor, "job:*:meta", 100).Result()
|
||||
if err != nil {
|
||||
return pending, err
|
||||
}
|
||||
for _, key := range keys {
|
||||
meta, err := s.client.HGetAll(ctx, key).Result()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if meta["status"] != domain.JobQueued && meta["status"] != domain.JobRunning {
|
||||
continue
|
||||
}
|
||||
total, _ := strconv.Atoi(meta["total"])
|
||||
processed, _ := strconv.Atoi(meta["processed"])
|
||||
pending += int64(total - processed)
|
||||
}
|
||||
cursor = next
|
||||
if cursor == 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
return pending, nil
|
||||
}
|
||||
|
||||
func (s *Store) IncrementTotal(ctx context.Context, jobID string, delta int) error {
|
||||
pipe := s.client.Pipeline()
|
||||
pipe.HIncrBy(ctx, metaKey(jobID), "total", int64(delta))
|
||||
pipe.Expire(ctx, metaKey(jobID), s.ttl)
|
||||
_, err := pipe.Exec(ctx)
|
||||
return err
|
||||
}
|
||||
|
||||
func boolStr(v bool) string {
|
||||
if v {
|
||||
return "true"
|
||||
}
|
||||
return "false"
|
||||
}
|
||||
@@ -0,0 +1,212 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"os/signal"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
"github.com/pys/pdf-count/internal/config"
|
||||
"github.com/pys/pdf-count/internal/domain"
|
||||
"github.com/pys/pdf-count/internal/model"
|
||||
"github.com/pys/pdf-count/internal/ocr"
|
||||
"github.com/pys/pdf-count/internal/pipeline"
|
||||
"github.com/pys/pdf-count/internal/queue"
|
||||
"github.com/pys/pdf-count/internal/storage"
|
||||
"github.com/pys/pdf-count/internal/store"
|
||||
)
|
||||
|
||||
type Runner struct {
|
||||
cfg config.Config
|
||||
predictor *model.Predictor
|
||||
ocrEngine *ocr.Engine
|
||||
store *store.Store
|
||||
storage *storage.Client
|
||||
consumer jetstream.Consumer
|
||||
ncClose func()
|
||||
}
|
||||
|
||||
func NewRunner(cfg config.Config) (*Runner, error) {
|
||||
predictor, err := model.NewPredictor(cfg.ModelPath, cfg.InferencePath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
poolSize := cfg.OCRPoolSize
|
||||
if poolSize <= 0 {
|
||||
poolSize = cfg.WorkerConcurrency
|
||||
}
|
||||
ocrEngine, err := ocr.NewEngine(cfg.OCRLang, poolSize)
|
||||
if err != nil {
|
||||
predictor.Close()
|
||||
return nil, err
|
||||
}
|
||||
st, err := store.New(cfg.RedisURL, cfg.JobResultTTL)
|
||||
if err != nil {
|
||||
ocrEngine.Close()
|
||||
predictor.Close()
|
||||
return nil, err
|
||||
}
|
||||
stg, err := storage.New(cfg.MinIOEndpoint, cfg.MinIOAccessKey, cfg.MinIOSecretKey, cfg.MinIOBucket, cfg.MinIOUseSSL)
|
||||
if err != nil {
|
||||
ocrEngine.Close()
|
||||
st.Close()
|
||||
predictor.Close()
|
||||
return nil, err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
if err := stg.EnsureBucket(ctx); err != nil {
|
||||
ocrEngine.Close()
|
||||
st.Close()
|
||||
predictor.Close()
|
||||
return nil, err
|
||||
}
|
||||
|
||||
_, cons, nc, err := queue.NewConsumer(cfg.NATSURL, cfg.StreamName, cfg.SubjectPrefix, "pdf-workers")
|
||||
if err != nil {
|
||||
ocrEngine.Close()
|
||||
st.Close()
|
||||
predictor.Close()
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &Runner{
|
||||
cfg: cfg,
|
||||
predictor: predictor,
|
||||
ocrEngine: ocrEngine,
|
||||
store: st,
|
||||
storage: stg,
|
||||
consumer: cons,
|
||||
ncClose: func() { nc.Close() },
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (r *Runner) Close() {
|
||||
r.ncClose()
|
||||
_ = r.ocrEngine.Close()
|
||||
_ = r.store.Close()
|
||||
_ = r.predictor.Close()
|
||||
}
|
||||
|
||||
func (r *Runner) processSlots() int {
|
||||
c := r.cfg.WorkerConcurrency
|
||||
pool := r.cfg.OCRPoolSize
|
||||
if pool <= 0 {
|
||||
pool = r.cfg.WorkerConcurrency
|
||||
}
|
||||
if pool <= 0 {
|
||||
pool = 4
|
||||
}
|
||||
if c > pool {
|
||||
c = pool
|
||||
}
|
||||
if c <= 0 {
|
||||
c = 4
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func (r *Runner) Run(ctx context.Context) error {
|
||||
slots := r.processSlots()
|
||||
sem := make(chan struct{}, slots)
|
||||
var wg sync.WaitGroup
|
||||
|
||||
go func() {
|
||||
<-ctx.Done()
|
||||
wg.Wait()
|
||||
}()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil
|
||||
default:
|
||||
}
|
||||
|
||||
msgs, err := r.consumer.Fetch(slots, jetstream.FetchMaxWait(r.cfg.NATSFetchMaxWait))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for msg := range msgs.Messages() {
|
||||
sem <- struct{}{}
|
||||
wg.Add(1)
|
||||
m := msg
|
||||
go func() {
|
||||
defer func() {
|
||||
<-sem
|
||||
wg.Done()
|
||||
}()
|
||||
r.handleMessage(ctx, m)
|
||||
}()
|
||||
}
|
||||
_ = msgs.Error()
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Runner) handleMessage(ctx context.Context, msg jetstream.Msg) {
|
||||
processCtx, cancel := context.WithTimeout(ctx, r.cfg.PDFProcessTimeout)
|
||||
defer cancel()
|
||||
|
||||
qmsg, err := queue.DecodeMessage(msg.Data())
|
||||
if err != nil {
|
||||
_ = msg.Term()
|
||||
return
|
||||
}
|
||||
|
||||
cancelled, err := r.store.IsCancelled(processCtx, qmsg.JobID)
|
||||
if err != nil || cancelled {
|
||||
_ = msg.Ack()
|
||||
return
|
||||
}
|
||||
|
||||
data, err := r.storage.Get(processCtx, qmsg.ObjectKey)
|
||||
if err != nil {
|
||||
_ = r.store.SaveResult(processCtx, qmsg.JobID, domain.PDFResult{
|
||||
Filename: qmsg.Filename,
|
||||
Status: domain.StatusError,
|
||||
Error: err.Error(),
|
||||
})
|
||||
_ = r.store.TryCompleteJob(processCtx, qmsg.JobID)
|
||||
_ = msg.Ack()
|
||||
return
|
||||
}
|
||||
|
||||
proc := pipeline.NewProcessor(r.predictor, r.ocrEngine, pipeline.Options{
|
||||
DPI: r.cfg.OCRDPI,
|
||||
Lang: r.cfg.OCRLang,
|
||||
DBSCANEps: r.cfg.DBSCANEps,
|
||||
DBSCANMinSamples: r.cfg.DBSCANMinSamples,
|
||||
IncludeRejected: qmsg.IncludeRejected,
|
||||
LogPipelineTiming: r.cfg.LogPipelineTiming,
|
||||
})
|
||||
result := proc.ProcessPDF(qmsg.Filename, data)
|
||||
data = nil
|
||||
|
||||
if err := r.store.SaveResult(processCtx, qmsg.JobID, result); err != nil {
|
||||
_ = msg.NakWithDelay(5 * time.Second)
|
||||
return
|
||||
}
|
||||
_ = r.store.TryCompleteJob(processCtx, qmsg.JobID)
|
||||
_ = msg.Ack()
|
||||
}
|
||||
|
||||
func RunMain() {
|
||||
cfg := config.Load()
|
||||
runner, err := NewRunner(cfg)
|
||||
if err != nil {
|
||||
log.Fatalf("worker init: %v", err)
|
||||
}
|
||||
defer runner.Close()
|
||||
|
||||
ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer cancel()
|
||||
|
||||
log.Printf("worker started slots=%d", runner.processSlots())
|
||||
if err := runner.Run(ctx); err != nil {
|
||||
log.Fatalf("worker: %v", err)
|
||||
}
|
||||
log.Println("worker stopped")
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
joblib
|
||||
scikit-learn
|
||||
skl2onnx
|
||||
onnx
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Compara baseline Python vs respuesta del microservicio Go."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SC_ROOT = Path(os.environ.get("SERVICES_COUNT_ROOT", ROOT.parent / "services-count"))
|
||||
BASELINE = ROOT / "data" / "parity_baseline.json"
|
||||
API_URL = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:8080/v1/count"
|
||||
|
||||
|
||||
def call_api(pdf_path: Path) -> dict:
|
||||
result = subprocess.run(
|
||||
["curl", "-s", "-F", f"files=@{pdf_path}", API_URL],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
payload = json.loads(result.stdout)
|
||||
for item in payload.get("results", []):
|
||||
if item.get("filename") == pdf_path.name:
|
||||
return item
|
||||
raise RuntimeError(f"sin resultado para {pdf_path.name}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
if not BASELINE.exists():
|
||||
raise SystemExit("Ejecuta primero: python scripts/parity_baseline.py")
|
||||
|
||||
import yaml
|
||||
|
||||
config_path = SC_ROOT / "config" / "default.yaml"
|
||||
pdfs_dir = SC_ROOT / yaml.safe_load(config_path.read_text())["paths"]["pdfs_dir"]
|
||||
|
||||
baseline = json.loads(BASELINE.read_text(encoding="utf-8"))
|
||||
mismatches = 0
|
||||
for expected in baseline:
|
||||
pdf = pdfs_dir / expected["filename"]
|
||||
if not pdf.exists():
|
||||
print(f"SKIP {expected['filename']} (no existe)")
|
||||
continue
|
||||
got = call_api(pdf)
|
||||
if got.get("service_count") != expected.get("service_count"):
|
||||
mismatches += 1
|
||||
print(f"MISMATCH {expected['filename']}: py={expected['service_count']} go={got.get('service_count')}")
|
||||
else:
|
||||
print(f"OK {expected['filename']}: count={got.get('service_count')}")
|
||||
|
||||
if mismatches:
|
||||
raise SystemExit(f"{mismatches} diferencias encontradas")
|
||||
print("Paridad OK")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,99 @@
|
||||
"""Exporta model.joblib a ONNX + inference.json (sin depender de services-count)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import joblib
|
||||
from skl2onnx import convert_sklearn
|
||||
from skl2onnx.common.data_types import FloatTensorType
|
||||
|
||||
FEATURE_COLUMNS = [
|
||||
"first_token_is_code",
|
||||
"currency_count",
|
||||
"integer_count",
|
||||
"x_span_norm",
|
||||
"y_norm",
|
||||
"token_count",
|
||||
"special_char_ratio",
|
||||
"digit_ratio",
|
||||
]
|
||||
|
||||
DEFAULT_OCR = {"dpi": 300, "lang": "spa"}
|
||||
DEFAULT_DBSCAN = {"eps": 10, "min_samples": 1}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
parser = argparse.ArgumentParser(description="Exportar Random Forest joblib a ONNX")
|
||||
parser.add_argument(
|
||||
"--joblib",
|
||||
type=Path,
|
||||
required=True,
|
||||
help="Ruta al model.joblib (ej. ../services-count/models/latest/model.joblib)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--out",
|
||||
type=Path,
|
||||
default=root / "models" / "latest",
|
||||
help="Directorio de salida para model.onnx e inference.json",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--metadata",
|
||||
type=Path,
|
||||
default=None,
|
||||
help="model_metadata.json opcional para pipeline_version y params OCR/DBSCAN",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def load_metadata(path: Path | None) -> dict:
|
||||
if path is None or not path.exists():
|
||||
return {}
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
if not args.joblib.exists():
|
||||
raise SystemExit(f"No se encontró: {args.joblib}")
|
||||
|
||||
bundle = joblib.load(args.joblib)
|
||||
model = bundle["model"]
|
||||
threshold = float(bundle.get("threshold", 0.5))
|
||||
meta = load_metadata(args.metadata)
|
||||
|
||||
initial_type = [("float_input", FloatTensorType([None, len(FEATURE_COLUMNS)]))]
|
||||
options = {id(model): {"zipmap": False}}
|
||||
onnx_model = convert_sklearn(
|
||||
model,
|
||||
initial_types=initial_type,
|
||||
options=options,
|
||||
target_opset=12,
|
||||
)
|
||||
|
||||
out_dir = args.out
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
onnx_path = out_dir / "model.onnx"
|
||||
meta_path = out_dir / "inference.json"
|
||||
|
||||
with open(onnx_path, "wb") as f:
|
||||
f.write(onnx_model.SerializeToString())
|
||||
|
||||
inference = {
|
||||
"threshold": threshold,
|
||||
"feature_columns": FEATURE_COLUMNS,
|
||||
"pipeline_version": meta.get("pipeline_version", "unknown"),
|
||||
"ocr": meta.get("ocr", DEFAULT_OCR),
|
||||
"dbscan": meta.get("dbscan", DEFAULT_DBSCAN),
|
||||
}
|
||||
meta_path.write_text(json.dumps(inference, indent=2), encoding="utf-8")
|
||||
|
||||
print(f"ONNX: {onnx_path}")
|
||||
print(f"Meta: {meta_path}")
|
||||
print(f"Umbral: {threshold}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Genera baseline Python desde services-count para comparar con el servicio Go."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SC_ROOT = Path(os.environ.get("SERVICES_COUNT_ROOT", ROOT.parent / "services-count"))
|
||||
sys.path.insert(0, str(SC_ROOT / "src"))
|
||||
|
||||
from services_count.config import load_config
|
||||
from services_count.inference import predict_pdf
|
||||
|
||||
|
||||
def python_result(pdf_path: Path) -> dict:
|
||||
config = load_config(SC_ROOT / "config" / "default.yaml")
|
||||
features_df, total = predict_pdf(pdf_path, config)
|
||||
services = []
|
||||
for row in features_df.itertuples():
|
||||
if row.prediction == 1:
|
||||
services.append({
|
||||
"text": row.texto_fila,
|
||||
"probability": round(float(row.probability), 4),
|
||||
})
|
||||
return {
|
||||
"filename": pdf_path.name,
|
||||
"service_count": total,
|
||||
"status": "ok",
|
||||
"services": services,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
config = load_config(SC_ROOT / "config" / "default.yaml")
|
||||
pdfs = sorted(config.pdfs_dir.glob("*.pdf"))
|
||||
if not pdfs:
|
||||
raise SystemExit(f"No hay PDFs en {config.pdfs_dir}")
|
||||
|
||||
out = ROOT / "data" / "parity_baseline.json"
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
results = [python_result(p) for p in pdfs]
|
||||
out.write_text(json.dumps(results, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
print(f"Baseline guardado: {out} ({len(results)} PDFs)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,347 @@
|
||||
const MAX_SYNC_FILES = 50;␍
|
||||
const MAX_ASYNC_FILES = 500;␍
|
||||
const JOB_CHUNK_SIZE = 500;␍
|
||||
const LONG_POLL_MS = 30000;␍
|
||||
const TERMINAL_STATUSES = new Set(["completed", "failed", "cancelled"]);␍
|
||||
␍
|
||||
function $(id) {␍
|
||||
return document.getElementById(id);␍
|
||||
}␍
|
||||
␍
|
||||
function getBaseUrl() {␍
|
||||
const input = $("baseUrl").value.trim();␍
|
||||
return (input || window.location.origin).replace(/\/$/, "");␍
|
||||
}␍
|
||||
␍
|
||||
function getIncludeRejected() {␍
|
||||
return $("includeRejected").checked;␍
|
||||
}␍
|
||||
␍
|
||||
function getSelectedFiles(inputId) {␍
|
||||
const input = $(inputId);␍
|
||||
return Array.from(input.files || []);␍
|
||||
}␍
|
||||
␍
|
||||
function validateFiles(files, maxFiles) {␍
|
||||
if (files.length === 0) {␍
|
||||
throw new Error("Selecciona al menos un archivo PDF.");␍
|
||||
}␍
|
||||
if (files.length > maxFiles) {␍
|
||||
throw new Error(`Máximo ${maxFiles} archivos por petición.`);␍
|
||||
}␍
|
||||
for (const file of files) {␍
|
||||
if (!file.name.toLowerCase().endsWith(".pdf")) {␍
|
||||
throw new Error(`"${file.name}" no es un PDF.`);␍
|
||||
}␍
|
||||
}␍
|
||||
}␍
|
||||
␍
|
||||
async function parseError(res) {␍
|
||||
try {␍
|
||||
const data = await res.json();␍
|
||||
if (data && data.error) return data.error;␍
|
||||
} catch (_) {␍
|
||||
/* ignore */␍
|
||||
}␍
|
||||
return `Error HTTP ${res.status}`;␍
|
||||
}␍
|
||||
␍
|
||||
function buildMultipart(files) {␍
|
||||
const form = new FormData();␍
|
||||
for (const file of files) {␍
|
||||
form.append("files", file, file.name);␍
|
||||
}␍
|
||||
return form;␍
|
||||
}␍
|
||||
␍
|
||||
function setLoading(button, loading, label) {␍
|
||||
button.disabled = loading;␍
|
||||
button.textContent = loading ? "Procesando…" : label;␍
|
||||
}␍
|
||||
␍
|
||||
function showError(containerId, message) {␍
|
||||
const el = $(containerId);␍
|
||||
el.hidden = false;␍
|
||||
el.textContent = message;␍
|
||||
}␍
|
||||
␍
|
||||
function hideError(containerId) {␍
|
||||
const el = $(containerId);␍
|
||||
el.hidden = true;␍
|
||||
el.textContent = "";␍
|
||||
}␍
|
||||
␍
|
||||
function formatProbability(value) {␍
|
||||
if (typeof value !== "number") return "—";␍
|
||||
return (value * 100).toFixed(1) + "%";␍
|
||||
}␍
|
||||
␍
|
||||
function renderResults(container, results) {␍
|
||||
container.innerHTML = "";␍
|
||||
␍
|
||||
if (!results || results.length === 0) {␍
|
||||
container.innerHTML = '<p class="empty">Sin resultados.</p>';␍
|
||||
return;␍
|
||||
}␍
|
||||
␍
|
||||
for (const item of results) {␍
|
||||
const card = document.createElement("article");␍
|
||||
card.className = "result-card";␍
|
||||
␍
|
||||
const header = document.createElement("div");␍
|
||||
header.className = "result-header";␍
|
||||
header.innerHTML = `␍
|
||||
<h3>${escapeHtml(item.filename)}</h3>␍
|
||||
<span class="badge badge-${item.status}">${escapeHtml(item.status)}</span>␍
|
||||
`;␍
|
||||
card.appendChild(header);␍
|
||||
␍
|
||||
if (item.status === "error" && item.error) {␍
|
||||
const err = document.createElement("p");␍
|
||||
err.className = "result-error";␍
|
||||
err.textContent = item.error;␍
|
||||
card.appendChild(err);␍
|
||||
} else {␍
|
||||
const meta = document.createElement("p");␍
|
||||
meta.className = "result-meta";␍
|
||||
meta.textContent = `Servicios detectados: ${item.service_count ?? 0}`;␍
|
||||
card.appendChild(meta);␍
|
||||
␍
|
||||
if (item.services && item.services.length > 0) {␍
|
||||
const table = document.createElement("table");␍
|
||||
table.innerHTML = `␍
|
||||
<thead>␍
|
||||
<tr>␍
|
||||
<th>Texto</th>␍
|
||||
<th>Probabilidad</th>␍
|
||||
<th>Es servicio</th>␍
|
||||
</tr>␍
|
||||
</thead>␍
|
||||
<tbody></tbody>␍
|
||||
`;␍
|
||||
const tbody = table.querySelector("tbody");␍
|
||||
for (const row of item.services) {␍
|
||||
const tr = document.createElement("tr");␍
|
||||
const isService =␍
|
||||
row.is_service === undefined || row.is_service === null␍
|
||||
? "—"␍
|
||||
: row.is_service␍
|
||||
? "Sí"␍
|
||||
: "No";␍
|
||||
tr.innerHTML = `␍
|
||||
<td>${escapeHtml(row.text)}</td>␍
|
||||
<td>${formatProbability(row.probability)}</td>␍
|
||||
<td>${isService}</td>␍
|
||||
`;␍
|
||||
tbody.appendChild(tr);␍
|
||||
}␍
|
||||
card.appendChild(table);␍
|
||||
}␍
|
||||
}␍
|
||||
␍
|
||||
container.appendChild(card);␍
|
||||
}␍
|
||||
}␍
|
||||
␍
|
||||
function escapeHtml(text) {␍
|
||||
const div = document.createElement("div");␍
|
||||
div.textContent = text ?? "";␍
|
||||
return div.innerHTML;␍
|
||||
}␍
|
||||
␍
|
||||
async function countSync(baseUrl, files, includeRejected) {␍
|
||||
const url = new URL(`${baseUrl}/v1/count`);␍
|
||||
if (includeRejected) url.searchParams.set("include_rejected", "true");␍
|
||||
␍
|
||||
const res = await fetch(url, {␍
|
||||
method: "POST",␍
|
||||
body: buildMultipart(files),␍
|
||||
});␍
|
||||
␍
|
||||
if (!res.ok) throw new Error(await parseError(res));␍
|
||||
const data = await res.json();␍
|
||||
return data.results || [];␍
|
||||
}␍
|
||||
␍
|
||||
async function createEmptyJob(baseUrl, includeRejected) {␍
|
||||
const res = await fetch(`${baseUrl}/v1/jobs`, {␍
|
||||
method: "POST",␍
|
||||
headers: { "Content-Type": "application/json" },␍
|
||||
body: JSON.stringify({ include_rejected: includeRejected }),␍
|
||||
});␍
|
||||
if (!res.ok) throw new Error(await parseError(res));␍
|
||||
const data = await res.json();␍
|
||||
return data.job_id;␍
|
||||
}␍
|
||||
␍
|
||||
async function uploadChunk(baseUrl, jobId, files, includeRejected) {␍
|
||||
const url = new URL(`${baseUrl}/v1/jobs/${jobId}/upload`);␍
|
||||
if (includeRejected) url.searchParams.set("include_rejected", "true");␍
|
||||
␍
|
||||
const res = await fetch(url, {␍
|
||||
method: "POST",␍
|
||||
body: buildMultipart(files),␍
|
||||
});␍
|
||||
if (!res.ok) throw new Error(await parseError(res));␍
|
||||
return res.json();␍
|
||||
}␍
|
||||
␍
|
||||
async function getJobStatus(baseUrl, jobId, waitMs = 0) {␍
|
||||
const url = new URL(`${baseUrl}/v1/jobs/${jobId}`);␍
|
||||
if (waitMs > 0) url.searchParams.set("wait_ms", String(waitMs));␍
|
||||
␍
|
||||
const res = await fetch(url);␍
|
||||
if (!res.ok) throw new Error(await parseError(res));␍
|
||||
return res.json();␍
|
||||
}␍
|
||||
␍
|
||||
async function fetchAllResults(baseUrl, jobId) {␍
|
||||
const results = [];␍
|
||||
let cursor = "";␍
|
||||
let hasMore = true;␍
|
||||
␍
|
||||
while (hasMore) {␍
|
||||
const url = new URL(`${baseUrl}/v1/jobs/${jobId}/results`);␍
|
||||
url.searchParams.set("limit", "500");␍
|
||||
if (cursor) url.searchParams.set("cursor", cursor);␍
|
||||
␍
|
||||
const res = await fetch(url);␍
|
||||
if (!res.ok) throw new Error(await parseError(res));␍
|
||||
const page = await res.json();␍
|
||||
results.push(...(page.results || []));␍
|
||||
cursor = page.next_cursor || "";␍
|
||||
hasMore = Boolean(page.has_more);␍
|
||||
}␍
|
||||
␍
|
||||
return results;␍
|
||||
}␍
|
||||
␍
|
||||
function chunkArray(items, size) {␍
|
||||
const chunks = [];␍
|
||||
for (let i = 0; i < items.length; i += size) {␍
|
||||
chunks.push(items.slice(i, i + size));␍
|
||||
}␍
|
||||
return chunks;␍
|
||||
}␍
|
||||
␍
|
||||
async function runAsyncJob(baseUrl, files, includeRejected, onProgress) {␍
|
||||
const jobId = await createEmptyJob(baseUrl, includeRejected);␍
|
||||
const chunks = chunkArray(files, JOB_CHUNK_SIZE);␍
|
||||
␍
|
||||
for (let i = 0; i < chunks.length; i++) {␍
|
||||
onProgress({␍
|
||||
status: "uploading",␍
|
||||
total: files.length,␍
|
||||
processed: 0,␍
|
||||
failed: 0,␍
|
||||
uploadChunk: i + 1,␍
|
||||
uploadChunks: chunks.length,␍
|
||||
});␍
|
||||
await uploadChunk(baseUrl, jobId, chunks[i], includeRejected);␍
|
||||
}␍
|
||||
␍
|
||||
while (true) {␍
|
||||
const status = await getJobStatus(baseUrl, jobId, LONG_POLL_MS);␍
|
||||
onProgress(status);␍
|
||||
if (TERMINAL_STATUSES.has(status.status)) {␍
|
||||
if (status.status === "cancelled") {␍
|
||||
throw new Error("El job fue cancelado.");␍
|
||||
}␍
|
||||
break;␍
|
||||
}␍
|
||||
}␍
|
||||
␍
|
||||
return fetchAllResults(baseUrl, jobId);␍
|
||||
}␍
|
||||
␍
|
||||
function updateProgress(status) {␍
|
||||
const bar = $("asyncProgress");␍
|
||||
const label = $("asyncProgressLabel");␍
|
||||
const total = status.total || 0;␍
|
||||
const done = (status.processed || 0) + (status.failed || 0);␍
|
||||
const pct = total > 0 ? Math.min(100, Math.round((done / total) * 100)) : 0;␍
|
||||
␍
|
||||
bar.hidden = false;␍
|
||||
label.hidden = false;␍
|
||||
␍
|
||||
if (status.status === "uploading") {␍
|
||||
const chunkPct =␍
|
||||
status.uploadChunks > 0␍
|
||||
? Math.round((status.uploadChunk / status.uploadChunks) * 100)␍
|
||||
: 0;␍
|
||||
bar.value = chunkPct;␍
|
||||
label.textContent = `Subiendo lote ${status.uploadChunk}/${status.uploadChunks} (${total} archivos)…`;␍
|
||||
return;␍
|
||||
}␍
|
||||
␍
|
||||
bar.value = pct;␍
|
||||
label.textContent = `Estado: ${status.status} — ${done}/${total} procesados (${status.failed || 0} fallidos)`;␍
|
||||
}␍
|
||||
␍
|
||||
function initTabs() {␍
|
||||
const tabs = document.querySelectorAll(".tab");␍
|
||||
const panels = document.querySelectorAll(".panel");␍
|
||||
␍
|
||||
tabs.forEach((tab) => {␍
|
||||
tab.addEventListener("click", () => {␍
|
||||
tabs.forEach((t) => t.classList.remove("active"));␍
|
||||
panels.forEach((p) => p.classList.remove("active"));␍
|
||||
tab.classList.add("active");␍
|
||||
document.getElementById(tab.dataset.panel).classList.add("active");␍
|
||||
});␍
|
||||
});␍
|
||||
}␍
|
||||
␍
|
||||
function init() {␍
|
||||
$("baseUrl").value = window.location.origin;␍
|
||||
␍
|
||||
initTabs();␍
|
||||
␍
|
||||
$("syncForm").addEventListener("submit", async (e) => {␍
|
||||
e.preventDefault();␍
|
||||
const btn = $("syncSubmit");␍
|
||||
hideError("syncError");␍
|
||||
$("syncResults").innerHTML = "";␍
|
||||
␍
|
||||
try {␍
|
||||
const files = getSelectedFiles("syncFiles");␍
|
||||
validateFiles(files, MAX_SYNC_FILES);␍
|
||||
setLoading(btn, true, "Contar servicios");␍
|
||||
const results = await countSync(getBaseUrl(), files, getIncludeRejected());␍
|
||||
renderResults($("syncResults"), results);␍
|
||||
} catch (err) {␍
|
||||
showError("syncError", err.message);␍
|
||||
} finally {␍
|
||||
setLoading(btn, false, "Contar servicios");␍
|
||||
}␍
|
||||
});␍
|
||||
␍
|
||||
$("asyncForm").addEventListener("submit", async (e) => {␍
|
||||
e.preventDefault();␍
|
||||
const btn = $("asyncSubmit");␍
|
||||
hideError("asyncError");␍
|
||||
$("asyncResults").innerHTML = "";␍
|
||||
$("asyncProgress").hidden = true;␍
|
||||
$("asyncProgressLabel").hidden = true;␍
|
||||
␍
|
||||
try {␍
|
||||
const files = getSelectedFiles("asyncFiles");␍
|
||||
validateFiles(files, MAX_ASYNC_FILES);␍
|
||||
setLoading(btn, true, "Iniciar job");␍
|
||||
const results = await runAsyncJob(␍
|
||||
getBaseUrl(),␍
|
||||
files,␍
|
||||
getIncludeRejected(),␍
|
||||
updateProgress,␍
|
||||
);␍
|
||||
renderResults($("asyncResults"), results);␍
|
||||
} catch (err) {␍
|
||||
showError("asyncError", err.message);␍
|
||||
} finally {␍
|
||||
setLoading(btn, false, "Iniciar job");␍
|
||||
}␍
|
||||
});␍
|
||||
}␍
|
||||
␍
|
||||
document.addEventListener("DOMContentLoaded", init);␍
|
||||
␍
|
||||
@@ -0,0 +1,6 @@
|
||||
package web
|
||||
|
||||
import "embed"
|
||||
|
||||
//go:embed index.html app.js
|
||||
var FS embed.FS
|
||||
@@ -0,0 +1,307 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="es">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Services Count — Demo</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #f4f6f8;
|
||||
--surface: #ffffff;
|
||||
--text: #1a2332;
|
||||
--muted: #5c6b7a;
|
||||
--border: #d8dee6;
|
||||
--primary: #2563eb;
|
||||
--primary-hover: #1d4ed8;
|
||||
--error-bg: #fef2f2;
|
||||
--error-text: #b91c1c;
|
||||
--ok: #15803d;
|
||||
--warn: #b45309;
|
||||
--radius: 8px;
|
||||
}
|
||||
|
||||
* { box-sizing: border-box; }
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
font-family: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.container {
|
||||
max-width: 900px;
|
||||
margin: 0 auto;
|
||||
padding: 2rem 1.25rem 3rem;
|
||||
}
|
||||
|
||||
header {
|
||||
margin-bottom: 1.5rem;
|
||||
}
|
||||
|
||||
header h1 {
|
||||
margin: 0 0 0.25rem;
|
||||
font-size: 1.75rem;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
header p {
|
||||
margin: 0;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
header a {
|
||||
color: var(--primary);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
header a:hover { text-decoration: underline; }
|
||||
|
||||
.card {
|
||||
background: var(--surface);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
padding: 1.25rem;
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
|
||||
.field {
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
|
||||
label {
|
||||
display: block;
|
||||
font-weight: 600;
|
||||
margin-bottom: 0.35rem;
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
|
||||
input[type="text"],
|
||||
input[type="file"] {
|
||||
width: 100%;
|
||||
padding: 0.55rem 0.65rem;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
font: inherit;
|
||||
}
|
||||
|
||||
.checkbox {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
font-weight: normal;
|
||||
}
|
||||
|
||||
.checkbox input { width: auto; }
|
||||
|
||||
.tabs {
|
||||
display: flex;
|
||||
gap: 0.5rem;
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
|
||||
.tab {
|
||||
flex: 1;
|
||||
padding: 0.65rem 1rem;
|
||||
border: 1px solid var(--border);
|
||||
background: var(--surface);
|
||||
border-radius: var(--radius);
|
||||
cursor: pointer;
|
||||
font: inherit;
|
||||
font-weight: 600;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.tab.active {
|
||||
background: var(--primary);
|
||||
border-color: var(--primary);
|
||||
color: #fff;
|
||||
}
|
||||
|
||||
.panel { display: none; }
|
||||
.panel.active { display: block; }
|
||||
|
||||
.btn {
|
||||
display: inline-block;
|
||||
padding: 0.6rem 1.25rem;
|
||||
background: var(--primary);
|
||||
color: #fff;
|
||||
border: none;
|
||||
border-radius: var(--radius);
|
||||
font: inherit;
|
||||
font-weight: 600;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.btn:hover:not(:disabled) { background: var(--primary-hover); }
|
||||
.btn:disabled { opacity: 0.6; cursor: not-allowed; }
|
||||
|
||||
.error {
|
||||
background: var(--error-bg);
|
||||
color: var(--error-text);
|
||||
padding: 0.75rem 1rem;
|
||||
border-radius: var(--radius);
|
||||
margin-bottom: 1rem;
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
|
||||
.hint {
|
||||
font-size: 0.85rem;
|
||||
color: var(--muted);
|
||||
margin-top: 0.35rem;
|
||||
}
|
||||
|
||||
progress {
|
||||
width: 100%;
|
||||
height: 0.65rem;
|
||||
margin: 0.75rem 0 0.25rem;
|
||||
}
|
||||
|
||||
.progress-label {
|
||||
font-size: 0.85rem;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.results { margin-top: 1.25rem; }
|
||||
|
||||
.result-card {
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
padding: 1rem;
|
||||
margin-bottom: 0.75rem;
|
||||
background: #fafbfc;
|
||||
}
|
||||
|
||||
.result-header {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
gap: 1rem;
|
||||
margin-bottom: 0.5rem;
|
||||
}
|
||||
|
||||
.result-header h3 {
|
||||
margin: 0;
|
||||
font-size: 1rem;
|
||||
word-break: break-all;
|
||||
}
|
||||
|
||||
.badge {
|
||||
font-size: 0.75rem;
|
||||
font-weight: 700;
|
||||
text-transform: uppercase;
|
||||
padding: 0.2rem 0.5rem;
|
||||
border-radius: 4px;
|
||||
background: #e2e8f0;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.badge-ok { background: #dcfce7; color: var(--ok); }
|
||||
.badge-error { background: #fee2e2; color: var(--error-text); }
|
||||
.badge-pending { background: #fef3c7; color: var(--warn); }
|
||||
|
||||
.result-meta {
|
||||
margin: 0 0 0.75rem;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.result-error {
|
||||
margin: 0;
|
||||
color: var(--error-text);
|
||||
}
|
||||
|
||||
table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
font-size: 0.875rem;
|
||||
}
|
||||
|
||||
th, td {
|
||||
text-align: left;
|
||||
padding: 0.45rem 0.5rem;
|
||||
border-bottom: 1px solid var(--border);
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
th {
|
||||
font-size: 0.75rem;
|
||||
text-transform: uppercase;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.empty {
|
||||
color: var(--muted);
|
||||
font-style: italic;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<header>
|
||||
<h1>Services Count — Demo</h1>
|
||||
<p>
|
||||
Cliente de ejemplo para la API de conteo de servicios en facturas médicas.
|
||||
<a href="/openapi.json" target="_blank" rel="noopener">OpenAPI</a>
|
||||
</p>
|
||||
</header>
|
||||
|
||||
<div class="card">
|
||||
<div class="field">
|
||||
<label for="baseUrl">URL base de la API</label>
|
||||
<input type="text" id="baseUrl" placeholder="http://localhost:8080">
|
||||
<p class="hint">Por defecto usa el mismo origen desde el que se sirve esta página.</p>
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
<label class="checkbox">
|
||||
<input type="checkbox" id="includeRejected">
|
||||
Incluir filas rechazadas (<code>include_rejected</code>)
|
||||
</label>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="tabs">
|
||||
<button type="button" class="tab active" data-panel="syncPanel">Síncrono</button>
|
||||
<button type="button" class="tab" data-panel="asyncPanel">Asíncrono</button>
|
||||
</div>
|
||||
|
||||
<div id="syncPanel" class="panel active">
|
||||
<div class="card">
|
||||
<form id="syncForm">
|
||||
<div class="field">
|
||||
<label for="syncFiles">Archivos PDF (máx. 50)</label>
|
||||
<input type="file" id="syncFiles" accept=".pdf,application/pdf" multiple required>
|
||||
<p class="hint">Envía una petición <code>POST /v1/count</code> y muestra la respuesta al instante.</p>
|
||||
</div>
|
||||
<div id="syncError" class="error" hidden></div>
|
||||
<button type="submit" class="btn" id="syncSubmit">Contar servicios</button>
|
||||
</form>
|
||||
<div id="syncResults" class="results"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="asyncPanel" class="panel">
|
||||
<div class="card">
|
||||
<form id="asyncForm">
|
||||
<div class="field">
|
||||
<label for="asyncFiles">Archivos PDF (hasta 500)</label>
|
||||
<input type="file" id="asyncFiles" accept=".pdf,application/pdf" multiple required>
|
||||
<p class="hint">
|
||||
Flujo producción: MinIO → cola NATS → workers (autoscaler).
|
||||
Sube por lotes de hasta 500; el progreso usa long-polling.
|
||||
</p>
|
||||
</div>
|
||||
<div id="asyncError" class="error" hidden></div>
|
||||
<progress id="asyncProgress" value="0" max="100" hidden></progress>
|
||||
<p id="asyncProgressLabel" class="progress-label" hidden></p>
|
||||
<button type="submit" class="btn" id="asyncSubmit">Iniciar job</button>
|
||||
</form>
|
||||
<div id="asyncResults" class="results"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script src="app.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user