PYS/IPSExtractor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
37 Commits 1 Branch 0 Tags
343ffb8ebe831d98aacca7b0c23a8c58ee902ca1
T
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
erik 343ffb8ebe feat: Implement job abandonment logic for reconnected workers in bot services. Introduce a mechanism to cancel pending and running jobs when a worker reconnects, ensuring a fresh start for new jobs. Update tests to verify job cancellation behavior upon worker re-registration. Enhance UI components in desktop application for better user experience and clarity on bot status.
2026-08-12 16:37:12 -05:00
bots
feat: Implement job abandonment logic for reconnected workers in bot services. Introduce a mechanism to cancel pending and running jobs when a worker reconnects, ensuring a fresh start for new jobs. Update tests to verify job cancellation behavior upon worker re-registration. Enhance UI components in desktop application for better user experience and clarity on bot status.
2026-08-12 16:37:12 -05:00
docs
feat: Add poblacion_apta service to docker-compose, enhance API gateway for new service integration, and update README with service details. Introduce new database tables for poblacion_apta_jobs and poblacion_apta_rows, and implement functionality for handling cop/mod calculations in the cleaning service.
2026-07-28 04:29:14 -05:00
services
feat: Implement job abandonment logic for reconnected workers in bot services. Introduce a mechanism to cancel pending and running jobs when a worker reconnects, ensuring a fresh start for new jobs. Update tests to verify job cancellation behavior upon worker re-registration. Enhance UI components in desktop application for better user experience and clarity on bot status.
2026-08-12 16:37:12 -05:00
.cursorignore
feat: Implement the initial IPSExtractor project structure, including application logic, GUI, core utilities, and documentation.
2026-02-23 09:24:06 -05:00
.env.example
feat: Update .env.example and Docker Compose configurations to enhance service performance and health checks. Adjust OCR settings for improved document processing efficiency, and refine API Gateway health checks for better service monitoring. Revise desktop application documentation to reflect new API Gateway URL and installation instructions.
2026-07-06 04:30:27 -05:00
.gitignore
feat: Enhance testing framework by adding new PDF files for count-api latency tests, updating .gitignore to include additional CSV reports, and refining documentation for improved clarity on test results and methodologies.
2026-07-13 22:46:19 -05:00
bot_saludtotal.py
feat: Introduce bot-digitador service for automated data entry in Integración Total. Add CLI and GUI interfaces for CSV processing, implement auto-start functionality on Windows, and enhance README with setup instructions and usage details. Include necessary configurations in docker-compose for service integration.
2026-08-09 21:26:56 -05:00
descarga.py
feat: Introduce bot-digitador service for automated data entry in Integración Total. Add CLI and GUI interfaces for CSV processing, implement auto-start functionality on Windows, and enhance README with setup instructions and usage details. Include necessary configurations in docker-compose for service integration.
2026-08-09 21:26:56 -05:00
docker-compose.yml
feat: Introduce bot-digitador service for automated data entry in Integración Total. Add CLI and GUI interfaces for CSV processing, implement auto-start functionality on Windows, and enhance README with setup instructions and usage details. Include necessary configurations in docker-compose for service integration.
2026-08-09 21:26:56 -05:00
pyproject.toml
feat: Implement the initial IPSExtractor project structure, including application logic, GUI, core utilities, and documentation.
2026-02-23 09:24:06 -05:00
README.md
feat: Add poblacion_apta service to docker-compose, enhance API gateway for new service integration, and update README with service details. Introduce new database tables for poblacion_apta_jobs and poblacion_apta_rows, and implement functionality for handling cop/mod calculations in the cleaning service.
2026-07-28 04:29:14 -05:00
Verificador_ST.py
feat: Add poblacion_apta service to docker-compose, enhance API gateway for new service integration, and update README with service details. Introduce new database tables for poblacion_apta_jobs and poblacion_apta_rows, and implement functionality for handling cop/mod calculations in the cleaning service.
2026-07-28 04:29:14 -05:00

README.md

IPSExtractor

Sistema de extracción y análisis de datos desde facturas médicas PDF. Arquitectura de microservicios: cada servicio tiene su propia API, dependencias, Dockerfile y código de dominio. No hay paquetes compartidos entre servicios.

Servicios

Servicio Puerto (host) Puerto (interno) Tecnología Descripción
api_gateway 8189 8000 Python/FastAPI Única puerta de entrada expuesta al host
extraction — 8001 Python/FastAPI Extracción LLM; solo red Docker
cleaning — 8002 Python/FastAPI Limpieza; solo red Docker
poblacion_apta — 8003 Python/FastAPI Población apta (Verificador ST); solo red Docker
count-api — 8080 Go Conteo OCR+ONNX; solo red Docker
database — 5432 PostgreSQL 16 Solo red Docker
redis — 6379 Redis 7 Solo red Docker
desktop_app local Electron + React Aplicación de escritorio
bot-descarga (API) — 8200 Python/FastAPI Control remoto de bots; solo red Docker

Bots externos (bots/)

Agents para instalar en PCs (no Docker). Carpetas hermanas de services/.

Carpeta Descripción
bots/bot_descarga Agent Selenium + MinIO; se conecta a la API services/bot_descarga vía gateway

Ver bots/README.md.

Flujo principal

Desktop app
  │
  ├─ Conteo (OCR + ML)
  │   ├─ POST /count/v1/count  →  count-api  (lotes de 10 PDFs)
  │   ├─ POST /count-jobs      →  extraction (guarda en PostgreSQL)
  │   └─ PATCH /count-jobs     →  extraction (validación humana)
  │
  └─ Extracción (LLM)
      ├─ POST /tasks/extraction/from-count  →  extraction (lee desde /data)
      ├─ POST /tasks/{id}/ingest-many       →  extraction (subida de PDFs)
      └─ GET  /tasks/{id}/rows              →  extraction (resultados)

Arrancar el stack completo

# Clonar y configurar
cp .env.example .env
# Editar .env con las API keys del LLM y la ruta de documentos

docker compose up --build

Health checks:

curl http://127.0.0.1:8189/health/live   # liveness del gateway (rápido)
curl http://127.0.0.1:8189/health        # gateway (agrega estado de los 3 servicios)
curl http://127.0.0.1:8189/jobs          # lista de trabajos

Windows: use 127.0.0.1 en lugar de localhost para el API Gateway. Docker publica el puerto en IPv4; localhost puede resolver a ::1 y la conexión queda colgada.

Los demás servicios (extraction, count-api, postgres, redis) no exponen puertos al host; solo el gateway en 8189.

Solución a fallos de build paralelo en Docker Desktop

Si docker compose up --build falla con frontend grpc server closed unexpectedly al construir count-api, es porque Docker Desktop (Compose v2.37+) lanza todos los builds en paralelo y BuildKit se satura. Hay dos opciones:

Opción A — pre-construir count-api antes del stack completo (recomendada):

docker compose build count-api
docker compose up --build

Opción B — desactivar Bake y limitar el paralelismo:

# PowerShell
$env:COMPOSE_BAKE="false"
$env:COMPOSE_PARALLEL_LIMIT="2"
docker compose up --build

# bash / WSL
COMPOSE_BAKE=false COMPOSE_PARALLEL_LIMIT=2 docker compose up --build

Ver también las variables COMPOSE_BAKE y COMPOSE_PARALLEL_LIMIT en .env.example.

Variables de entorno principales

Variable Descripción
IPS_DOCUMENT_ROOT Carpeta local con los documentos PDF (montada en /data)
EXTRACTOR_API_KEY API key del LLM (OpenAI, xAI, etc.)
EXTRACTOR_BASE_URL URL base del LLM (default: OpenAI)
EXTRACTOR_MODEL Modelo de extracción (default: gpt-5.4-mini)
WORKER_REPLICAS Réplicas del worker de extracción (default: 2)

Ver .env.example para la lista completa.

Estructura del repositorio

bots/
  bot_descarga/       — agent Selenium para PCs (instalación externa) → README
services/
  api_gateway/        — proxy HTTP (Python)           → README
  extraction/         — LLM + jobs + conteo BD (Python) → README
  cleaning/           — limpieza de datos (Python)     → README
  poblacion_apta/     — población apta / Verificador ST → README
  bot_descarga/       — API control remoto de bots     → README
  count-api/          — OCR + ONNX stateless (Go)      → README
  db/                 — migraciones Alembic             → README
  desktop_app/        — app Electron + React            → README
docs/
  database.dbml       — esquema completo de la BD
docker-compose.yml    — stack completo
.env.example          — variables de entorno de referencia

Base de datos (esquema resumido)

jobs                   — contenedor de un trabajo del usuario
  tasks                — módulo ejecutado (counting / extraction / cleaning / poblacion_apta)
    count_job_files    — resultado de conteo por PDF
    extraction_batches — lote de extracción
      extraction_rows  — fila extraída por documento
    cleaning_jobs      — job de limpieza
      cleaned_rows     — fila transformada
    poblacion_apta_jobs — consolidado validado (PASO_BOT)
      poblacion_apta_rows

Esquema completo: docs/database.dbml

Documentación por servicio

Reference in New Issue View Git Blame Copy Permalink
S
Description
No description provided
Readme
10 MiB
Languages
Python 56.7%
TypeScript 31.3%
CSS 5.5%
JavaScript 3.7%
Go 2%
Other 0.8%