feat: Introduce bot-descarga service for PDF downloads and remote bot control. Add Docker configuration, environment variables, and a GUI for agent management. Enhance README with bot details and installation instructions.
This commit is contained in:
@@ -16,9 +16,20 @@ Arquitectura de microservicios: cada servicio tiene su propia API, dependencias,
|
||||
| `database` | — | 5432 | PostgreSQL 16 | Solo red Docker |
|
||||
| `redis` | — | 6379 | Redis 7 | Solo red Docker |
|
||||
| `desktop_app` | local | Electron + React | Aplicación de escritorio |
|
||||
| `bot-descarga` (API) | — | 8200 | Python/FastAPI | Control remoto de bots; solo red Docker |
|
||||
|
||||
---
|
||||
|
||||
## Bots externos (`bots/`)
|
||||
|
||||
Agents para instalar **en PCs** (no Docker). Carpetas hermanas de `services/`.
|
||||
|
||||
| Carpeta | Descripción |
|
||||
|---------|-------------|
|
||||
| [`bots/bot_descarga`](bots/bot_descarga) | Agent Selenium + MinIO; se conecta a la API `services/bot_descarga` vía gateway |
|
||||
|
||||
Ver [`bots/README.md`](bots/README.md).
|
||||
|
||||
## Flujo principal
|
||||
|
||||
```
|
||||
@@ -105,10 +116,13 @@ Ver `.env.example` para la lista completa.
|
||||
## Estructura del repositorio
|
||||
|
||||
```
|
||||
bots/
|
||||
bot_descarga/ — agent Selenium para PCs (instalación externa) → README
|
||||
services/
|
||||
api_gateway/ — proxy HTTP (Python) → README
|
||||
extraction/ — LLM + jobs + conteo BD (Python) → README
|
||||
cleaning/ — limpieza de datos (Python) → README
|
||||
bot_descarga/ — API control remoto de bots → README
|
||||
count-api/ — OCR + ONNX stateless (Go) → README
|
||||
db/ — migraciones Alembic → README
|
||||
desktop_app/ — app Electron + React → README
|
||||
@@ -145,3 +159,5 @@ Esquema completo: [`docs/database.dbml`](docs/database.dbml)
|
||||
- [`services/services-count-facturas-medicas-serving/README.md`](services/services-count-facturas-medicas-serving/README.md)
|
||||
- [`services/db/README.md`](services/db/README.md)
|
||||
- [`services/desktop_app/README.md`](services/desktop_app/README.md)
|
||||
- [`services/bot_descarga/README.md`](services/bot_descarga/README.md)
|
||||
- [`bots/bot_descarga/README.md`](bots/bot_descarga/README.md)
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
# Bots externos
|
||||
|
||||
Agents que se **instalan en PCs** (Chrome, Imaging, etc.). No van en Docker ni en `services/`.
|
||||
|
||||
| Carpeta | Rol |
|
||||
|---------|-----|
|
||||
| [`bot_descarga/`](bot_descarga/) | Agent Selenium: descarga PDFs y sube a MinIO; habla con `services/bot_descarga` vía gateway |
|
||||
|
||||
La API / persistencia correspondiente vive en `services/<nombre>/`.
|
||||
@@ -0,0 +1,14 @@
|
||||
# Token hacia la API IPSExtractor (mismo que BOT_API_TOKEN del servidor)
|
||||
BOT_API_TOKEN=dev-token-change-me
|
||||
BOT_API_URL=https://pys1.builker.xyz/bot-descarga
|
||||
|
||||
# Imaging (hostname interno; requiere VPN/DNS o IP alcanzable)
|
||||
BOT_IMAGING_URL=http://serviciosimaging:8180/
|
||||
|
||||
# MinIO — bucket pys-cuentas-medicas: sin credenciales (PutObject anónimo)
|
||||
MINIO_ENDPOINT=https://s3.builker.xyz
|
||||
MINIO_BUCKET=pys-cuentas-medicas
|
||||
MINIO_ANONYMOUS=1
|
||||
MINIO_ACCESS_KEY=
|
||||
MINIO_SECRET_KEY=
|
||||
MINIO_REGION=us-east-1
|
||||
@@ -0,0 +1,18 @@
|
||||
logs/*
|
||||
!logs/.gitkeep
|
||||
data/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
.venv/
|
||||
venv/
|
||||
env/
|
||||
descargas/
|
||||
*.egg-info/
|
||||
.pytest_cache/
|
||||
.mypy_cache/
|
||||
dist/
|
||||
build/
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
.env
|
||||
@@ -0,0 +1,80 @@
|
||||
# Bot de descarga (agent para PC remota)
|
||||
|
||||
Agent Selenium que corre **fuera de Docker**, en la PC con Chrome e Imaging.
|
||||
Vive en este monorepo bajo `bots/` (no es un microservicio de `services/`).
|
||||
|
||||
- **No expone puertos.** Solo HTTP saliente al gateway IPSExtractor y PutObject a MinIO.
|
||||
- La API de control está en [`services/bot_descarga`](../../services/bot_descarga) (Docker + PostgreSQL).
|
||||
|
||||
## Requisitos en la PC
|
||||
|
||||
- Python 3.11+
|
||||
- Google Chrome
|
||||
- Acceso de red a Imaging (`BOT_IMAGING_URL`) y al gateway (`BOT_API_URL`)
|
||||
- Token `BOT_API_TOKEN` (mismo que el servicio en Docker)
|
||||
|
||||
## Instalación
|
||||
|
||||
```bash
|
||||
cd bots/bot_descarga
|
||||
python -m venv .venv
|
||||
|
||||
# Windows
|
||||
.venv\Scripts\activate
|
||||
# Linux/macOS
|
||||
# source .venv/bin/activate
|
||||
|
||||
pip install -r requirements.txt
|
||||
cp .env.example .env
|
||||
# Editar .env
|
||||
```
|
||||
|
||||
## Arranque con GUI
|
||||
|
||||
```bash
|
||||
python run_gui.py
|
||||
```
|
||||
|
||||
Configura URL API (ej. `https://pys1.builker.xyz/bot-descarga` o `http://127.0.0.1:8189/bot-descarga`), Worker ID y token. Pulsa **Conectar agent**. Queda idle hasta que el desktop asigne un job.
|
||||
|
||||
## Arranque solo agent (CLI)
|
||||
|
||||
```bash
|
||||
python -m bot_descarga.worker.run \
|
||||
--api-url http://127.0.0.1:8189/bot-descarga \
|
||||
--worker-id PC-01 \
|
||||
--token $BOT_API_TOKEN
|
||||
```
|
||||
|
||||
## Arquitectura
|
||||
|
||||
```
|
||||
Desktop IPSExtractor → Gateway :8189 → services/bot_descarga (Docker)
|
||||
↑
|
||||
PC bot (esta carpeta) ────────┘ HTTP outbound
|
||||
└─ PutObject PDFs → MinIO pys-cuentas-medicas/downloads/
|
||||
```
|
||||
|
||||
## Variables
|
||||
|
||||
Ver [`.env.example`](.env.example):
|
||||
|
||||
| Variable | Uso |
|
||||
|----------|-----|
|
||||
| `BOT_API_URL` | Base del API vía gateway (`…/bot-descarga`) |
|
||||
| `BOT_API_TOKEN` | Bearer compartido con el servicio |
|
||||
| `BOT_IMAGING_URL` | URL del sistema Imaging (requiere VPN/DNS) |
|
||||
| `MINIO_*` | Bucket de PDFs; por defecto anónimo |
|
||||
|
||||
## Debug local (sin API remota)
|
||||
|
||||
```bash
|
||||
python main.py --excel datos.xlsx --carpeta descargas
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
cd bots/bot_descarga
|
||||
python -m pytest tests/ -q
|
||||
```
|
||||
@@ -0,0 +1,3 @@
|
||||
"""Bot de Descargas PDF — automatización de facturas desde Imaging."""
|
||||
|
||||
__version__ = "2.0.0"
|
||||
@@ -0,0 +1,10 @@
|
||||
from bot_descarga.audit.events import BotEvent, EventType, create_event
|
||||
from bot_descarga.audit.jsonl_logger import JsonlLogger, EventBus
|
||||
|
||||
__all__ = [
|
||||
"BotEvent",
|
||||
"EventType",
|
||||
"create_event",
|
||||
"JsonlLogger",
|
||||
"EventBus",
|
||||
]
|
||||
@@ -0,0 +1,81 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field, asdict
|
||||
from datetime import datetime, timezone
|
||||
from enum import Enum
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
|
||||
class EventType(str, Enum):
|
||||
RUN_STARTED = "run_started"
|
||||
NIT_STARTED = "nit_started"
|
||||
NIT_FINISHED = "nit_finished"
|
||||
RADICADO_PROCESSING = "radicado_processing"
|
||||
RADICADO_SKIPPED = "radicado_skipped"
|
||||
RADICADO_SAVED = "radicado_saved"
|
||||
RADICADO_FAILED = "radicado_failed"
|
||||
BROWSER_RECONNECTED = "browser_reconnected"
|
||||
RUN_FINISHED = "run_finished"
|
||||
INFO = "info"
|
||||
WARNING = "warning"
|
||||
|
||||
|
||||
@dataclass
|
||||
class BotEvent:
|
||||
timestamp: str
|
||||
event_type: EventType
|
||||
message: str
|
||||
radicado: Optional[str] = None
|
||||
nit: Optional[str] = None
|
||||
index: Optional[int] = None
|
||||
total: Optional[int] = None
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
data = asdict(self)
|
||||
data["event_type"] = self.event_type.value
|
||||
return data
|
||||
|
||||
def display_message(self) -> str:
|
||||
"""Mensaje legible para consola / UI."""
|
||||
prefix = ""
|
||||
if self.index is not None and self.total is not None:
|
||||
prefix = f"[{self.index}/{self.total}] "
|
||||
|
||||
icons = {
|
||||
EventType.RUN_STARTED: "",
|
||||
EventType.NIT_STARTED: "📂 ",
|
||||
EventType.NIT_FINISHED: "🏁 ",
|
||||
EventType.RADICADO_PROCESSING: "🔄 ",
|
||||
EventType.RADICADO_SKIPPED: "✅ ",
|
||||
EventType.RADICADO_SAVED: "✅ ",
|
||||
EventType.RADICADO_FAILED: "❌ ",
|
||||
EventType.BROWSER_RECONNECTED: "⚠️ ",
|
||||
EventType.RUN_FINISHED: "",
|
||||
EventType.INFO: "",
|
||||
EventType.WARNING: "⚠️ ",
|
||||
}
|
||||
icon = icons.get(self.event_type, "")
|
||||
return f"{prefix}{icon}{self.message}"
|
||||
|
||||
|
||||
def create_event(
|
||||
event_type: EventType,
|
||||
message: str,
|
||||
*,
|
||||
radicado: Optional[str] = None,
|
||||
nit: Optional[str] = None,
|
||||
index: Optional[int] = None,
|
||||
total: Optional[int] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
) -> BotEvent:
|
||||
return BotEvent(
|
||||
timestamp=datetime.now(timezone.utc).astimezone().isoformat(timespec="seconds"),
|
||||
event_type=event_type,
|
||||
message=message,
|
||||
radicado=radicado,
|
||||
nit=nit,
|
||||
index=index,
|
||||
total=total,
|
||||
metadata=metadata or {},
|
||||
)
|
||||
@@ -0,0 +1,61 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Callable, List, Optional
|
||||
|
||||
from bot_descarga.audit.events import BotEvent
|
||||
|
||||
|
||||
class JsonlLogger:
|
||||
"""Escribe un evento por línea en un archivo .jsonl de auditoría."""
|
||||
|
||||
def __init__(self, logs_dir: Path | str = "logs"):
|
||||
self.logs_dir = Path(logs_dir)
|
||||
self.logs_dir.mkdir(parents=True, exist_ok=True)
|
||||
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
self.path = self.logs_dir / f"audit_{stamp}.jsonl"
|
||||
self._file = open(self.path, "a", encoding="utf-8")
|
||||
|
||||
def write(self, event: BotEvent) -> None:
|
||||
self._file.write(json.dumps(event.to_dict(), ensure_ascii=False) + "\n")
|
||||
self._file.flush()
|
||||
|
||||
def close(self) -> None:
|
||||
if self._file and not self._file.closed:
|
||||
self._file.close()
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
self.close()
|
||||
|
||||
|
||||
EventCallback = Callable[[BotEvent], None]
|
||||
|
||||
|
||||
class EventBus:
|
||||
"""Distribuye eventos a logger JSONL y callbacks (UI / consola)."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
jsonl_logger: Optional[JsonlLogger] = None,
|
||||
callbacks: Optional[List[EventCallback]] = None,
|
||||
):
|
||||
self.jsonl_logger = jsonl_logger
|
||||
self.callbacks: List[EventCallback] = list(callbacks or [])
|
||||
|
||||
def emit(self, event: BotEvent) -> None:
|
||||
if self.jsonl_logger is not None:
|
||||
self.jsonl_logger.write(event)
|
||||
for cb in self.callbacks:
|
||||
cb(event)
|
||||
|
||||
def add_callback(self, callback: EventCallback) -> None:
|
||||
self.callbacks.append(callback)
|
||||
|
||||
def close(self) -> None:
|
||||
if self.jsonl_logger is not None:
|
||||
self.jsonl_logger.close()
|
||||
@@ -0,0 +1,3 @@
|
||||
from bot_descarga.config.settings import Settings, DEFAULT_SETTINGS
|
||||
|
||||
__all__ = ["Settings", "DEFAULT_SETTINGS"]
|
||||
@@ -0,0 +1,33 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Settings:
|
||||
url_sistema: str = "http://serviciosimaging:8180/"
|
||||
peso_minimo_bytes: int = 10_000
|
||||
max_intentos_click: int = 3
|
||||
max_intentos_pdf: int = 2
|
||||
timeout_descarga: int = 35
|
||||
max_reconexiones: int = 5
|
||||
wait_timeout: int = 10
|
||||
grupo: str = "CUENTAS MEDICAS"
|
||||
perfil: str = "Cuentas Medicas"
|
||||
logs_dir: Path = field(default_factory=lambda: Path("logs"))
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> "Settings":
|
||||
url = (
|
||||
os.getenv("BOT_IMAGING_URL")
|
||||
or os.getenv("IMAGING_URL")
|
||||
or cls.url_sistema
|
||||
).strip()
|
||||
if url and not url.endswith("/"):
|
||||
url = f"{url}/"
|
||||
return cls(url_sistema=url or cls.url_sistema)
|
||||
|
||||
|
||||
DEFAULT_SETTINGS = Settings.from_env()
|
||||
@@ -0,0 +1,3 @@
|
||||
from bot_descarga.core.download_service import DownloadService
|
||||
|
||||
__all__ = ["DownloadService"]
|
||||
@@ -0,0 +1,377 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Callable, Optional
|
||||
|
||||
from bot_descarga.audit.events import EventType, create_event
|
||||
from bot_descarga.audit.jsonl_logger import EventBus, JsonlLogger
|
||||
from bot_descarga.config.settings import Settings, DEFAULT_SETTINGS
|
||||
from bot_descarga.infrastructure.browser import BrowserManager, SESION_MUERTA
|
||||
from bot_descarga.infrastructure.excel_reader import ExcelReader
|
||||
from bot_descarga.infrastructure.file_manager import FileManager
|
||||
from bot_descarga.infrastructure.imaging_client import ImagingClient
|
||||
from bot_descarga.infrastructure.pdf_validator import PdfValidator
|
||||
from bot_descarga.models.resultado import (
|
||||
DownloadResult,
|
||||
NitStats,
|
||||
ResultStatus,
|
||||
RunSummary,
|
||||
)
|
||||
|
||||
|
||||
class DownloadService:
|
||||
"""Orquesta la descarga masiva de facturas PDF por radicado/NIT."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
ruta_excel: str,
|
||||
carpeta_descarga: str,
|
||||
settings: Settings = DEFAULT_SETTINGS,
|
||||
event_callback: Optional[Callable] = None,
|
||||
should_cancel: Optional[Callable[[], bool]] = None,
|
||||
on_pdf_saved: Optional[Callable[[str, str, str], None]] = None,
|
||||
):
|
||||
self.ruta_excel = ruta_excel
|
||||
self.carpeta_descarga = carpeta_descarga
|
||||
self.settings = settings
|
||||
self.should_cancel = should_cancel or (lambda: False)
|
||||
self.on_pdf_saved = on_pdf_saved
|
||||
|
||||
self.validator = PdfValidator(settings)
|
||||
self.files = FileManager(carpeta_descarga, self.validator)
|
||||
self.excel = ExcelReader()
|
||||
self.browser = BrowserManager(settings)
|
||||
|
||||
jsonl = JsonlLogger(settings.logs_dir)
|
||||
callbacks = [event_callback] if event_callback else []
|
||||
self.bus = EventBus(jsonl_logger=jsonl, callbacks=callbacks)
|
||||
|
||||
def _emit(self, event_type: EventType, message: str, **kwargs) -> None:
|
||||
self.bus.emit(create_event(event_type, message, **kwargs))
|
||||
|
||||
def _log_text(self, message: str) -> None:
|
||||
"""Adaptador texto → evento INFO (para BrowserManager / ImagingClient)."""
|
||||
self._emit(EventType.INFO, message)
|
||||
|
||||
def _friendly_browser_error(self, exc: Exception) -> str:
|
||||
raw = str(exc)
|
||||
url = self.settings.url_sistema
|
||||
if "ERR_NAME_NOT_RESOLVED" in raw:
|
||||
return (
|
||||
f"No se pudo resolver el host de Imaging ({url}). "
|
||||
"Revisa VPN/DNS o define BOT_IMAGING_URL. "
|
||||
"Detalle: net::ERR_NAME_NOT_RESOLVED"
|
||||
)
|
||||
if "ERR_CONNECTION_REFUSED" in raw or "ERR_CONNECTION_TIMED_OUT" in raw:
|
||||
return f"Imaging no responde en {url}. Detalle: {raw.splitlines()[0]}"
|
||||
return f"No se pudo abrir Imaging ({url}): {raw.splitlines()[0]}"
|
||||
|
||||
def run(self) -> RunSummary:
|
||||
summary = RunSummary()
|
||||
reconexiones = 0
|
||||
nit_actual = None
|
||||
|
||||
try:
|
||||
filas = self.excel.read(self.ruta_excel)
|
||||
summary.total = len(filas)
|
||||
|
||||
if summary.total == 0:
|
||||
self._emit(
|
||||
EventType.RADICADO_FAILED,
|
||||
"No se encontraron radicados en el Excel.",
|
||||
)
|
||||
return summary
|
||||
|
||||
self._emit(
|
||||
EventType.RUN_STARTED,
|
||||
f"Iniciando descarga de {summary.total} radicados",
|
||||
total=summary.total,
|
||||
metadata={
|
||||
"excel": self.ruta_excel,
|
||||
"carpeta": self.carpeta_descarga,
|
||||
"audit_log": str(self.bus.jsonl_logger.path),
|
||||
"total": summary.total,
|
||||
"url_sistema": self.settings.url_sistema,
|
||||
},
|
||||
)
|
||||
|
||||
try:
|
||||
driver, wait = self.browser.start(
|
||||
self.carpeta_descarga, self._log_text
|
||||
)
|
||||
except Exception as e:
|
||||
summary.fatal_error = self._friendly_browser_error(e)
|
||||
self._emit(
|
||||
EventType.WARNING,
|
||||
summary.fatal_error,
|
||||
total=summary.total,
|
||||
metadata={"motivo": str(e), "url_sistema": self.settings.url_sistema},
|
||||
)
|
||||
return summary
|
||||
|
||||
imaging = ImagingClient(
|
||||
driver, wait, self.settings, self.validator, self.files
|
||||
)
|
||||
|
||||
idx = 0
|
||||
while idx < summary.total:
|
||||
if self.should_cancel():
|
||||
summary.cancelled = True
|
||||
self._emit(
|
||||
EventType.WARNING,
|
||||
"Cancelación solicitada — deteniendo proceso",
|
||||
index=idx + 1,
|
||||
total=summary.total,
|
||||
)
|
||||
break
|
||||
|
||||
record = filas[idx]
|
||||
numero, nit = record.radicado, record.nit
|
||||
|
||||
if nit not in summary.stats_por_nit:
|
||||
summary.stats_por_nit[nit] = NitStats()
|
||||
|
||||
stats: NitStats = summary.stats_por_nit[nit]
|
||||
stats.total += 1
|
||||
|
||||
if nit != nit_actual:
|
||||
if nit_actual is not None:
|
||||
prev = summary.stats_por_nit[nit_actual]
|
||||
self._emit(
|
||||
EventType.NIT_FINISHED,
|
||||
(
|
||||
f"FINALIZÓ NIT {nit_actual} | "
|
||||
f"OK: {prev.ok} | "
|
||||
f"Fallidos: {prev.fallidos} | "
|
||||
f"Ya existían: {prev.omitidos}"
|
||||
),
|
||||
nit=nit_actual,
|
||||
metadata={
|
||||
"ok": prev.ok,
|
||||
"fallidos": prev.fallidos,
|
||||
"omitidos": prev.omitidos,
|
||||
},
|
||||
)
|
||||
|
||||
nit_actual = nit
|
||||
self._emit(
|
||||
EventType.NIT_STARTED,
|
||||
f"INICIANDO NIT {nit_actual}",
|
||||
nit=nit_actual,
|
||||
)
|
||||
|
||||
ruta_final = self.files.ruta_final(nit, numero)
|
||||
existe, _, motivo_exist = self.files.exists_valid(nit, numero)
|
||||
|
||||
if existe:
|
||||
self._emit(
|
||||
EventType.RADICADO_SKIPPED,
|
||||
f"Ya existe: {nit}/{numero}.pdf",
|
||||
radicado=numero,
|
||||
nit=nit,
|
||||
index=idx + 1,
|
||||
total=summary.total,
|
||||
)
|
||||
summary.exitosos += 1
|
||||
stats.omitidos += 1
|
||||
reconexiones = 0
|
||||
idx += 1
|
||||
continue
|
||||
|
||||
if motivo_exist not in ("no existe", "OK"):
|
||||
self._emit(
|
||||
EventType.WARNING,
|
||||
f"Corrupto ({motivo_exist}), re-descargando: {nit}/{numero}.pdf",
|
||||
radicado=numero,
|
||||
nit=nit,
|
||||
index=idx + 1,
|
||||
total=summary.total,
|
||||
)
|
||||
|
||||
self._emit(
|
||||
EventType.RADICADO_PROCESSING,
|
||||
f"Procesando: {numero} | NIT: {nit}",
|
||||
radicado=numero,
|
||||
nit=nit,
|
||||
index=idx + 1,
|
||||
total=summary.total,
|
||||
)
|
||||
|
||||
if not self.browser.is_alive():
|
||||
try:
|
||||
driver, wait = self.browser.reconnect(
|
||||
self.carpeta_descarga,
|
||||
reconexiones,
|
||||
self._log_text,
|
||||
)
|
||||
imaging.update_driver(driver, wait)
|
||||
reconexiones += 1
|
||||
self._emit(
|
||||
EventType.BROWSER_RECONNECTED,
|
||||
f"Navegador reconectado (intento {reconexiones})",
|
||||
metadata={"intento": reconexiones},
|
||||
)
|
||||
except RuntimeError as e:
|
||||
self._emit(EventType.RADICADO_FAILED, str(e))
|
||||
break
|
||||
continue
|
||||
|
||||
try:
|
||||
ok, motivo = imaging.search_and_download(
|
||||
numero,
|
||||
self.carpeta_descarga,
|
||||
ruta_final,
|
||||
self._log_text,
|
||||
)
|
||||
|
||||
if ok:
|
||||
if self.on_pdf_saved is not None:
|
||||
try:
|
||||
self.on_pdf_saved(nit, numero, ruta_final)
|
||||
except Exception as upload_err:
|
||||
self._emit(
|
||||
EventType.WARNING,
|
||||
f"Post-save falló: {upload_err}",
|
||||
radicado=numero,
|
||||
nit=nit,
|
||||
)
|
||||
self._emit(
|
||||
EventType.RADICADO_SAVED,
|
||||
f"Guardado: {nit}/{numero}.pdf",
|
||||
radicado=numero,
|
||||
nit=nit,
|
||||
index=idx + 1,
|
||||
total=summary.total,
|
||||
metadata={"ruta": f"{nit}/{numero}.pdf"},
|
||||
)
|
||||
summary.exitosos += 1
|
||||
stats.ok += 1
|
||||
else:
|
||||
self._emit(
|
||||
EventType.RADICADO_FAILED,
|
||||
f"{motivo}: {numero}",
|
||||
radicado=numero,
|
||||
nit=nit,
|
||||
index=idx + 1,
|
||||
total=summary.total,
|
||||
metadata={"motivo": motivo},
|
||||
)
|
||||
summary.fallidos.append(
|
||||
DownloadResult(
|
||||
radicado=numero,
|
||||
nit=nit,
|
||||
status=ResultStatus.FAILED,
|
||||
motivo=motivo,
|
||||
)
|
||||
)
|
||||
stats.fallidos += 1
|
||||
|
||||
reconexiones = 0
|
||||
idx += 1
|
||||
|
||||
except SESION_MUERTA as e:
|
||||
self._emit(
|
||||
EventType.WARNING,
|
||||
f"Sesión perdida: {type(e).__name__}",
|
||||
radicado=numero,
|
||||
nit=nit,
|
||||
index=idx + 1,
|
||||
total=summary.total,
|
||||
)
|
||||
try:
|
||||
driver, wait = self.browser.reconnect(
|
||||
self.carpeta_descarga,
|
||||
reconexiones,
|
||||
self._log_text,
|
||||
)
|
||||
imaging.update_driver(driver, wait)
|
||||
reconexiones += 1
|
||||
self._emit(
|
||||
EventType.BROWSER_RECONNECTED,
|
||||
f"Navegador reconectado (intento {reconexiones})",
|
||||
metadata={"intento": reconexiones},
|
||||
)
|
||||
except RuntimeError as re_err:
|
||||
self._emit(EventType.RADICADO_FAILED, str(re_err))
|
||||
break
|
||||
|
||||
except Exception as e:
|
||||
self._emit(
|
||||
EventType.RADICADO_FAILED,
|
||||
f"Error con {numero}: {e}",
|
||||
radicado=numero,
|
||||
nit=nit,
|
||||
index=idx + 1,
|
||||
total=summary.total,
|
||||
metadata={"motivo": str(e)},
|
||||
)
|
||||
summary.fallidos.append(
|
||||
DownloadResult(
|
||||
radicado=numero,
|
||||
nit=nit,
|
||||
status=ResultStatus.FAILED,
|
||||
motivo=str(e),
|
||||
)
|
||||
)
|
||||
stats.fallidos += 1
|
||||
reconexiones = 0
|
||||
idx += 1
|
||||
|
||||
if nit_actual is not None:
|
||||
prev = summary.stats_por_nit[nit_actual]
|
||||
self._emit(
|
||||
EventType.NIT_FINISHED,
|
||||
(
|
||||
f"FINALIZÓ NIT {nit_actual} | "
|
||||
f"OK: {prev.ok} | "
|
||||
f"Fallidos: {prev.fallidos} | "
|
||||
f"Ya existían: {prev.omitidos}"
|
||||
),
|
||||
nit=nit_actual,
|
||||
metadata={
|
||||
"ok": prev.ok,
|
||||
"fallidos": prev.fallidos,
|
||||
"omitidos": prev.omitidos,
|
||||
},
|
||||
)
|
||||
|
||||
finally:
|
||||
self.browser.close()
|
||||
|
||||
if summary.cancelled:
|
||||
finish_msg = (
|
||||
f"PROCESO CANCELADO — {summary.exitosos}/{summary.total} exitosos"
|
||||
)
|
||||
elif summary.fatal_error:
|
||||
finish_msg = f"PROCESO FALLIDO — {summary.fatal_error}"
|
||||
else:
|
||||
finish_msg = (
|
||||
f"PROCESO TERMINADO — {summary.exitosos}/{summary.total} exitosos"
|
||||
)
|
||||
|
||||
self._emit(
|
||||
EventType.RUN_FINISHED,
|
||||
finish_msg,
|
||||
metadata={
|
||||
"exitosos": summary.exitosos,
|
||||
"total": summary.total,
|
||||
"fallidos": len(summary.fallidos),
|
||||
"cancelled": summary.cancelled,
|
||||
},
|
||||
)
|
||||
|
||||
if summary.fallidos:
|
||||
self._emit(
|
||||
EventType.INFO,
|
||||
f"Fallidos ({len(summary.fallidos)}):",
|
||||
)
|
||||
for item in summary.fallidos:
|
||||
self._emit(
|
||||
EventType.INFO,
|
||||
f"NIT {item.nit} | {item.radicado}: {item.motivo}",
|
||||
radicado=item.radicado,
|
||||
nit=item.nit,
|
||||
)
|
||||
|
||||
self.bus.close()
|
||||
|
||||
return summary
|
||||
@@ -0,0 +1,25 @@
|
||||
"""Infraestructura: Excel, PDF, archivos, navegador e Imaging."""
|
||||
|
||||
from bot_descarga.infrastructure.excel_reader import ExcelReader, limpiar_nombre_carpeta
|
||||
from bot_descarga.infrastructure.pdf_validator import PdfValidator, esperar_pdf_completo
|
||||
from bot_descarga.infrastructure.file_manager import FileManager
|
||||
|
||||
__all__ = [
|
||||
"ExcelReader",
|
||||
"limpiar_nombre_carpeta",
|
||||
"PdfValidator",
|
||||
"esperar_pdf_completo",
|
||||
"FileManager",
|
||||
"BrowserManager",
|
||||
"ImagingClient",
|
||||
]
|
||||
|
||||
|
||||
def __getattr__(name: str):
|
||||
if name == "BrowserManager":
|
||||
from bot_descarga.infrastructure.browser import BrowserManager
|
||||
return BrowserManager
|
||||
if name == "ImagingClient":
|
||||
from bot_descarga.infrastructure.imaging_client import ImagingClient
|
||||
return ImagingClient
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||
@@ -0,0 +1,135 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from typing import Callable, Optional, Tuple
|
||||
|
||||
from selenium import webdriver
|
||||
from selenium.common.exceptions import (
|
||||
InvalidSessionIdException,
|
||||
NoSuchWindowException,
|
||||
)
|
||||
from selenium.webdriver.common.by import By
|
||||
from selenium.webdriver.support import expected_conditions as EC
|
||||
from selenium.webdriver.support.ui import Select, WebDriverWait
|
||||
|
||||
from bot_descarga.config.settings import Settings, DEFAULT_SETTINGS
|
||||
|
||||
SESION_MUERTA = (InvalidSessionIdException, NoSuchWindowException)
|
||||
|
||||
LogFunc = Callable[[str], None]
|
||||
|
||||
|
||||
class BrowserManager:
|
||||
"""Ciclo de vida del driver Chrome y sesión inicial en Imaging."""
|
||||
|
||||
def __init__(self, settings: Settings = DEFAULT_SETTINGS):
|
||||
self.settings = settings
|
||||
self.driver = None
|
||||
self.wait = None
|
||||
|
||||
def start(self, carpeta_descarga: str, log: LogFunc = print) -> Tuple:
|
||||
options = webdriver.ChromeOptions()
|
||||
|
||||
prefs = {
|
||||
"printing.print_preview_sticky_settings.appState":
|
||||
'{"recentDestinations":[{"id":"Save as PDF","origin":"local"}],'
|
||||
'"selectedDestinationId":"Save as PDF","version":2}',
|
||||
"savefile.default_directory": carpeta_descarga,
|
||||
"download.default_directory": carpeta_descarga,
|
||||
"download.prompt_for_download": False,
|
||||
"download.directory_upgrade": True,
|
||||
"plugins.always_open_pdf_externally": True,
|
||||
}
|
||||
|
||||
options.add_experimental_option("prefs", prefs)
|
||||
options.add_argument("--kiosk-printing")
|
||||
options.add_argument("--disable-infobars")
|
||||
options.add_argument("--disable-extensions")
|
||||
options.add_argument("--disable-gpu")
|
||||
options.add_argument("--no-sandbox")
|
||||
options.add_argument("--disable-dev-shm-usage")
|
||||
options.add_argument("--disable-background-networking")
|
||||
options.add_argument("--disable-default-apps")
|
||||
options.add_argument("--disable-sync")
|
||||
options.add_argument("--disable-translate")
|
||||
options.add_argument("--disable-features=TranslateUI")
|
||||
|
||||
self.driver = webdriver.Chrome(options=options)
|
||||
self.wait = WebDriverWait(self.driver, self.settings.wait_timeout)
|
||||
|
||||
self.driver.get(self.settings.url_sistema)
|
||||
self._configurar_sesion(log)
|
||||
|
||||
return self.driver, self.wait
|
||||
|
||||
def _configurar_sesion(self, log: LogFunc) -> None:
|
||||
try:
|
||||
self.wait.until(EC.element_to_be_clickable((By.ID, "search"))).click()
|
||||
log("Botón 'Buscar' clickeado")
|
||||
except Exception as e:
|
||||
log(f"No se pudo clicar 'Buscar': {e}")
|
||||
|
||||
try:
|
||||
grupo_el = self.wait.until(
|
||||
EC.presence_of_element_located((By.ID, "groupSelect"))
|
||||
)
|
||||
self.wait.until(
|
||||
lambda d: len(grupo_el.find_elements(By.TAG_NAME, "option")) > 1
|
||||
)
|
||||
Select(grupo_el).select_by_visible_text(self.settings.grupo)
|
||||
log(f"Grupo: {self.settings.grupo}")
|
||||
except Exception as e:
|
||||
log(f"Error groupSelect: {e}")
|
||||
|
||||
try:
|
||||
perfil_el = self.wait.until(
|
||||
EC.presence_of_element_located((By.ID, "profileSelect"))
|
||||
)
|
||||
self.wait.until(
|
||||
lambda d: len(perfil_el.find_elements(By.TAG_NAME, "option")) > 1
|
||||
)
|
||||
Select(perfil_el).select_by_visible_text(self.settings.perfil)
|
||||
log(f"Perfil: {self.settings.perfil} — esperando 2s...")
|
||||
time.sleep(2)
|
||||
except Exception as e:
|
||||
log(f"Error profileSelect: {e}")
|
||||
|
||||
def is_alive(self) -> bool:
|
||||
if self.driver is None:
|
||||
return False
|
||||
try:
|
||||
_ = self.driver.window_handles
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def reconnect(
|
||||
self,
|
||||
carpeta_descarga: str,
|
||||
n_intento: int,
|
||||
log: LogFunc = print,
|
||||
) -> Tuple:
|
||||
if n_intento >= self.settings.max_reconexiones:
|
||||
raise RuntimeError(
|
||||
f"Chrome crasheó {n_intento} veces seguidas. Abortando."
|
||||
)
|
||||
|
||||
self.close()
|
||||
|
||||
espera = 3 * (n_intento + 1)
|
||||
log(
|
||||
f"Chrome caído — reconectando "
|
||||
f"({n_intento + 1}/{self.settings.max_reconexiones}), espera {espera}s..."
|
||||
)
|
||||
time.sleep(espera)
|
||||
return self.start(carpeta_descarga, log)
|
||||
|
||||
def close(self) -> None:
|
||||
if self.driver is None:
|
||||
return
|
||||
try:
|
||||
self.driver.quit()
|
||||
except Exception:
|
||||
pass
|
||||
self.driver = None
|
||||
self.wait = None
|
||||
@@ -0,0 +1,45 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import List
|
||||
|
||||
from openpyxl import load_workbook
|
||||
|
||||
from bot_descarga.models.radicado import RadicadoRecord
|
||||
|
||||
_HEADER_NAMES = ("radicado", "proceso", "numero", "número")
|
||||
|
||||
|
||||
def limpiar_nombre_carpeta(valor) -> str:
|
||||
valor = str(valor).strip()
|
||||
valor = re.sub(r'[\\/:*?"<>|]', "_", valor)
|
||||
return valor if valor else "SIN_NIT"
|
||||
|
||||
|
||||
class ExcelReader:
|
||||
"""Lee radicados (col A) y NITs (col B) desde un Excel."""
|
||||
|
||||
def read(self, ruta_excel: str) -> List[RadicadoRecord]:
|
||||
wb = load_workbook(ruta_excel, data_only=True)
|
||||
ws = wb.active
|
||||
filas: List[RadicadoRecord] = []
|
||||
|
||||
for fila in ws.iter_rows(values_only=True):
|
||||
radicado = fila[0] if len(fila) > 0 else None
|
||||
nit = fila[1] if len(fila) > 1 else None
|
||||
|
||||
if not radicado or str(radicado).strip() in ("", "None"):
|
||||
continue
|
||||
|
||||
radicado = str(radicado).strip()
|
||||
|
||||
if radicado.lower() in _HEADER_NAMES:
|
||||
continue
|
||||
|
||||
if nit is None or str(nit).strip() in ("", "None"):
|
||||
nit = "SIN_NIT"
|
||||
|
||||
nit = limpiar_nombre_carpeta(nit)
|
||||
filas.append(RadicadoRecord(radicado=radicado, nit=nit))
|
||||
|
||||
return filas
|
||||
@@ -0,0 +1,60 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Tuple
|
||||
|
||||
from bot_descarga.infrastructure.pdf_validator import PdfValidator
|
||||
|
||||
|
||||
class FileManager:
|
||||
"""Gestión de carpetas por NIT y archivos PDF finales."""
|
||||
|
||||
def __init__(self, carpeta_descarga: str, validator: PdfValidator):
|
||||
self.carpeta_descarga = carpeta_descarga
|
||||
self.validator = validator
|
||||
os.makedirs(carpeta_descarga, exist_ok=True)
|
||||
|
||||
def ruta_final(self, nit: str, radicado: str) -> str:
|
||||
carpeta_nit = os.path.join(self.carpeta_descarga, nit)
|
||||
os.makedirs(carpeta_nit, exist_ok=True)
|
||||
return os.path.join(carpeta_nit, radicado + ".pdf")
|
||||
|
||||
def exists_valid(self, nit: str, radicado: str) -> Tuple[bool, str, str]:
|
||||
"""
|
||||
Returns (existe_y_valido, ruta_final, motivo).
|
||||
Si existe pero está corrupto, lo elimina.
|
||||
"""
|
||||
ruta = self.ruta_final(nit, radicado)
|
||||
|
||||
if not os.path.exists(ruta):
|
||||
return False, ruta, "no existe"
|
||||
|
||||
valido, motivo = self.validator.validate(ruta)
|
||||
if valido:
|
||||
return True, ruta, "OK"
|
||||
|
||||
try:
|
||||
os.remove(ruta)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return False, ruta, motivo
|
||||
|
||||
def move_to_final(self, archivo_temp: str, ruta_final: str) -> None:
|
||||
try:
|
||||
os.replace(archivo_temp, ruta_final)
|
||||
except Exception:
|
||||
os.rename(archivo_temp, ruta_final)
|
||||
|
||||
def remove_safe(self, ruta: str) -> None:
|
||||
try:
|
||||
if ruta and os.path.exists(ruta):
|
||||
os.remove(ruta)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def list_pdfs(self) -> set:
|
||||
return {
|
||||
f for f in os.listdir(self.carpeta_descarga)
|
||||
if f.lower().endswith(".pdf")
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
from typing import Callable, Optional, Tuple
|
||||
|
||||
from selenium.webdriver.common.action_chains import ActionChains
|
||||
from selenium.webdriver.common.by import By
|
||||
from selenium.webdriver.support import expected_conditions as EC
|
||||
from selenium.webdriver.support.ui import WebDriverWait
|
||||
|
||||
from bot_descarga.config.settings import Settings, DEFAULT_SETTINGS
|
||||
from bot_descarga.infrastructure.browser import SESION_MUERTA
|
||||
from bot_descarga.infrastructure.file_manager import FileManager
|
||||
from bot_descarga.infrastructure.pdf_validator import PdfValidator, esperar_pdf_completo
|
||||
|
||||
LogFunc = Callable[[str], None]
|
||||
|
||||
|
||||
class ImagingClient:
|
||||
"""Interacciones con el portal Imaging: búsqueda, factura y descarga."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
driver,
|
||||
wait: WebDriverWait,
|
||||
settings: Settings = DEFAULT_SETTINGS,
|
||||
validator: Optional[PdfValidator] = None,
|
||||
files: Optional[FileManager] = None,
|
||||
):
|
||||
self.driver = driver
|
||||
self.wait = wait
|
||||
self.settings = settings
|
||||
self.validator = validator or PdfValidator(settings)
|
||||
self.files = files
|
||||
|
||||
def update_driver(self, driver, wait: WebDriverWait) -> None:
|
||||
self.driver = driver
|
||||
self.wait = wait
|
||||
|
||||
def esperar_documento_listo(self, log: LogFunc = print) -> bool:
|
||||
try:
|
||||
WebDriverWait(self.driver, 8).until(
|
||||
lambda d: d.execute_script("return document.readyState") == "complete"
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
WebDriverWait(self.driver, 8).until(
|
||||
EC.invisibility_of_element_located((By.ID, "imagingSearchOverlay"))
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
WebDriverWait(self.driver, 8).until(
|
||||
EC.presence_of_element_located((By.CLASS_NAME, "document"))
|
||||
)
|
||||
return True
|
||||
except Exception:
|
||||
log("No se encontraron documentos en pantalla")
|
||||
return False
|
||||
|
||||
def esperar_visor_pdf(self) -> bool:
|
||||
try:
|
||||
WebDriverWait(self.driver, 8).until(
|
||||
lambda d: d.execute_script("return document.readyState") == "complete"
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(0.8)
|
||||
return True
|
||||
|
||||
def buscar_factura_en_documentos(self, log: LogFunc = print):
|
||||
docs = self.driver.find_elements(By.CLASS_NAME, "document")
|
||||
|
||||
log("Documentos encontrados:")
|
||||
for d in docs:
|
||||
log(f" - {d.get_attribute('title')}")
|
||||
|
||||
for d in docs:
|
||||
titulo = d.get_attribute("title") or ""
|
||||
titulo_limpio = titulo.lower().strip()
|
||||
if "factura" in titulo_limpio and titulo_limpio.endswith(".pdf"):
|
||||
return d
|
||||
|
||||
return None
|
||||
|
||||
def search(self, numero: str) -> None:
|
||||
self.driver.switch_to.window(self.driver.window_handles[0])
|
||||
|
||||
self.wait.until(EC.element_to_be_clickable((By.ID, "search"))).click()
|
||||
|
||||
campo = self.wait.until(
|
||||
EC.presence_of_element_located((By.ID, "searchValue3"))
|
||||
)
|
||||
campo.clear()
|
||||
campo.send_keys(numero)
|
||||
|
||||
self.wait.until(
|
||||
EC.element_to_be_clickable((By.ID, "searchButton"))
|
||||
).click()
|
||||
|
||||
def click_factura(self, factura) -> bool:
|
||||
for _ in range(self.settings.max_intentos_click):
|
||||
try:
|
||||
ActionChains(self.driver).move_to_element(factura).click().perform()
|
||||
return True
|
||||
except SESION_MUERTA:
|
||||
raise
|
||||
except Exception:
|
||||
time.sleep(0.4)
|
||||
return False
|
||||
|
||||
def download_pdf(
|
||||
self,
|
||||
carpeta_descarga: str,
|
||||
ruta_final: str,
|
||||
log: LogFunc = print,
|
||||
) -> Tuple[bool, str]:
|
||||
"""Ejecuta window.print y valida el PDF. Returns (ok, motivo)."""
|
||||
for intento in range(self.settings.max_intentos_pdf):
|
||||
archivos_antes = {
|
||||
f for f in os.listdir(carpeta_descarga)
|
||||
if f.lower().endswith(".pdf")
|
||||
}
|
||||
|
||||
self.driver.execute_script("window.print();")
|
||||
|
||||
archivo_temp = esperar_pdf_completo(
|
||||
carpeta_descarga,
|
||||
archivos_antes,
|
||||
timeout=self.settings.timeout_descarga,
|
||||
)
|
||||
|
||||
if not archivo_temp or not os.path.exists(archivo_temp):
|
||||
log(
|
||||
f"Intento {intento + 1}/{self.settings.max_intentos_pdf}: sin archivo"
|
||||
)
|
||||
time.sleep(0.5)
|
||||
continue
|
||||
|
||||
valido, motivo = self.validator.validate(archivo_temp)
|
||||
|
||||
if valido:
|
||||
if self.files is not None:
|
||||
self.files.move_to_final(archivo_temp, ruta_final)
|
||||
else:
|
||||
try:
|
||||
os.replace(archivo_temp, ruta_final)
|
||||
except Exception:
|
||||
os.rename(archivo_temp, ruta_final)
|
||||
return True, "OK"
|
||||
|
||||
log(
|
||||
f"Intento {intento + 1}/{self.settings.max_intentos_pdf}: "
|
||||
f"inválido → {motivo}"
|
||||
)
|
||||
try:
|
||||
os.remove(archivo_temp)
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(0.8)
|
||||
|
||||
return False, "PDF inválido repetido"
|
||||
|
||||
def search_and_download(
|
||||
self,
|
||||
numero: str,
|
||||
carpeta_descarga: str,
|
||||
ruta_final: str,
|
||||
log: LogFunc = print,
|
||||
) -> Tuple[bool, str]:
|
||||
"""
|
||||
Flujo completo: buscar → localizar factura → clic → print → guardar.
|
||||
Returns (ok, motivo_error).
|
||||
"""
|
||||
self.search(numero)
|
||||
|
||||
if not self.esperar_documento_listo(log):
|
||||
return False, "Página no cargó"
|
||||
|
||||
factura = self.buscar_factura_en_documentos(log)
|
||||
if not factura:
|
||||
return False, "Sin factura detectada"
|
||||
|
||||
if not self.click_factura(factura):
|
||||
return False, "Click fallido"
|
||||
|
||||
log("Esperando visor...")
|
||||
self.esperar_visor_pdf()
|
||||
|
||||
return self.download_pdf(carpeta_descarga, ruta_final, log)
|
||||
@@ -0,0 +1,102 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
from typing import Optional, Set, Tuple
|
||||
|
||||
import pikepdf
|
||||
|
||||
from bot_descarga.config.settings import Settings, DEFAULT_SETTINGS
|
||||
|
||||
|
||||
def _validar_firma(ruta: str) -> bool:
|
||||
try:
|
||||
with open(ruta, "rb") as f:
|
||||
return f.read(5) == b"%PDF-"
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _validar_eof(ruta: str) -> bool:
|
||||
try:
|
||||
with open(ruta, "rb") as f:
|
||||
f.seek(-1024, 2)
|
||||
return b"%%EOF" in f.read()
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _validar_pikepdf(ruta: str) -> bool:
|
||||
try:
|
||||
with pikepdf.open(ruta) as pdf:
|
||||
_ = len(pdf.pages)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
class PdfValidator:
|
||||
def __init__(self, settings: Settings = DEFAULT_SETTINGS):
|
||||
self.settings = settings
|
||||
|
||||
def validate(self, ruta: str) -> Tuple[bool, str]:
|
||||
try:
|
||||
peso = os.path.getsize(ruta)
|
||||
|
||||
if peso < self.settings.peso_minimo_bytes:
|
||||
return False, f"Peso insuficiente ({peso} bytes)"
|
||||
|
||||
if not _validar_firma(ruta):
|
||||
return False, "Sin firma %PDF-"
|
||||
|
||||
if not _validar_eof(ruta):
|
||||
return False, "Sin %%EOF (truncado)"
|
||||
|
||||
if not _validar_pikepdf(ruta):
|
||||
return False, "pikepdf: estructura corrupta"
|
||||
|
||||
return True, "OK"
|
||||
|
||||
except Exception as e:
|
||||
return False, str(e)
|
||||
|
||||
|
||||
def esperar_pdf_completo(
|
||||
carpeta: str,
|
||||
archivos_antes: Set[str],
|
||||
timeout: int = 35,
|
||||
) -> Optional[str]:
|
||||
inicio = time.time()
|
||||
tamaño_anterior = -1
|
||||
estable = 0
|
||||
|
||||
while time.time() - inicio < timeout:
|
||||
actuales = {
|
||||
f for f in os.listdir(carpeta)
|
||||
if f.lower().endswith(".pdf")
|
||||
}
|
||||
|
||||
nuevos = actuales - archivos_antes
|
||||
|
||||
if nuevos:
|
||||
ruta = os.path.join(carpeta, list(nuevos)[0])
|
||||
|
||||
if not os.path.exists(ruta):
|
||||
time.sleep(0.2)
|
||||
continue
|
||||
|
||||
tamaño = os.path.getsize(ruta)
|
||||
|
||||
if tamaño == tamaño_anterior and tamaño > 1000:
|
||||
estable += 1
|
||||
else:
|
||||
estable = 0
|
||||
|
||||
tamaño_anterior = tamaño
|
||||
|
||||
if estable >= 2:
|
||||
return ruta
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
return None
|
||||
@@ -0,0 +1,4 @@
|
||||
from bot_descarga.models.radicado import RadicadoRecord
|
||||
from bot_descarga.models.resultado import DownloadResult, NitStats, RunSummary
|
||||
|
||||
__all__ = ["RadicadoRecord", "DownloadResult", "NitStats", "RunSummary"]
|
||||
@@ -0,0 +1,7 @@
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RadicadoRecord:
|
||||
radicado: str
|
||||
nit: str
|
||||
@@ -0,0 +1,36 @@
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
from typing import List, Optional
|
||||
|
||||
|
||||
class ResultStatus(str, Enum):
|
||||
SAVED = "saved"
|
||||
SKIPPED = "skipped"
|
||||
FAILED = "failed"
|
||||
CANCELLED = "cancelled"
|
||||
|
||||
|
||||
@dataclass
|
||||
class DownloadResult:
|
||||
radicado: str
|
||||
nit: str
|
||||
status: ResultStatus
|
||||
motivo: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class NitStats:
|
||||
total: int = 0
|
||||
ok: int = 0
|
||||
fallidos: int = 0
|
||||
omitidos: int = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
class RunSummary:
|
||||
total: int = 0
|
||||
exitosos: int = 0
|
||||
cancelled: bool = False
|
||||
fatal_error: Optional[str] = None
|
||||
fallidos: List[DownloadResult] = field(default_factory=list)
|
||||
stats_por_nit: dict = field(default_factory=dict)
|
||||
@@ -0,0 +1,4 @@
|
||||
from bot_descarga.ui.dashboard import BotDashboard
|
||||
from bot_descarga.ui.settings_store import DEFAULT_API_URL, normalize_api_url
|
||||
|
||||
__all__ = ["BotDashboard", "DEFAULT_API_URL", "normalize_api_url"]
|
||||
@@ -0,0 +1,268 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
import threading
|
||||
|
||||
import customtkinter as ctk
|
||||
|
||||
from bot_descarga.ui.settings_store import (
|
||||
DEFAULT_API_URL,
|
||||
load_gui_settings,
|
||||
normalize_api_url,
|
||||
save_gui_settings,
|
||||
)
|
||||
from bot_descarga.worker.agent import WorkerAgent
|
||||
|
||||
ctk.set_appearance_mode("dark")
|
||||
ctk.set_default_color_theme("blue")
|
||||
|
||||
|
||||
class BotDashboard(ctk.CTk):
|
||||
"""GUI del agent: configura API remota y conecta como worker idle."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
|
||||
self.title("Bot de Descargas PDF — Agent")
|
||||
self.geometry("920x680")
|
||||
self.resizable(False, False)
|
||||
|
||||
self._agent: WorkerAgent | None = None
|
||||
self._agent_thread: threading.Thread | None = None
|
||||
self._connected = False
|
||||
|
||||
self._construir_ui()
|
||||
self._cargar_settings()
|
||||
|
||||
self.protocol("WM_DELETE_WINDOW", self._on_close)
|
||||
|
||||
def _construir_ui(self):
|
||||
root = ctk.CTkFrame(self, corner_radius=18)
|
||||
root.pack(fill="both", expand=True, padx=18, pady=18)
|
||||
|
||||
ctk.CTkLabel(
|
||||
root,
|
||||
text="Bot de Descargas PDF",
|
||||
font=ctk.CTkFont(size=26, weight="bold"),
|
||||
).pack(pady=(18, 4))
|
||||
|
||||
ctk.CTkLabel(
|
||||
root,
|
||||
text="Agent remoto — idle hasta que el desktop asigne un Excel a este bot",
|
||||
text_color="gray",
|
||||
font=ctk.CTkFont(size=13),
|
||||
).pack(pady=(0, 18))
|
||||
|
||||
config = ctk.CTkFrame(root, corner_radius=14)
|
||||
config.pack(fill="x", padx=24, pady=(0, 14))
|
||||
|
||||
ctk.CTkLabel(
|
||||
config,
|
||||
text="API remota (IPSExtractor)",
|
||||
font=ctk.CTkFont(size=16, weight="bold"),
|
||||
).pack(anchor="w", padx=18, pady=(14, 8))
|
||||
|
||||
row_api = ctk.CTkFrame(config, fg_color="transparent")
|
||||
row_api.pack(fill="x", padx=18, pady=(0, 8))
|
||||
ctk.CTkLabel(row_api, text="URL API", width=90, anchor="w").pack(side="left")
|
||||
self.entry_api = ctk.CTkEntry(
|
||||
row_api,
|
||||
placeholder_text=DEFAULT_API_URL,
|
||||
)
|
||||
self.entry_api.pack(side="left", fill="x", expand=True)
|
||||
|
||||
row_id = ctk.CTkFrame(config, fg_color="transparent")
|
||||
row_id.pack(fill="x", padx=18, pady=(0, 8))
|
||||
ctk.CTkLabel(row_id, text="Worker ID", width=90, anchor="w").pack(side="left")
|
||||
self.entry_worker = ctk.CTkEntry(
|
||||
row_id,
|
||||
placeholder_text=socket.gethostname() or "PC-01",
|
||||
)
|
||||
self.entry_worker.pack(side="left", fill="x", expand=True)
|
||||
|
||||
row_token = ctk.CTkFrame(config, fg_color="transparent")
|
||||
row_token.pack(fill="x", padx=18, pady=(0, 14))
|
||||
ctk.CTkLabel(row_token, text="Token", width=90, anchor="w").pack(side="left")
|
||||
self.entry_token = ctk.CTkEntry(
|
||||
row_token,
|
||||
placeholder_text="BOT_API_TOKEN",
|
||||
show="•",
|
||||
)
|
||||
self.entry_token.pack(side="left", fill="x", expand=True, padx=(0, 8))
|
||||
self._token_visible = False
|
||||
ctk.CTkButton(
|
||||
row_token,
|
||||
text="👁",
|
||||
width=40,
|
||||
command=self._toggle_token,
|
||||
).pack(side="left")
|
||||
|
||||
status = ctk.CTkFrame(root, corner_radius=14)
|
||||
status.pack(fill="x", padx=24, pady=(0, 14))
|
||||
|
||||
self.lbl_estado = ctk.CTkLabel(
|
||||
status,
|
||||
text="Estado: desconectado",
|
||||
font=ctk.CTkFont(size=15, weight="bold"),
|
||||
)
|
||||
self.lbl_estado.pack(anchor="w", padx=18, pady=(14, 2))
|
||||
|
||||
self.lbl_hint = ctk.CTkLabel(
|
||||
status,
|
||||
text="Configura la API y pulsa Conectar. Este PC no expone puertos.",
|
||||
text_color="gray",
|
||||
font=ctk.CTkFont(size=13),
|
||||
)
|
||||
self.lbl_hint.pack(anchor="w", padx=18, pady=(0, 14))
|
||||
|
||||
botones = ctk.CTkFrame(root, fg_color="transparent")
|
||||
botones.pack(fill="x", padx=24, pady=(0, 14))
|
||||
|
||||
self.btn_connect = ctk.CTkButton(
|
||||
botones,
|
||||
text="▶ Conectar agent",
|
||||
height=42,
|
||||
width=180,
|
||||
font=ctk.CTkFont(size=14, weight="bold"),
|
||||
command=self._toggle_connect,
|
||||
)
|
||||
self.btn_connect.pack(side="left", padx=(0, 10))
|
||||
|
||||
ctk.CTkButton(
|
||||
botones,
|
||||
text="Guardar config",
|
||||
height=42,
|
||||
width=140,
|
||||
command=self._guardar_settings,
|
||||
).pack(side="left", padx=(0, 10))
|
||||
|
||||
ctk.CTkButton(
|
||||
botones,
|
||||
text="Limpiar logs",
|
||||
height=42,
|
||||
width=130,
|
||||
command=self._limpiar_logs,
|
||||
).pack(side="left")
|
||||
|
||||
logs_frame = ctk.CTkFrame(root, corner_radius=14)
|
||||
logs_frame.pack(fill="both", expand=True, padx=24, pady=(0, 18))
|
||||
|
||||
ctk.CTkLabel(
|
||||
logs_frame,
|
||||
text="Detalle del agent",
|
||||
font=ctk.CTkFont(size=15, weight="bold"),
|
||||
).pack(anchor="w", padx=18, pady=(12, 6))
|
||||
|
||||
self.text_logs = ctk.CTkTextbox(
|
||||
logs_frame, font=ctk.CTkFont(family="Courier", size=12)
|
||||
)
|
||||
self.text_logs.pack(fill="both", expand=True, padx=18, pady=(0, 16))
|
||||
|
||||
def _cargar_settings(self):
|
||||
s = load_gui_settings()
|
||||
self.entry_api.insert(0, s.get("api_url") or DEFAULT_API_URL)
|
||||
self.entry_worker.insert(0, s.get("worker_id") or socket.gethostname())
|
||||
self.entry_token.insert(0, s.get("token") or "")
|
||||
|
||||
def _guardar_settings(self):
|
||||
api = normalize_api_url(self.entry_api.get())
|
||||
self.entry_api.delete(0, "end")
|
||||
self.entry_api.insert(0, api)
|
||||
save_gui_settings(api, self.entry_worker.get(), self.entry_token.get())
|
||||
self._log("Configuración guardada en data/gui_settings.json")
|
||||
|
||||
def _toggle_token(self):
|
||||
self._token_visible = not self._token_visible
|
||||
self.entry_token.configure(show="" if self._token_visible else "•")
|
||||
|
||||
def _toggle_connect(self):
|
||||
if self._connected:
|
||||
self._desconectar()
|
||||
else:
|
||||
self._conectar()
|
||||
|
||||
def _conectar(self):
|
||||
api = normalize_api_url(self.entry_api.get())
|
||||
worker_id = self.entry_worker.get().strip()
|
||||
token = self.entry_token.get().strip()
|
||||
|
||||
if not worker_id:
|
||||
self._log("Debes indicar un Worker ID.")
|
||||
return
|
||||
if not token:
|
||||
self._log("Debes indicar el token (BOT_API_TOKEN).")
|
||||
return
|
||||
|
||||
self.entry_api.delete(0, "end")
|
||||
self.entry_api.insert(0, api)
|
||||
save_gui_settings(api, worker_id, token)
|
||||
|
||||
self._agent = WorkerAgent(
|
||||
api_url=api,
|
||||
worker_id=worker_id,
|
||||
token=token,
|
||||
log_func=self._log,
|
||||
)
|
||||
self._connected = True
|
||||
self.btn_connect.configure(text="⏹ Desconectar", fg_color="#c0392b", hover_color="#a93226")
|
||||
self.lbl_estado.configure(text=f"Estado: conectado como {worker_id}")
|
||||
self.lbl_hint.configure(text=f"API: {api} — esperando jobs del desktop…")
|
||||
self._set_config_enabled(False)
|
||||
|
||||
self._agent_thread = threading.Thread(
|
||||
target=self._run_agent,
|
||||
daemon=True,
|
||||
)
|
||||
self._agent_thread.start()
|
||||
self._log(f"Conectando a {api}…")
|
||||
|
||||
def _run_agent(self):
|
||||
try:
|
||||
assert self._agent is not None
|
||||
self._agent.run_forever()
|
||||
except Exception as e:
|
||||
self._log(f"Error del agent: {e}")
|
||||
finally:
|
||||
self.after(0, self._on_agent_stopped)
|
||||
|
||||
def _desconectar(self):
|
||||
if self._agent:
|
||||
self._agent.stop()
|
||||
self._log("Solicitando desconexión…")
|
||||
self.btn_connect.configure(state="disabled", text="Desconectando…")
|
||||
|
||||
def _on_agent_stopped(self):
|
||||
self._connected = False
|
||||
self._agent = None
|
||||
self.btn_connect.configure(
|
||||
state="normal",
|
||||
text="▶ Conectar agent",
|
||||
fg_color=["#3B8ED0", "#1F6AA5"],
|
||||
hover_color=["#36719F", "#144870"],
|
||||
)
|
||||
self.lbl_estado.configure(text="Estado: desconectado")
|
||||
self.lbl_hint.configure(
|
||||
text="Configura la API y pulsa Conectar. Este PC no expone puertos."
|
||||
)
|
||||
self._set_config_enabled(True)
|
||||
|
||||
def _set_config_enabled(self, enabled: bool):
|
||||
state = "normal" if enabled else "disabled"
|
||||
self.entry_api.configure(state=state)
|
||||
self.entry_worker.configure(state=state)
|
||||
self.entry_token.configure(state=state)
|
||||
|
||||
def _limpiar_logs(self):
|
||||
self.text_logs.delete("1.0", "end")
|
||||
|
||||
def _log(self, mensaje: str):
|
||||
def _escribir():
|
||||
self.text_logs.insert("end", mensaje + "\n")
|
||||
self.text_logs.see("end")
|
||||
|
||||
self.after(0, _escribir)
|
||||
|
||||
def _on_close(self):
|
||||
if self._agent:
|
||||
self._agent.stop()
|
||||
self.destroy()
|
||||
@@ -0,0 +1,65 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from bot_descarga.audit.events import BotEvent, EventType
|
||||
|
||||
|
||||
class ProgressTracker:
|
||||
"""Traduce BotEvent a contadores de progreso para la UI."""
|
||||
|
||||
def __init__(self):
|
||||
self.total = 0
|
||||
self.procesados = 0
|
||||
self.exitosos = 0
|
||||
self.fallidos = 0
|
||||
self.skipped = 0
|
||||
self.nit_actual = "—"
|
||||
self.nit_estado = "Esperando inicio del proceso..."
|
||||
|
||||
def reset(self) -> None:
|
||||
self.total = 0
|
||||
self.procesados = 0
|
||||
self.exitosos = 0
|
||||
self.fallidos = 0
|
||||
self.skipped = 0
|
||||
self.nit_actual = "—"
|
||||
self.nit_estado = "Esperando inicio del proceso..."
|
||||
|
||||
def handle(self, event: BotEvent) -> None:
|
||||
if event.index is not None and event.total is not None:
|
||||
self.procesados = event.index
|
||||
self.total = event.total
|
||||
|
||||
if event.event_type == EventType.RUN_STARTED:
|
||||
if "total" in event.metadata:
|
||||
self.total = int(event.metadata["total"])
|
||||
elif event.total is not None:
|
||||
self.total = event.total
|
||||
|
||||
elif event.event_type == EventType.NIT_STARTED:
|
||||
self.nit_actual = event.nit or "—"
|
||||
self.nit_estado = "Procesando radicados de este NIT..."
|
||||
|
||||
elif event.event_type == EventType.NIT_FINISHED:
|
||||
nit = event.nit or ""
|
||||
meta = event.metadata
|
||||
self.nit_estado = (
|
||||
f"Finalizó NIT {nit} | "
|
||||
f"OK: {meta.get('ok', 0)} | "
|
||||
f"Fallidos: {meta.get('fallidos', 0)} | "
|
||||
f"Ya existían: {meta.get('omitidos', 0)}"
|
||||
)
|
||||
|
||||
elif event.event_type == EventType.RADICADO_SAVED:
|
||||
self.exitosos += 1
|
||||
|
||||
elif event.event_type == EventType.RADICADO_SKIPPED:
|
||||
self.skipped += 1
|
||||
|
||||
elif event.event_type == EventType.RADICADO_FAILED:
|
||||
if "PROCESO TERMINADO" not in event.message:
|
||||
self.fallidos += 1
|
||||
|
||||
@property
|
||||
def percent(self) -> float:
|
||||
total = self.total if self.total > 0 else 1
|
||||
return min(self.procesados / total, 1.0)
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Persistencia simple de configuración de la GUI del worker."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
DEFAULT_API_URL = os.getenv(
|
||||
"BOT_API_URL",
|
||||
"https://pys1.builker.xyz/bot-descarga",
|
||||
)
|
||||
DEFAULT_TOKEN = os.getenv("BOT_API_TOKEN", "dev-token-change-me")
|
||||
|
||||
_SETTINGS_PATH = Path("data") / "gui_settings.json"
|
||||
|
||||
|
||||
def load_gui_settings() -> Dict[str, Any]:
|
||||
defaults = {
|
||||
"api_url": DEFAULT_API_URL,
|
||||
"worker_id": "",
|
||||
"token": DEFAULT_TOKEN,
|
||||
}
|
||||
if not _SETTINGS_PATH.exists():
|
||||
return defaults
|
||||
try:
|
||||
data = json.loads(_SETTINGS_PATH.read_text(encoding="utf-8"))
|
||||
if not isinstance(data, dict):
|
||||
return defaults
|
||||
return {**defaults, **data}
|
||||
except Exception:
|
||||
return defaults
|
||||
|
||||
|
||||
def save_gui_settings(api_url: str, worker_id: str, token: str) -> None:
|
||||
_SETTINGS_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
payload = {
|
||||
"api_url": api_url.strip(),
|
||||
"worker_id": worker_id.strip(),
|
||||
"token": token.strip(),
|
||||
}
|
||||
_SETTINGS_PATH.write_text(
|
||||
json.dumps(payload, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def normalize_api_url(url: str) -> str:
|
||||
"""Normaliza la URL base del API bot-descarga."""
|
||||
u = (url or "").strip().rstrip("/")
|
||||
if not u:
|
||||
return DEFAULT_API_URL.rstrip("/")
|
||||
# Si el usuario pone solo el host del gateway, añade el prefijo
|
||||
if u.endswith("/bot-descarga"):
|
||||
return u
|
||||
if "builker.xyz" in u or u.endswith(":8189"):
|
||||
return f"{u}/bot-descarga"
|
||||
return u
|
||||
@@ -0,0 +1,3 @@
|
||||
from bot_descarga.worker.agent import WorkerAgent
|
||||
|
||||
__all__ = ["WorkerAgent"]
|
||||
@@ -0,0 +1,6 @@
|
||||
"""Permite: python -m bot_descarga.worker.run"""
|
||||
|
||||
from bot_descarga.worker.run import main
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,255 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import httpx
|
||||
|
||||
from bot_descarga.core.download_service import DownloadService
|
||||
from bot_descarga.worker.minio_upload import MinioSettings, WorkerMinioUploader
|
||||
|
||||
JOB_COMPLETED = "completed"
|
||||
JOB_FAILED = "failed"
|
||||
JOB_CANCELLED = "cancelled"
|
||||
|
||||
|
||||
class RemoteApiClient:
|
||||
"""Cliente HTTP hacia la API en IPSExtractor (vía gateway). Sin puertos locales."""
|
||||
|
||||
def __init__(self, base_url: str, token: str):
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.token = token
|
||||
self._client = httpx.Client(
|
||||
timeout=httpx.Timeout(60.0, connect=10.0),
|
||||
headers={"Authorization": f"Bearer {token}"},
|
||||
)
|
||||
|
||||
def close(self) -> None:
|
||||
self._client.close()
|
||||
|
||||
def _url(self, path: str) -> str:
|
||||
return f"{self.base_url}{path}"
|
||||
|
||||
def register(self, worker_id: str, hostname: str = "", max_concurrent: int = 1) -> dict:
|
||||
r = self._client.post(
|
||||
self._url("/workers/register"),
|
||||
json={
|
||||
"worker_id": worker_id,
|
||||
"hostname": hostname or socket.gethostname(),
|
||||
"max_concurrent": max_concurrent,
|
||||
},
|
||||
)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
def heartbeat(self, worker_id: str) -> dict:
|
||||
r = self._client.post(
|
||||
self._url("/workers/heartbeat"),
|
||||
json={"worker_id": worker_id},
|
||||
)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
def claim_pending(self, worker_id: str) -> Optional[dict]:
|
||||
r = self._client.get(
|
||||
self._url("/workers/me/pending"),
|
||||
params={"worker_id": worker_id},
|
||||
)
|
||||
r.raise_for_status()
|
||||
data = r.json()
|
||||
return data.get("job")
|
||||
|
||||
def download_excel(self, job_id: str, dest: Path) -> Path:
|
||||
r = self._client.get(self._url(f"/jobs/{job_id}/file"))
|
||||
r.raise_for_status()
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_bytes(r.content)
|
||||
return dest
|
||||
|
||||
def post_event(self, job_id: str, event: Dict[str, Any]) -> None:
|
||||
r = self._client.post(self._url(f"/jobs/{job_id}/events"), json=event)
|
||||
r.raise_for_status()
|
||||
|
||||
def cancel_flag(self, job_id: str) -> bool:
|
||||
r = self._client.get(self._url(f"/jobs/{job_id}/cancel-flag"))
|
||||
r.raise_for_status()
|
||||
return bool(r.json().get("cancel_requested"))
|
||||
|
||||
def patch_job(self, job_id: str, **body) -> dict:
|
||||
r = self._client.patch(self._url(f"/jobs/{job_id}"), json=body)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
|
||||
class WorkerAgent:
|
||||
"""Agent idle: heartbeat + claim solo jobs asignados por el usuario."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
api_url: str,
|
||||
worker_id: str,
|
||||
token: str,
|
||||
poll_seconds: float = 8.0,
|
||||
staging_root: Optional[Path] = None,
|
||||
log_func=None,
|
||||
):
|
||||
self.api = RemoteApiClient(api_url, token)
|
||||
self.worker_id = worker_id
|
||||
self.poll_seconds = poll_seconds
|
||||
self.staging_root = staging_root or Path(tempfile.gettempdir()) / "bot_descarga"
|
||||
self.minio = WorkerMinioUploader(MinioSettings.from_env())
|
||||
self._running = True
|
||||
self._uploaded_keys: List[str] = []
|
||||
self._log = log_func or print
|
||||
|
||||
def stop(self) -> None:
|
||||
self._running = False
|
||||
|
||||
def run_forever(self) -> None:
|
||||
self.api.register(self.worker_id)
|
||||
self._log(f"[worker {self.worker_id}] registrado — idle (esperando jobs manuales)")
|
||||
try:
|
||||
while self._running:
|
||||
try:
|
||||
self.api.heartbeat(self.worker_id)
|
||||
job = self.api.claim_pending(self.worker_id)
|
||||
if job:
|
||||
self._run_job(job)
|
||||
else:
|
||||
time.sleep(self.poll_seconds)
|
||||
except KeyboardInterrupt:
|
||||
raise
|
||||
except Exception as e:
|
||||
self._log(f"[worker {self.worker_id}] error en loop: {e}")
|
||||
time.sleep(self.poll_seconds)
|
||||
finally:
|
||||
self.api.close()
|
||||
self._log(f"[worker {self.worker_id}] detenido")
|
||||
|
||||
def _run_job(self, job: dict) -> None:
|
||||
job_id = job["id"]
|
||||
self._uploaded_keys = []
|
||||
work_dir = self.staging_root / job_id
|
||||
out_dir = work_dir / "out"
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
excel_path = work_dir / (Path(job.get("excel_key") or "input.xlsx").name)
|
||||
|
||||
self._log(f"[worker {self.worker_id}] job {job_id} iniciado")
|
||||
|
||||
try:
|
||||
self.api.download_excel(job_id, excel_path)
|
||||
|
||||
def should_cancel() -> bool:
|
||||
try:
|
||||
return self.api.cancel_flag(job_id)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def on_event(event) -> None:
|
||||
self._log(event.display_message())
|
||||
payload = {
|
||||
"event_type": event.event_type.value
|
||||
if hasattr(event.event_type, "value")
|
||||
else str(event.event_type),
|
||||
"message": event.message,
|
||||
"radicado": event.radicado,
|
||||
"nit": event.nit,
|
||||
"index": event.index,
|
||||
"total": event.total,
|
||||
"metadata": event.metadata,
|
||||
"timestamp": event.timestamp,
|
||||
}
|
||||
try:
|
||||
self.api.post_event(job_id, payload)
|
||||
except Exception as e:
|
||||
self._log(f"[worker] no se pudo enviar evento: {e}")
|
||||
|
||||
def on_pdf_saved(nit: str, radicado: str, local_path: str) -> None:
|
||||
key = self.minio.upload_pdf(job_id, nit, radicado, local_path)
|
||||
self._uploaded_keys.append(key)
|
||||
|
||||
service = DownloadService(
|
||||
ruta_excel=str(excel_path),
|
||||
carpeta_descarga=str(out_dir),
|
||||
event_callback=on_event,
|
||||
should_cancel=should_cancel,
|
||||
on_pdf_saved=on_pdf_saved,
|
||||
)
|
||||
summary = service.run()
|
||||
|
||||
try:
|
||||
self.minio.write_manifest(job_id, self._uploaded_keys)
|
||||
except Exception as e:
|
||||
self._log(f"[worker] manifest no escrito: {e}")
|
||||
|
||||
fallidos_count = len(summary.fallidos)
|
||||
if summary.fatal_error and fallidos_count == 0 and summary.total:
|
||||
fallidos_count = summary.total - summary.exitosos
|
||||
|
||||
result = {
|
||||
"total": summary.total,
|
||||
"exitosos": summary.exitosos,
|
||||
"fallidos": fallidos_count,
|
||||
"cancelled": summary.cancelled,
|
||||
"downloads_prefix": job.get("downloads_prefix"),
|
||||
"uploaded_count": len(self._uploaded_keys),
|
||||
}
|
||||
|
||||
if summary.cancelled:
|
||||
status = JOB_CANCELLED
|
||||
error_msg = None
|
||||
elif summary.fatal_error:
|
||||
status = JOB_FAILED
|
||||
error_msg = summary.fatal_error
|
||||
elif summary.fallidos and summary.exitosos == 0:
|
||||
status = JOB_FAILED
|
||||
error_msg = "Todos los radicados fallaron"
|
||||
else:
|
||||
status = JOB_COMPLETED
|
||||
error_msg = None
|
||||
|
||||
if error_msg:
|
||||
try:
|
||||
self.api.post_event(
|
||||
job_id,
|
||||
{
|
||||
"event_type": "warning",
|
||||
"message": error_msg,
|
||||
"metadata": {"fatal": bool(summary.fatal_error)},
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
self.api.patch_job(
|
||||
job_id,
|
||||
status=status,
|
||||
result=result,
|
||||
error=error_msg,
|
||||
)
|
||||
self._log(f"[worker {self.worker_id}] job {job_id} → {status}")
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"[worker {self.worker_id}] job {job_id} error: {e}")
|
||||
try:
|
||||
self.api.post_event(
|
||||
job_id,
|
||||
{
|
||||
"event_type": "warning",
|
||||
"message": f"Error fatal: {e}",
|
||||
"metadata": {"fatal": True},
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
self.api.patch_job(
|
||||
job_id,
|
||||
status=JOB_FAILED,
|
||||
error=str(e),
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
@@ -0,0 +1,113 @@
|
||||
"""Cliente MinIO mínimo para subir PDFs desde el worker (solo salida)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import List, Optional
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MinioSettings:
|
||||
endpoint: str = "https://s3.builker.xyz"
|
||||
bucket: str = "pys-cuentas-medicas"
|
||||
access_key: str = ""
|
||||
secret_key: str = ""
|
||||
region: str = "us-east-1"
|
||||
anonymous: bool = True
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> "MinioSettings":
|
||||
access_key = (os.getenv("MINIO_ACCESS_KEY") or "").strip()
|
||||
secret_key = (os.getenv("MINIO_SECRET_KEY") or "").strip()
|
||||
# pys-cuentas-medicas suele ser anónimo; solo firma si hay ambas keys.
|
||||
anon_flag = (os.getenv("MINIO_ANONYMOUS") or "").strip().lower()
|
||||
if anon_flag in ("0", "false", "no"):
|
||||
anonymous = False
|
||||
elif anon_flag in ("1", "true", "yes"):
|
||||
anonymous = True
|
||||
else:
|
||||
anonymous = not (access_key and secret_key)
|
||||
return cls(
|
||||
endpoint=os.getenv("MINIO_ENDPOINT", "https://s3.builker.xyz").rstrip("/"),
|
||||
bucket=os.getenv("MINIO_BUCKET", "pys-cuentas-medicas"),
|
||||
access_key=access_key,
|
||||
secret_key=secret_key,
|
||||
region=os.getenv("MINIO_REGION", "us-east-1"),
|
||||
anonymous=anonymous,
|
||||
)
|
||||
|
||||
|
||||
def pdf_key(job_id: str, nit: str, radicado: str) -> str:
|
||||
return f"downloads/{job_id}/{nit}/{radicado}.pdf"
|
||||
|
||||
|
||||
def manifest_key(job_id: str) -> str:
|
||||
return f"downloads/{job_id}/_manifest.json"
|
||||
|
||||
|
||||
def _s3_client(settings: MinioSettings):
|
||||
import boto3
|
||||
from botocore import UNSIGNED
|
||||
from botocore.client import Config
|
||||
|
||||
s3_cfg = {
|
||||
"addressing_style": "path",
|
||||
# Evita aws-chunked / checksums que rompen PutObject anónimo en MinIO.
|
||||
"request_checksum_calculation": "when_required",
|
||||
"response_checksum_validation": "when_required",
|
||||
}
|
||||
if settings.anonymous:
|
||||
return boto3.client(
|
||||
"s3",
|
||||
endpoint_url=settings.endpoint,
|
||||
region_name=settings.region,
|
||||
config=Config(signature_version=UNSIGNED, s3=s3_cfg),
|
||||
)
|
||||
return boto3.client(
|
||||
"s3",
|
||||
endpoint_url=settings.endpoint,
|
||||
aws_access_key_id=settings.access_key,
|
||||
aws_secret_access_key=settings.secret_key,
|
||||
region_name=settings.region,
|
||||
config=Config(signature_version="s3v4", s3=s3_cfg),
|
||||
)
|
||||
|
||||
|
||||
class WorkerMinioUploader:
|
||||
"""Solo PutObject de PDFs / manifest (sin API ni SQLite)."""
|
||||
|
||||
def __init__(self, settings: Optional[MinioSettings] = None):
|
||||
self.settings = settings or MinioSettings.from_env()
|
||||
self._client = None
|
||||
|
||||
@property
|
||||
def client(self):
|
||||
if self._client is None:
|
||||
self._client = _s3_client(self.settings)
|
||||
return self._client
|
||||
|
||||
def upload_pdf(self, job_id: str, nit: str, radicado: str, local_path: str | Path) -> str:
|
||||
key = pdf_key(job_id, nit, radicado)
|
||||
path = Path(local_path)
|
||||
# put_object (no multipart) es más fiable en buckets anónimos.
|
||||
self.client.put_object(
|
||||
Bucket=self.settings.bucket,
|
||||
Key=key,
|
||||
Body=path.read_bytes(),
|
||||
ContentType="application/pdf",
|
||||
)
|
||||
return key
|
||||
|
||||
def write_manifest(self, job_id: str, keys: List[str]) -> str:
|
||||
key = manifest_key(job_id)
|
||||
payload = {"job_id": job_id, "files": keys, "count": len(keys)}
|
||||
self.client.put_object(
|
||||
Bucket=self.settings.bucket,
|
||||
Key=key,
|
||||
Body=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
|
||||
ContentType="application/json",
|
||||
)
|
||||
return key
|
||||
@@ -0,0 +1,57 @@
|
||||
"""CLI del agent worker: idle hasta que el usuario asigne un job."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
|
||||
from bot_descarga.worker.agent import WorkerAgent
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(
|
||||
description="Agent BotDescarga: registra el PC y espera jobs asignados manualmente."
|
||||
)
|
||||
p.add_argument(
|
||||
"--api-url",
|
||||
default=os.getenv("BOT_API_URL", "https://pys1.builker.xyz/bot-descarga"),
|
||||
help="URL del API vía gateway (default: https://pys1.builker.xyz/bot-descarga)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--worker-id",
|
||||
required=True,
|
||||
help="Identificador único de este PC (ej. PC-CONTABILIDAD-01)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--token",
|
||||
default=os.getenv("BOT_API_TOKEN", "dev-token-change-me"),
|
||||
help="Bearer token (BOT_API_TOKEN)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--poll",
|
||||
type=float,
|
||||
default=8.0,
|
||||
help="Segundos entre heartbeat/claim",
|
||||
)
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
agent = WorkerAgent(
|
||||
api_url=args.api_url,
|
||||
worker_id=args.worker_id,
|
||||
token=args.token,
|
||||
poll_seconds=args.poll,
|
||||
)
|
||||
try:
|
||||
agent.run_forever()
|
||||
except KeyboardInterrupt:
|
||||
print("\nDetenido por usuario")
|
||||
agent.stop()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,58 @@
|
||||
"""CLI para el Bot de Descargas PDF."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
|
||||
from bot_descarga.audit.events import BotEvent
|
||||
from bot_descarga.config.settings import DEFAULT_SETTINGS
|
||||
from bot_descarga.core.download_service import DownloadService
|
||||
|
||||
|
||||
def _print_event(event: BotEvent) -> None:
|
||||
print(event.display_message(), flush=True)
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Descarga facturas PDF desde Imaging y las organiza por NIT."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--excel",
|
||||
required=True,
|
||||
help="Ruta al archivo Excel (col A=radicado, col B=NIT)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--carpeta",
|
||||
required=True,
|
||||
help="Carpeta destino de las descargas",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
|
||||
service = DownloadService(
|
||||
ruta_excel=args.excel,
|
||||
carpeta_descarga=args.carpeta,
|
||||
settings=DEFAULT_SETTINGS,
|
||||
event_callback=_print_event,
|
||||
)
|
||||
summary = service.run()
|
||||
|
||||
audit_path = getattr(service.bus.jsonl_logger, "path", None)
|
||||
if audit_path:
|
||||
print(f"Auditoría JSONL: {audit_path}", flush=True)
|
||||
|
||||
print(
|
||||
f"Resumen: {summary.exitosos}/{summary.total} exitosos, "
|
||||
f"{len(summary.fallidos)} fallidos",
|
||||
flush=True,
|
||||
)
|
||||
return 0 if not summary.fallidos else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,7 @@
|
||||
selenium>=4.15.0
|
||||
openpyxl>=3.1.0
|
||||
pikepdf>=8.0.0
|
||||
httpx>=0.27.0
|
||||
boto3>=1.34.0
|
||||
customtkinter>=5.2.0
|
||||
pytest>=7.0.0
|
||||
@@ -0,0 +1,12 @@
|
||||
"""Punto de entrada de la interfaz gráfica."""
|
||||
|
||||
from bot_descarga.ui.dashboard import BotDashboard
|
||||
|
||||
|
||||
def main():
|
||||
app = BotDashboard()
|
||||
app.mainloop()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,60 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from bot_descarga.config.settings import Settings
|
||||
from bot_descarga.core.download_service import DownloadService
|
||||
from bot_descarga.models.resultado import RunSummary
|
||||
|
||||
|
||||
def test_friendly_dns_error_message():
|
||||
svc = DownloadService(ruta_excel="x.xlsx", carpeta_descarga=".", event_callback=lambda e: None)
|
||||
msg = svc._friendly_browser_error(
|
||||
Exception("unknown error: net::ERR_NAME_NOT_RESOLVED\n (Session info: chrome=1)")
|
||||
)
|
||||
assert "ERR_NAME_NOT_RESOLVED" in msg
|
||||
assert "Imaging" in msg
|
||||
assert svc.settings.url_sistema in msg
|
||||
|
||||
|
||||
def test_browser_start_failure_sets_fatal_error(monkeypatch):
|
||||
events = []
|
||||
|
||||
def fake_read(_self, _path):
|
||||
from bot_descarga.models.radicado import RadicadoRecord
|
||||
|
||||
return [RadicadoRecord("1", "NIT1")]
|
||||
|
||||
class BoomBrowser:
|
||||
def start(self, *a, **k):
|
||||
raise Exception("unknown error: net::ERR_NAME_NOT_RESOLVED")
|
||||
|
||||
def close(self):
|
||||
pass
|
||||
|
||||
monkeypatch.setattr(
|
||||
"bot_descarga.core.download_service.ExcelReader.read",
|
||||
fake_read,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"bot_descarga.core.download_service.BrowserManager",
|
||||
lambda settings=None: BoomBrowser(),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"bot_descarga.core.download_service.JsonlLogger",
|
||||
lambda *a, **k: type(
|
||||
"L",
|
||||
(),
|
||||
{"path": "x", "write": lambda *x: None, "close": lambda *x: None},
|
||||
)(),
|
||||
)
|
||||
|
||||
svc = DownloadService(
|
||||
ruta_excel="dummy.xlsx",
|
||||
carpeta_descarga=".",
|
||||
settings=Settings(url_sistema="http://serviciosimaging:8180/"),
|
||||
event_callback=lambda e: events.append(e),
|
||||
)
|
||||
summary = svc.run()
|
||||
assert isinstance(summary, RunSummary)
|
||||
assert summary.fatal_error
|
||||
assert "ERR_NAME_NOT_RESOLVED" in summary.fatal_error
|
||||
assert any("FALLIDO" in e.message or "ERR_NAME_NOT_RESOLVED" in e.message for e in events)
|
||||
@@ -0,0 +1,62 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from bot_descarga.models.resultado import RunSummary
|
||||
from bot_descarga.core.download_service import DownloadService
|
||||
|
||||
|
||||
def test_download_service_accepts_should_cancel(monkeypatch):
|
||||
"""Verifica que should_cancel se honra antes de procesar radicados."""
|
||||
calls = {"cancel_checks": 0}
|
||||
|
||||
def fake_read(_self, _path):
|
||||
from bot_descarga.models.radicado import RadicadoRecord
|
||||
return [
|
||||
RadicadoRecord("1", "NIT1"),
|
||||
RadicadoRecord("2", "NIT1"),
|
||||
]
|
||||
|
||||
class FakeBrowser:
|
||||
def start(self, *a, **k):
|
||||
return object(), object()
|
||||
|
||||
def is_alive(self):
|
||||
return True
|
||||
|
||||
def close(self):
|
||||
pass
|
||||
|
||||
def reconnect(self, *a, **k):
|
||||
return object(), object()
|
||||
|
||||
monkeypatch.setattr(
|
||||
"bot_descarga.core.download_service.ExcelReader.read",
|
||||
fake_read,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"bot_descarga.core.download_service.BrowserManager",
|
||||
lambda settings=None: FakeBrowser(),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"bot_descarga.core.download_service.ImagingClient",
|
||||
lambda *a, **k: object(),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"bot_descarga.core.download_service.JsonlLogger",
|
||||
lambda *a, **k: type("L", (), {"path": "x", "write": lambda *x: None, "close": lambda *x: None})(),
|
||||
)
|
||||
|
||||
def should_cancel():
|
||||
calls["cancel_checks"] += 1
|
||||
return True
|
||||
|
||||
svc = DownloadService(
|
||||
ruta_excel="dummy.xlsx",
|
||||
carpeta_descarga=".",
|
||||
should_cancel=should_cancel,
|
||||
event_callback=lambda e: None,
|
||||
)
|
||||
# Evitar crear carpeta real problemática: FileManager.mkdirs ok
|
||||
summary = svc.run()
|
||||
assert isinstance(summary, RunSummary)
|
||||
assert summary.cancelled is True
|
||||
assert calls["cancel_checks"] >= 1
|
||||
@@ -0,0 +1,61 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from openpyxl import Workbook
|
||||
|
||||
from bot_descarga.infrastructure.excel_reader import ExcelReader, limpiar_nombre_carpeta
|
||||
|
||||
|
||||
def _make_excel(rows, path: Path) -> Path:
|
||||
wb = Workbook()
|
||||
ws = wb.active
|
||||
for row in rows:
|
||||
ws.append(row)
|
||||
wb.save(path)
|
||||
return path
|
||||
|
||||
|
||||
def test_limpiar_nombre_carpeta_caracteres_invalidos():
|
||||
assert limpiar_nombre_carpeta(r"900/123:456") == "900_123_456"
|
||||
assert limpiar_nombre_carpeta(" ") == "SIN_NIT"
|
||||
assert limpiar_nombre_carpeta("") == "SIN_NIT"
|
||||
|
||||
|
||||
def test_excel_reader_lee_radicados_y_nit():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = Path(tmp) / "datos.xlsx"
|
||||
_make_excel(
|
||||
[
|
||||
["Radicado", "NIT"],
|
||||
["100001", "900111222"],
|
||||
["100002", "900111222"],
|
||||
["100003", None],
|
||||
["", "800"],
|
||||
],
|
||||
path,
|
||||
)
|
||||
records = ExcelReader().read(str(path))
|
||||
|
||||
assert len(records) == 3
|
||||
assert records[0].radicado == "100001"
|
||||
assert records[0].nit == "900111222"
|
||||
assert records[2].nit == "SIN_NIT"
|
||||
|
||||
|
||||
def test_excel_reader_omite_encabezados_variantes():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = Path(tmp) / "datos.xlsx"
|
||||
_make_excel(
|
||||
[
|
||||
["proceso", "nit"],
|
||||
["número", "nit"],
|
||||
["555", "1"],
|
||||
],
|
||||
path,
|
||||
)
|
||||
records = ExcelReader().read(str(path))
|
||||
|
||||
assert len(records) == 1
|
||||
assert records[0].radicado == "555"
|
||||
@@ -0,0 +1,69 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from bot_descarga.config.settings import Settings
|
||||
from bot_descarga.infrastructure.file_manager import FileManager
|
||||
from bot_descarga.infrastructure.pdf_validator import PdfValidator
|
||||
|
||||
|
||||
class AlwaysValid:
|
||||
def validate(self, ruta: str):
|
||||
return True, "OK"
|
||||
|
||||
|
||||
class AlwaysInvalid:
|
||||
def validate(self, ruta: str):
|
||||
return False, "corrupto"
|
||||
|
||||
|
||||
def test_ruta_final_crea_carpeta_nit():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fm = FileManager(tmp, AlwaysValid())
|
||||
ruta = fm.ruta_final("900111", "12345")
|
||||
assert Path(ruta).parent.exists()
|
||||
assert Path(ruta).name == "12345.pdf"
|
||||
assert "900111" in ruta
|
||||
|
||||
|
||||
def test_exists_valid_cuando_no_existe():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fm = FileManager(tmp, AlwaysValid())
|
||||
ok, ruta, motivo = fm.exists_valid("900", "1")
|
||||
assert ok is False
|
||||
assert motivo == "no existe"
|
||||
assert ruta.endswith("1.pdf")
|
||||
|
||||
|
||||
def test_exists_valid_cuando_valido():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fm = FileManager(tmp, AlwaysValid())
|
||||
ruta = fm.ruta_final("900", "1")
|
||||
Path(ruta).write_bytes(b"%PDF-dummy")
|
||||
ok, _, motivo = fm.exists_valid("900", "1")
|
||||
assert ok is True
|
||||
assert motivo == "OK"
|
||||
assert Path(ruta).exists()
|
||||
|
||||
|
||||
def test_exists_valid_elimina_corrupto():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fm = FileManager(tmp, AlwaysInvalid())
|
||||
ruta = fm.ruta_final("900", "1")
|
||||
Path(ruta).write_bytes(b"basura")
|
||||
ok, _, motivo = fm.exists_valid("900", "1")
|
||||
assert ok is False
|
||||
assert motivo == "corrupto"
|
||||
assert not Path(ruta).exists()
|
||||
|
||||
|
||||
def test_move_to_final():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fm = FileManager(tmp, PdfValidator(Settings(peso_minimo_bytes=1)))
|
||||
src = Path(tmp) / "temp.pdf"
|
||||
src.write_bytes(b"%PDF-1.4")
|
||||
dest = fm.ruta_final("NIT1", "R1")
|
||||
fm.move_to_final(str(src), dest)
|
||||
assert Path(dest).exists()
|
||||
assert not src.exists()
|
||||
@@ -0,0 +1,15 @@
|
||||
from bot_descarga.ui.settings_store import DEFAULT_API_URL, normalize_api_url
|
||||
|
||||
|
||||
def test_default_api_is_pys1():
|
||||
assert "pys1.builker.xyz" in DEFAULT_API_URL
|
||||
assert DEFAULT_API_URL.rstrip("/").endswith("bot-descarga") or "/bot-descarga" in DEFAULT_API_URL
|
||||
|
||||
|
||||
def test_normalize_adds_prefix():
|
||||
assert normalize_api_url("https://pys1.builker.xyz") == "https://pys1.builker.xyz/bot-descarga"
|
||||
assert (
|
||||
normalize_api_url("https://pys1.builker.xyz/bot-descarga/")
|
||||
== "https://pys1.builker.xyz/bot-descarga"
|
||||
)
|
||||
assert normalize_api_url("http://127.0.0.1:8189") == "http://127.0.0.1:8189/bot-descarga"
|
||||
@@ -0,0 +1,54 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from bot_descarga.config.settings import Settings
|
||||
from bot_descarga.infrastructure.pdf_validator import PdfValidator
|
||||
|
||||
|
||||
def test_validate_rechaza_archivo_pequeno():
|
||||
settings = Settings(peso_minimo_bytes=10_000)
|
||||
validator = PdfValidator(settings)
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = Path(tmp) / "tiny.pdf"
|
||||
path.write_bytes(b"%PDF-1.4\n%%EOF\n")
|
||||
ok, motivo = validator.validate(str(path))
|
||||
|
||||
assert ok is False
|
||||
assert "Peso insuficiente" in motivo
|
||||
|
||||
|
||||
def test_validate_rechaza_sin_firma_pdf():
|
||||
settings = Settings(peso_minimo_bytes=10)
|
||||
validator = PdfValidator(settings)
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = Path(tmp) / "fake.pdf"
|
||||
path.write_bytes(b"NOTA PDF" + b"x" * 100)
|
||||
ok, motivo = validator.validate(str(path))
|
||||
|
||||
assert ok is False
|
||||
assert "firma" in motivo.lower()
|
||||
|
||||
|
||||
def test_validate_acepta_pdf_minimo_valido():
|
||||
"""PDF mínimo con firma, EOF y estructura pikepdf-compatible."""
|
||||
import pikepdf
|
||||
|
||||
settings = Settings(peso_minimo_bytes=50)
|
||||
validator = PdfValidator(settings)
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = Path(tmp) / "ok.pdf"
|
||||
pdf = pikepdf.Pdf.new()
|
||||
# Varias páginas para superar el peso mínimo sin corromper el archivo
|
||||
for _ in range(5):
|
||||
pdf.add_blank_page(page_size=(612, 792))
|
||||
pdf.save(path)
|
||||
|
||||
ok, motivo = validator.validate(str(path))
|
||||
|
||||
assert ok is True, motivo
|
||||
assert motivo == "OK"
|
||||
@@ -0,0 +1,26 @@
|
||||
import os
|
||||
|
||||
from bot_descarga.worker.minio_upload import MinioSettings, manifest_key, pdf_key
|
||||
|
||||
|
||||
def test_pdf_and_manifest_keys():
|
||||
assert pdf_key("abc", "900", "123") == "downloads/abc/900/123.pdf"
|
||||
assert manifest_key("abc") == "downloads/abc/_manifest.json"
|
||||
|
||||
|
||||
def test_minio_defaults_to_anonymous(monkeypatch):
|
||||
monkeypatch.delenv("MINIO_ACCESS_KEY", raising=False)
|
||||
monkeypatch.delenv("MINIO_SECRET_KEY", raising=False)
|
||||
monkeypatch.delenv("MINIO_ANONYMOUS", raising=False)
|
||||
monkeypatch.delenv("AWS_ACCESS_KEY_ID", raising=False)
|
||||
monkeypatch.delenv("AWS_SECRET_ACCESS_KEY", raising=False)
|
||||
settings = MinioSettings.from_env()
|
||||
assert settings.anonymous is True
|
||||
|
||||
|
||||
def test_minio_signed_only_with_both_keys(monkeypatch):
|
||||
monkeypatch.setenv("MINIO_ACCESS_KEY", "user")
|
||||
monkeypatch.setenv("MINIO_SECRET_KEY", "secret")
|
||||
monkeypatch.delenv("MINIO_ANONYMOUS", raising=False)
|
||||
settings = MinioSettings.from_env()
|
||||
assert settings.anonymous is False
|
||||
@@ -151,6 +151,33 @@ services:
|
||||
limits:
|
||||
memory: ${COUNT_API_MEMORY:-4g}
|
||||
|
||||
bot-descarga:
|
||||
build:
|
||||
context: ./services/bot_descarga
|
||||
dockerfile: Dockerfile
|
||||
container_name: ipsextractor-bot-descarga
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
database:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
BOT_API_TOKEN: ${BOT_API_TOKEN:-dev-token-change-me}
|
||||
BOT_API_HOST: 0.0.0.0
|
||||
BOT_API_PORT: "8200"
|
||||
DATABASE_URL: postgresql+psycopg://ipsextractor:ipsextractor@database:5432/ipsextractor
|
||||
MINIO_ENDPOINT: ${MINIO_ENDPOINT:-https://s3.builker.xyz}
|
||||
MINIO_BUCKET: ${MINIO_BUCKET:-pys-cuentas-medicas}
|
||||
MINIO_ANONYMOUS: ${MINIO_ANONYMOUS:-1}
|
||||
MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY:-}
|
||||
MINIO_SECRET_KEY: ${MINIO_SECRET_KEY:-}
|
||||
MINIO_REGION: ${MINIO_REGION:-us-east-1}
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://127.0.0.1:8200/health"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 20s
|
||||
|
||||
api_gateway:
|
||||
build:
|
||||
context: ./services/api_gateway
|
||||
@@ -163,10 +190,13 @@ services:
|
||||
condition: service_started
|
||||
count-api:
|
||||
condition: service_healthy
|
||||
bot-descarga:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
EXTRACTION_URL: http://extraction:8001
|
||||
CLEANING_URL: http://cleaning:8002
|
||||
COUNT_URL: http://count-api:8080
|
||||
BOT_DESCARGA_URL: http://bot-descarga:8200
|
||||
ports:
|
||||
# Único puerto expuesto al host. Los demás servicios solo son accesibles en la red Docker.
|
||||
- "0.0.0.0:8189:8000"
|
||||
|
||||
+76
-4
@@ -9,8 +9,8 @@ Project ipsextractor {
|
||||
database_type: 'PostgreSQL'
|
||||
Note: '''
|
||||
Base de datos compartida entre todos los microservicios (extraction, cleaning,
|
||||
count-facturas). Cada servicio accede directamente a las tablas que necesita.
|
||||
Las migraciones se gestionan con Alembic (services/extraction/alembic/).
|
||||
count-facturas, bot-descarga). Cada servicio accede directamente a las tablas
|
||||
que necesita. Las migraciones se gestionan con Alembic (services/db/alembic/).
|
||||
Cada servicio es desacoplado; comparten solo la DB como fuente de verdad.
|
||||
'''
|
||||
}
|
||||
@@ -35,6 +35,12 @@ TableGroup counting {
|
||||
count_job_files
|
||||
}
|
||||
|
||||
TableGroup bot_descarga {
|
||||
download_workers
|
||||
download_jobs
|
||||
download_job_events
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Orchestration — trabajo multimodular del usuario
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -61,7 +67,7 @@ Table jobs {
|
||||
Table tasks {
|
||||
id varchar(36) [pk, note: 'UUID v4']
|
||||
job_id varchar(36) [ref: > jobs.id, note: 'FK nullable — compat. con tasks anteriores']
|
||||
type varchar(64) [not null, note: 'counting | extraction | cleaning']
|
||||
type varchar(64) [not null, note: 'counting | extraction | cleaning | downloads']
|
||||
status varchar(32) [not null, default: 'pending', note: 'pending | running | completed | failed']
|
||||
title text
|
||||
request jsonb [not null, default: '{}', note: 'Payload de la petición que originó la tarea']
|
||||
@@ -82,11 +88,77 @@ Table tasks {
|
||||
|
||||
Note: '''
|
||||
Módulo/resultado dentro de un Job. Una task representa la ejecución
|
||||
de un servicio específico (conteo, extracción, limpieza).
|
||||
de un servicio específico (conteo, extracción, limpieza, bot de descarga).
|
||||
job_id es nullable para compatibilidad con tasks creadas antes de jobs.
|
||||
'''
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Bot de descarga — workers remotos + corridas PDF → MinIO
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
Table download_workers {
|
||||
id varchar(36) [pk, note: 'ID del agent (ej. PC-01 / zer0)']
|
||||
hostname text [not null, default: '']
|
||||
max_concurrent integer [not null, default: 1]
|
||||
status varchar(32) [not null, note: 'online | offline | busy']
|
||||
last_heartbeat timestamptz [not null]
|
||||
current_job_id varchar(36) [note: 'download_jobs.id en ejecución']
|
||||
|
||||
indexes {
|
||||
status
|
||||
}
|
||||
|
||||
Note: 'PC remota registrada por el agent BotDescarga (heartbeat).'
|
||||
}
|
||||
|
||||
Table download_jobs {
|
||||
id varchar(36) [pk, note: 'UUID v4 de la corrida']
|
||||
job_id varchar(36) [null, ref: > jobs.id, note: 'Job Desktop contenedor']
|
||||
task_id varchar(36) [null, ref: > tasks.id, note: 'Task type=downloads']
|
||||
worker_id varchar(36) [not null, note: 'download_workers.id asignado']
|
||||
status varchar(32) [not null, default: 'pending', note: 'pending | running | completed | failed | cancelled']
|
||||
title text
|
||||
excel_key text [not null, default: '', note: 'Clave MinIO del Excel']
|
||||
downloads_prefix text [not null, default: '', note: 'downloads/{id}/']
|
||||
original_filename text [not null, default: '']
|
||||
cancel_requested boolean [not null, default: false]
|
||||
result jsonb [not null, default: '{}']
|
||||
error text
|
||||
created_at timestamptz [not null]
|
||||
started_at timestamptz
|
||||
finished_at timestamptz
|
||||
|
||||
indexes {
|
||||
job_id
|
||||
task_id
|
||||
worker_id
|
||||
status
|
||||
(worker_id, status) [name: 'ix_download_jobs_worker_status']
|
||||
}
|
||||
|
||||
Note: 'Corrida de descarga remota. PDFs en MinIO bajo downloads/{id}/.'
|
||||
}
|
||||
|
||||
Table download_job_events {
|
||||
id integer [pk, increment]
|
||||
download_job_id varchar(36) [not null, ref: > download_jobs.id]
|
||||
event_type varchar(64) [not null]
|
||||
message text [not null]
|
||||
timestamp timestamptz [not null]
|
||||
radicado varchar(255)
|
||||
nit varchar(64)
|
||||
idx integer
|
||||
total integer
|
||||
metadata jsonb [not null, default: '{}']
|
||||
|
||||
indexes {
|
||||
(download_job_id, id) [name: 'ix_download_job_events_job_id']
|
||||
}
|
||||
|
||||
Note: 'Log incremental de progreso / errores de una corrida.'
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Counting — resultados del servicio de conteo de facturas médicas
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -18,6 +18,7 @@ logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name
|
||||
EXTRACTION_URL = os.getenv("EXTRACTION_URL", "http://extraction:8001").rstrip("/")
|
||||
CLEANING_URL = os.getenv("CLEANING_URL", "http://cleaning:8002").rstrip("/")
|
||||
COUNT_URL = os.getenv("COUNT_URL", "http://count-api:8080").rstrip("/")
|
||||
BOT_DESCARGA_URL = os.getenv("BOT_DESCARGA_URL", "http://bot-descarga:8200").rstrip("/")
|
||||
|
||||
# Timeouts: JSON ligero vs uploads OCR/conteo
|
||||
_TIMEOUT_JSON = httpx.Timeout(connect=5.0, read=30.0, write=30.0, pool=5.0)
|
||||
@@ -33,7 +34,13 @@ async def lifespan(app: FastAPI):
|
||||
timeout=_TIMEOUT_JSON,
|
||||
limits=httpx.Limits(max_connections=100, max_keepalive_connections=20),
|
||||
)
|
||||
_log.info("Gateway iniciado — extraction=%s cleaning=%s count=%s", EXTRACTION_URL, CLEANING_URL, COUNT_URL)
|
||||
_log.info(
|
||||
"Gateway iniciado — extraction=%s cleaning=%s count=%s bot_descarga=%s",
|
||||
EXTRACTION_URL,
|
||||
CLEANING_URL,
|
||||
COUNT_URL,
|
||||
BOT_DESCARGA_URL,
|
||||
)
|
||||
yield
|
||||
await _http_client.aclose()
|
||||
_http_client = None
|
||||
@@ -72,6 +79,7 @@ def _is_upload_path(path: str) -> bool:
|
||||
or "/upload" in path
|
||||
or path.startswith("/cleaning/analyze-csv")
|
||||
or path.startswith("/cleaning/jobs/from-csv")
|
||||
or path.startswith("/bot-descarga/jobs")
|
||||
)
|
||||
|
||||
|
||||
@@ -91,12 +99,14 @@ async def health() -> dict[str, Any]:
|
||||
extraction = await _safe_health(client, EXTRACTION_URL)
|
||||
cleaning = await _safe_health(client, CLEANING_URL)
|
||||
counting = await _safe_health(client, COUNT_URL)
|
||||
bot_descarga = await _safe_health(client, BOT_DESCARGA_URL)
|
||||
return {
|
||||
"status": "ok" if extraction["ok"] and cleaning["ok"] else "degraded",
|
||||
"services": {
|
||||
"extraction": extraction,
|
||||
"cleaning": cleaning,
|
||||
"counting": counting,
|
||||
"bot_descarga": bot_descarga,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -152,6 +162,11 @@ async def count_jobs_proxy(path: str, request: Request) -> Response:
|
||||
return await _proxy_request(request, f"{EXTRACTION_URL}/count-jobs/{path}")
|
||||
|
||||
|
||||
@app.api_route("/bot-descarga/{path:path}", methods=["GET", "POST", "PUT", "PATCH", "DELETE"])
|
||||
async def bot_descarga_proxy(path: str, request: Request) -> Response:
|
||||
return await _proxy_request(request, f"{BOT_DESCARGA_URL}/{path}")
|
||||
|
||||
|
||||
@app.api_route("/jobs", methods=["GET", "POST"])
|
||||
async def jobs_root_proxy(request: Request) -> Response:
|
||||
return await _proxy_request(request, f"{EXTRACTION_URL}/jobs")
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PYTHONPATH=/app
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY requirements.txt /app/requirements.txt
|
||||
RUN pip install --no-cache-dir -r /app/requirements.txt
|
||||
|
||||
COPY domain /app/domain
|
||||
COPY infrastructure /app/infrastructure
|
||||
COPY api /app/api
|
||||
|
||||
ENV BOT_API_HOST=0.0.0.0
|
||||
ENV BOT_API_PORT=8200
|
||||
|
||||
EXPOSE 8200
|
||||
|
||||
HEALTHCHECK --interval=10s --timeout=5s --retries=5 --start-period=15s \
|
||||
CMD curl -f http://127.0.0.1:8200/health || exit 1
|
||||
|
||||
CMD ["uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8200"]
|
||||
@@ -0,0 +1,51 @@
|
||||
# Bot Descarga API (infraestructura IPSExtractor)
|
||||
|
||||
Microservicio FastAPI de control remoto de bots. Corre en Docker;
|
||||
las PCs de bot **no** exponen puertos — solo el agent en [`bots/bot_descarga`](../../bots/bot_descarga).
|
||||
|
||||
Persistencia: **PostgreSQL compartido** (`DATABASE_URL`), tablas
|
||||
`download_workers`, `download_jobs`, `download_job_events` + `tasks` (`type=downloads`).
|
||||
|
||||
## Layout MinIO (`pys-cuentas-medicas`)
|
||||
|
||||
```
|
||||
uploads/{download_job_id}/input.xlsx
|
||||
downloads/{download_job_id}/{nit}/{radicado}.pdf
|
||||
```
|
||||
|
||||
## Arranque (compose)
|
||||
|
||||
Servicio `bot-descarga` en el `docker-compose.yml` raíz.
|
||||
Gateway: `/bot-descarga/*` → este servicio `:8200`.
|
||||
|
||||
## Variables
|
||||
|
||||
```
|
||||
BOT_API_TOKEN=dev-token-change-me
|
||||
DATABASE_URL=postgresql+psycopg://ipsextractor:ipsextractor@database:5432/ipsextractor
|
||||
MINIO_ENDPOINT=https://s3.builker.xyz
|
||||
MINIO_BUCKET=pys-cuentas-medicas
|
||||
MINIO_ANONYMOUS=1
|
||||
```
|
||||
|
||||
## Worker en PCs (`bots/bot_descarga`)
|
||||
|
||||
```bash
|
||||
cd bots/bot_descarga
|
||||
python -m bot_descarga.worker.run \
|
||||
--api-url http://GATEWAY:8189/bot-descarga \
|
||||
--worker-id PC-01 \
|
||||
--token $BOT_API_TOKEN
|
||||
```
|
||||
|
||||
O con GUI: `python run_gui.py`. Ver [README del agent](../../bots/bot_descarga/README.md).
|
||||
|
||||
El agent queda idle hasta que el usuario cree un job desde el desktop
|
||||
(enviando `job_id` del Job IPSExtractor).
|
||||
|
||||
## Tests (API)
|
||||
|
||||
```bash
|
||||
cd services/bot_descarga
|
||||
pytest -q
|
||||
```
|
||||
@@ -0,0 +1,293 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from fastapi import Depends, FastAPI, File, Form, HTTPException, Query, UploadFile
|
||||
from fastapi.responses import Response
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from domain.models import JobStatus
|
||||
from infrastructure.auth import require_bearer
|
||||
from infrastructure.storage_minio import MinioSettings, MinioStorage
|
||||
from infrastructure.store import JobStore
|
||||
|
||||
store = JobStore()
|
||||
minio = MinioStorage(MinioSettings.from_env())
|
||||
|
||||
app = FastAPI(title="BotDescarga Remote API", version="1.0.0")
|
||||
|
||||
|
||||
class RegisterWorkerBody(BaseModel):
|
||||
worker_id: str
|
||||
hostname: str = ""
|
||||
max_concurrent: int = 1
|
||||
|
||||
|
||||
class HeartbeatBody(BaseModel):
|
||||
worker_id: str
|
||||
|
||||
|
||||
class EventBody(BaseModel):
|
||||
event_type: str
|
||||
message: str
|
||||
radicado: Optional[str] = None
|
||||
nit: Optional[str] = None
|
||||
index: Optional[int] = None
|
||||
total: Optional[int] = None
|
||||
metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
timestamp: Optional[str] = None
|
||||
|
||||
|
||||
class PatchJobBody(BaseModel):
|
||||
status: Optional[str] = None
|
||||
result: Optional[Dict[str, Any]] = None
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
def health() -> dict:
|
||||
return {"status": "ok", "service": "bot-descarga"}
|
||||
|
||||
|
||||
@app.post("/workers/register")
|
||||
def register_worker(body: RegisterWorkerBody, _: str = Depends(require_bearer)) -> dict:
|
||||
worker = store.register_worker(
|
||||
body.worker_id,
|
||||
hostname=body.hostname,
|
||||
max_concurrent=body.max_concurrent,
|
||||
)
|
||||
return worker.to_dict()
|
||||
|
||||
|
||||
@app.post("/workers/heartbeat")
|
||||
def heartbeat(body: HeartbeatBody, _: str = Depends(require_bearer)) -> dict:
|
||||
worker = store.heartbeat(body.worker_id)
|
||||
if not worker:
|
||||
raise HTTPException(status_code=404, detail="Worker no registrado")
|
||||
return worker.to_dict()
|
||||
|
||||
|
||||
@app.get("/workers")
|
||||
def list_workers(_: str = Depends(require_bearer)) -> List[dict]:
|
||||
return [w.to_dict() for w in store.list_workers()]
|
||||
|
||||
|
||||
@app.get("/workers/me/pending")
|
||||
def claim_pending(
|
||||
worker_id: str = Query(...),
|
||||
_: str = Depends(require_bearer),
|
||||
) -> dict:
|
||||
job = store.claim_pending(worker_id)
|
||||
if not job:
|
||||
return {"job": None}
|
||||
return {"job": job.to_dict()}
|
||||
|
||||
|
||||
@app.post("/jobs")
|
||||
async def create_job(
|
||||
worker_id: str = Form(...),
|
||||
job_id: str = Form(...),
|
||||
title: str = Form(""),
|
||||
file: UploadFile = File(...),
|
||||
_: str = Depends(require_bearer),
|
||||
) -> dict:
|
||||
if not worker_id.strip():
|
||||
raise HTTPException(status_code=400, detail="worker_id obligatorio")
|
||||
parent_job_id = job_id.strip()
|
||||
if not parent_job_id:
|
||||
raise HTTPException(status_code=400, detail="job_id (Job Desktop) obligatorio")
|
||||
if not store.parent_job_exists(parent_job_id):
|
||||
raise HTTPException(status_code=404, detail="Trabajo (job) no encontrado")
|
||||
|
||||
if not store.is_worker_online(worker_id):
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=f"Worker '{worker_id}' no está online. Enciende el agent primero.",
|
||||
)
|
||||
|
||||
data = await file.read()
|
||||
if not data:
|
||||
raise HTTPException(status_code=400, detail="Archivo vacío")
|
||||
|
||||
original = file.filename or "input.xlsx"
|
||||
content_type = file.content_type or "application/octet-stream"
|
||||
title_clean = title.strip()
|
||||
|
||||
job = store.create_job(
|
||||
worker_id=worker_id.strip(),
|
||||
excel_key="",
|
||||
original_filename=original,
|
||||
title=title_clean,
|
||||
job_id=parent_job_id,
|
||||
)
|
||||
task_id = store.create_downloads_task(
|
||||
parent_job_id=parent_job_id,
|
||||
download_job_id=job.id,
|
||||
worker_id=worker_id.strip(),
|
||||
title=title_clean or f"Bot descarga ({original})",
|
||||
original_filename=original,
|
||||
)
|
||||
job = store.get_job(job.id)
|
||||
assert job is not None
|
||||
|
||||
try:
|
||||
key = minio.upload_excel(job.id, data, original, content_type=content_type)
|
||||
except Exception as e:
|
||||
store.patch_job(job.id, status=JobStatus.FAILED.value, error=f"MinIO upload failed: {e}")
|
||||
raise HTTPException(status_code=502, detail=f"Error subiendo a MinIO: {e}") from e
|
||||
|
||||
job = store.update_excel_key(job.id, key)
|
||||
assert job is not None
|
||||
store.add_event(
|
||||
job.id,
|
||||
"run_started",
|
||||
f"Job creado y asignado a {worker_id}",
|
||||
metadata={
|
||||
"excel_key": key,
|
||||
"worker_id": worker_id,
|
||||
"job_id": parent_job_id,
|
||||
"task_id": task_id,
|
||||
},
|
||||
)
|
||||
return job.to_dict()
|
||||
|
||||
|
||||
@app.get("/jobs")
|
||||
def list_jobs(
|
||||
limit: int = Query(100, ge=1, le=500),
|
||||
job_id: Optional[str] = Query(None, description="Filtrar por Job Desktop"),
|
||||
_: str = Depends(require_bearer),
|
||||
) -> List[dict]:
|
||||
return [j.to_dict() for j in store.list_jobs(limit=limit, job_id=job_id)]
|
||||
|
||||
|
||||
@app.get("/jobs/{download_job_id}")
|
||||
def get_job(download_job_id: str, _: str = Depends(require_bearer)) -> dict:
|
||||
job = store.get_job(download_job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job no encontrado")
|
||||
return job.to_dict()
|
||||
|
||||
|
||||
@app.get("/jobs/{download_job_id}/files")
|
||||
def list_job_files(download_job_id: str, _: str = Depends(require_bearer)) -> dict:
|
||||
job = store.get_job(download_job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job no encontrado")
|
||||
try:
|
||||
return minio.list_download_files(download_job_id, prefix=job.downloads_prefix)
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=502, detail=f"Error listando MinIO: {e}") from e
|
||||
|
||||
|
||||
@app.get("/jobs/{download_job_id}/events")
|
||||
def get_events(
|
||||
download_job_id: str,
|
||||
since: int = Query(0, ge=0),
|
||||
_: str = Depends(require_bearer),
|
||||
) -> dict:
|
||||
if not store.get_job(download_job_id):
|
||||
raise HTTPException(status_code=404, detail="Job no encontrado")
|
||||
events = store.list_events(download_job_id, since_id=since)
|
||||
return {"events": [e.to_dict() for e in events]}
|
||||
|
||||
|
||||
@app.post("/jobs/{download_job_id}/cancel")
|
||||
def cancel_job(download_job_id: str, _: str = Depends(require_bearer)) -> dict:
|
||||
job = store.request_cancel(download_job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job no encontrado")
|
||||
store.add_event(download_job_id, "warning", "Cancelación solicitada por el usuario")
|
||||
return job.to_dict()
|
||||
|
||||
|
||||
@app.get("/jobs/{download_job_id}/file")
|
||||
def get_job_file(download_job_id: str, _: str = Depends(require_bearer)):
|
||||
job = store.get_job(download_job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job no encontrado")
|
||||
if not job.excel_key:
|
||||
raise HTTPException(status_code=404, detail="Excel no disponible")
|
||||
|
||||
try:
|
||||
data = minio.get_bytes(job.excel_key)
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=502, detail=f"Error leyendo MinIO: {e}") from e
|
||||
|
||||
filename = Path(job.excel_key).name
|
||||
return Response(
|
||||
content=data,
|
||||
media_type="application/octet-stream",
|
||||
headers={"Content-Disposition": f'attachment; filename="{filename}"'},
|
||||
)
|
||||
|
||||
|
||||
@app.get("/jobs/{download_job_id}/file-url")
|
||||
def get_job_file_url(download_job_id: str, _: str = Depends(require_bearer)) -> dict:
|
||||
job = store.get_job(download_job_id)
|
||||
if not job or not job.excel_key:
|
||||
raise HTTPException(status_code=404, detail="Job/archivo no encontrado")
|
||||
try:
|
||||
url = minio.presigned_get(job.excel_key)
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=502, detail=str(e)) from e
|
||||
return {"url": url, "excel_key": job.excel_key}
|
||||
|
||||
|
||||
@app.post("/jobs/{download_job_id}/events")
|
||||
def post_event(
|
||||
download_job_id: str,
|
||||
body: EventBody,
|
||||
_: str = Depends(require_bearer),
|
||||
) -> dict:
|
||||
if not store.get_job(download_job_id):
|
||||
raise HTTPException(status_code=404, detail="Job no encontrado")
|
||||
event = store.add_event(
|
||||
download_job_id,
|
||||
body.event_type,
|
||||
body.message,
|
||||
radicado=body.radicado,
|
||||
nit=body.nit,
|
||||
index=body.index,
|
||||
total=body.total,
|
||||
metadata=body.metadata,
|
||||
timestamp=body.timestamp,
|
||||
)
|
||||
job = store.get_job(download_job_id)
|
||||
if job and body.index is not None and body.total is not None:
|
||||
result = dict(job.result or {})
|
||||
result["processed"] = body.index
|
||||
result["total"] = body.total
|
||||
if body.nit:
|
||||
result["nit_actual"] = body.nit
|
||||
store.patch_job(download_job_id, result=result)
|
||||
return event.to_dict()
|
||||
|
||||
|
||||
@app.patch("/jobs/{download_job_id}")
|
||||
def patch_job(
|
||||
download_job_id: str,
|
||||
body: PatchJobBody,
|
||||
_: str = Depends(require_bearer),
|
||||
) -> dict:
|
||||
if body.status and body.status not in {s.value for s in JobStatus}:
|
||||
raise HTTPException(status_code=400, detail="status inválido")
|
||||
job = store.patch_job(
|
||||
download_job_id,
|
||||
status=body.status,
|
||||
result=body.result,
|
||||
error=body.error,
|
||||
)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job no encontrado")
|
||||
return job.to_dict()
|
||||
|
||||
|
||||
@app.get("/jobs/{download_job_id}/cancel-flag")
|
||||
def cancel_flag(download_job_id: str, _: str = Depends(require_bearer)) -> dict:
|
||||
job = store.get_job(download_job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job no encontrado")
|
||||
return {"cancel_requested": job.cancel_requested, "status": job.status}
|
||||
@@ -0,0 +1,76 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from enum import Enum
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
|
||||
def utc_now() -> str:
|
||||
return datetime.now(timezone.utc).astimezone().isoformat(timespec="seconds")
|
||||
|
||||
|
||||
class WorkerStatus(str, Enum):
|
||||
ONLINE = "online"
|
||||
OFFLINE = "offline"
|
||||
BUSY = "busy"
|
||||
|
||||
|
||||
class JobStatus(str, Enum):
|
||||
PENDING = "pending"
|
||||
RUNNING = "running"
|
||||
COMPLETED = "completed"
|
||||
FAILED = "failed"
|
||||
CANCELLED = "cancelled"
|
||||
|
||||
|
||||
@dataclass
|
||||
class Worker:
|
||||
id: str
|
||||
hostname: str = ""
|
||||
max_concurrent: int = 1
|
||||
status: str = WorkerStatus.ONLINE.value
|
||||
last_heartbeat: str = field(default_factory=utc_now)
|
||||
current_job_id: Optional[str] = None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Job:
|
||||
id: str
|
||||
worker_id: str
|
||||
status: str = JobStatus.PENDING.value
|
||||
title: str = ""
|
||||
excel_key: str = ""
|
||||
downloads_prefix: str = ""
|
||||
original_filename: str = ""
|
||||
cancel_requested: bool = False
|
||||
result: Dict[str, Any] = field(default_factory=dict)
|
||||
error: Optional[str] = None
|
||||
created_at: str = field(default_factory=utc_now)
|
||||
started_at: Optional[str] = None
|
||||
finished_at: Optional[str] = None
|
||||
job_id: Optional[str] = None
|
||||
task_id: Optional[str] = None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
|
||||
|
||||
@dataclass
|
||||
class JobEventRecord:
|
||||
id: int
|
||||
job_id: str
|
||||
event_type: str
|
||||
message: str
|
||||
timestamp: str
|
||||
radicado: Optional[str] = None
|
||||
nit: Optional[str] = None
|
||||
index: Optional[int] = None
|
||||
total: Optional[int] = None
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
@@ -0,0 +1,20 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import Header, HTTPException
|
||||
|
||||
|
||||
def get_api_token() -> str:
|
||||
return os.getenv("BOT_API_TOKEN", "dev-token-change-me")
|
||||
|
||||
|
||||
def require_bearer(authorization: Optional[str] = Header(default=None)) -> str:
|
||||
expected = get_api_token()
|
||||
if not authorization or not authorization.lower().startswith("bearer "):
|
||||
raise HTTPException(status_code=401, detail="Authorization Bearer requerido")
|
||||
token = authorization.split(" ", 1)[1].strip()
|
||||
if token != expected:
|
||||
raise HTTPException(status_code=401, detail="Token inválido")
|
||||
return token
|
||||
@@ -0,0 +1,159 @@
|
||||
"""Persistencia Postgres del servicio bot-descarga (DB compartida IPSExtractor)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import uuid
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Iterator
|
||||
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
Boolean,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Integer,
|
||||
String,
|
||||
Text,
|
||||
create_engine,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.orm import DeclarativeBase, Mapped, Session, mapped_column, sessionmaker
|
||||
|
||||
DATABASE_URL = os.getenv(
|
||||
"DATABASE_URL",
|
||||
"postgresql+psycopg://ipsextractor:ipsextractor@database:5432/ipsextractor",
|
||||
)
|
||||
|
||||
engine = create_engine(DATABASE_URL, pool_pre_ping=True)
|
||||
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False)
|
||||
|
||||
|
||||
class Base(DeclarativeBase):
|
||||
pass
|
||||
|
||||
|
||||
def utcnow() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def new_id() -> str:
|
||||
return str(uuid.uuid4())
|
||||
|
||||
|
||||
def _iso(dt: datetime | None) -> str | None:
|
||||
if dt is None:
|
||||
return None
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
return dt.astimezone().isoformat(timespec="seconds")
|
||||
|
||||
|
||||
class Job(Base):
|
||||
"""Contenedor Desktop (lectura / FK)."""
|
||||
|
||||
__tablename__ = "jobs"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_id)
|
||||
status: Mapped[str] = mapped_column(String(32), default="pending", index=True)
|
||||
title: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
job_metadata: Mapped[dict[str, Any]] = mapped_column("metadata", JSON, default=dict)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
|
||||
|
||||
|
||||
class Task(Base):
|
||||
__tablename__ = "tasks"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_id)
|
||||
job_id: Mapped[str | None] = mapped_column(
|
||||
String(36), ForeignKey("jobs.id"), nullable=True, index=True
|
||||
)
|
||||
type: Mapped[str] = mapped_column(String(64), index=True)
|
||||
status: Mapped[str] = mapped_column(String(32), default="pending", index=True)
|
||||
title: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
request: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict)
|
||||
result: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict)
|
||||
error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
owner_id: Mapped[str | None] = mapped_column(String(36), nullable=True, index=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
|
||||
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class DownloadWorkerRow(Base):
|
||||
__tablename__ = "download_workers"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True)
|
||||
hostname: Mapped[str] = mapped_column(Text, default="")
|
||||
max_concurrent: Mapped[int] = mapped_column(Integer, default=1)
|
||||
status: Mapped[str] = mapped_column(String(32), index=True)
|
||||
last_heartbeat: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
|
||||
current_job_id: Mapped[str | None] = mapped_column(String(36), nullable=True)
|
||||
|
||||
|
||||
class DownloadJobRow(Base):
|
||||
__tablename__ = "download_jobs"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_id)
|
||||
job_id: Mapped[str | None] = mapped_column(
|
||||
String(36), ForeignKey("jobs.id"), nullable=True, index=True
|
||||
)
|
||||
task_id: Mapped[str | None] = mapped_column(
|
||||
String(36), ForeignKey("tasks.id"), nullable=True, index=True
|
||||
)
|
||||
worker_id: Mapped[str] = mapped_column(String(36), index=True)
|
||||
status: Mapped[str] = mapped_column(String(32), default="pending", index=True)
|
||||
title: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
excel_key: Mapped[str] = mapped_column(Text, default="")
|
||||
downloads_prefix: Mapped[str] = mapped_column(Text, default="")
|
||||
original_filename: Mapped[str] = mapped_column(Text, default="")
|
||||
cancel_requested: Mapped[bool] = mapped_column(Boolean, default=False)
|
||||
result: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict)
|
||||
error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
|
||||
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class DownloadJobEventRow(Base):
|
||||
__tablename__ = "download_job_events"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||
download_job_id: Mapped[str] = mapped_column(
|
||||
String(36), ForeignKey("download_jobs.id"), index=True
|
||||
)
|
||||
event_type: Mapped[str] = mapped_column(String(64))
|
||||
message: Mapped[str] = mapped_column(Text)
|
||||
timestamp: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
|
||||
radicado: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
nit: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
idx: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
total: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
metadata_json: Mapped[dict[str, Any]] = mapped_column("metadata", JSON, default=dict)
|
||||
|
||||
|
||||
def init_db() -> None:
|
||||
if DATABASE_URL.startswith("postgresql"):
|
||||
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as connection:
|
||||
connection.execute(text("SELECT pg_advisory_lock(74512004)"))
|
||||
try:
|
||||
Base.metadata.create_all(bind=connection)
|
||||
finally:
|
||||
connection.execute(text("SELECT pg_advisory_unlock(74512004)"))
|
||||
return
|
||||
Base.metadata.create_all(bind=engine)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def session_scope() -> Iterator[Session]:
|
||||
session = SessionLocal()
|
||||
try:
|
||||
yield session
|
||||
session.commit()
|
||||
except Exception:
|
||||
session.rollback()
|
||||
raise
|
||||
finally:
|
||||
session.close()
|
||||
@@ -0,0 +1,215 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import BinaryIO, List, Optional
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MinioSettings:
|
||||
endpoint: str = "https://s3.builker.xyz"
|
||||
bucket: str = "pys-cuentas-medicas"
|
||||
access_key: str = ""
|
||||
secret_key: str = ""
|
||||
region: str = "us-east-1"
|
||||
anonymous: bool = True
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> "MinioSettings":
|
||||
access_key = (os.getenv("MINIO_ACCESS_KEY") or "").strip()
|
||||
secret_key = (os.getenv("MINIO_SECRET_KEY") or "").strip()
|
||||
anon_flag = (os.getenv("MINIO_ANONYMOUS") or "").strip().lower()
|
||||
if anon_flag in ("0", "false", "no"):
|
||||
anonymous = False
|
||||
elif anon_flag in ("1", "true", "yes"):
|
||||
anonymous = True
|
||||
else:
|
||||
anonymous = not (access_key and secret_key)
|
||||
return cls(
|
||||
endpoint=os.getenv("MINIO_ENDPOINT", "https://s3.builker.xyz").rstrip("/"),
|
||||
bucket=os.getenv("MINIO_BUCKET", "pys-cuentas-medicas"),
|
||||
access_key=access_key,
|
||||
secret_key=secret_key,
|
||||
region=os.getenv("MINIO_REGION", "us-east-1"),
|
||||
anonymous=anonymous,
|
||||
)
|
||||
|
||||
|
||||
def excel_key(job_id: str, filename: str = "input.xlsx") -> str:
|
||||
return f"uploads/{job_id}/{filename}"
|
||||
|
||||
|
||||
def downloads_prefix(job_id: str) -> str:
|
||||
return f"downloads/{job_id}/"
|
||||
|
||||
|
||||
def pdf_key(job_id: str, nit: str, radicado: str) -> str:
|
||||
return f"downloads/{job_id}/{nit}/{radicado}.pdf"
|
||||
|
||||
|
||||
def meta_key(job_id: str) -> str:
|
||||
return f"uploads/{job_id}/meta.json"
|
||||
|
||||
|
||||
def manifest_key(job_id: str) -> str:
|
||||
return f"downloads/{job_id}/_manifest.json"
|
||||
|
||||
|
||||
def _s3_client(settings: MinioSettings):
|
||||
import boto3
|
||||
from botocore import UNSIGNED
|
||||
from botocore.client import Config
|
||||
|
||||
s3_cfg = {
|
||||
"addressing_style": "path",
|
||||
"request_checksum_calculation": "when_required",
|
||||
"response_checksum_validation": "when_required",
|
||||
}
|
||||
if settings.anonymous:
|
||||
return boto3.client(
|
||||
"s3",
|
||||
endpoint_url=settings.endpoint,
|
||||
region_name=settings.region,
|
||||
config=Config(signature_version=UNSIGNED, s3=s3_cfg),
|
||||
)
|
||||
return boto3.client(
|
||||
"s3",
|
||||
endpoint_url=settings.endpoint,
|
||||
aws_access_key_id=settings.access_key,
|
||||
aws_secret_access_key=settings.secret_key,
|
||||
region_name=settings.region,
|
||||
config=Config(signature_version="s3v4", s3=s3_cfg),
|
||||
)
|
||||
|
||||
|
||||
class MinioStorage:
|
||||
"""Cliente S3-compatible para MinIO (uploads/downloads por job_id)."""
|
||||
|
||||
def __init__(self, settings: Optional[MinioSettings] = None):
|
||||
self.settings = settings or MinioSettings.from_env()
|
||||
self._client = None
|
||||
|
||||
@property
|
||||
def client(self):
|
||||
if self._client is None:
|
||||
self._client = _s3_client(self.settings)
|
||||
return self._client
|
||||
|
||||
@property
|
||||
def bucket(self) -> str:
|
||||
return self.settings.bucket
|
||||
|
||||
def put_bytes(self, key: str, data: bytes, content_type: str = "application/octet-stream") -> str:
|
||||
self.client.put_object(
|
||||
Bucket=self.bucket,
|
||||
Key=key,
|
||||
Body=data,
|
||||
ContentType=content_type,
|
||||
)
|
||||
return key
|
||||
|
||||
def put_file(self, key: str, path: str | Path, content_type: str = "application/octet-stream") -> str:
|
||||
path = Path(path)
|
||||
return self.put_bytes(key, path.read_bytes(), content_type=content_type)
|
||||
|
||||
def put_stream(self, key: str, stream: BinaryIO, content_type: str = "application/octet-stream") -> str:
|
||||
data = stream.read()
|
||||
return self.put_bytes(key, data, content_type=content_type)
|
||||
|
||||
def get_bytes(self, key: str) -> bytes:
|
||||
obj = self.client.get_object(Bucket=self.bucket, Key=key)
|
||||
return obj["Body"].read()
|
||||
|
||||
def download_file(self, key: str, dest: str | Path) -> Path:
|
||||
dest = Path(dest)
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
self.client.download_file(self.bucket, key, str(dest))
|
||||
return dest
|
||||
|
||||
def exists(self, key: str) -> bool:
|
||||
try:
|
||||
self.client.head_object(Bucket=self.bucket, Key=key)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def list_keys(self, prefix: str) -> List[str]:
|
||||
keys: List[str] = []
|
||||
token = None
|
||||
while True:
|
||||
kwargs = {"Bucket": self.bucket, "Prefix": prefix}
|
||||
if token:
|
||||
kwargs["ContinuationToken"] = token
|
||||
resp = self.client.list_objects_v2(**kwargs)
|
||||
for item in resp.get("Contents") or []:
|
||||
keys.append(item["Key"])
|
||||
if not resp.get("IsTruncated"):
|
||||
break
|
||||
token = resp.get("NextContinuationToken")
|
||||
return keys
|
||||
|
||||
def presigned_get(self, key: str, expires_in: int = 3600) -> str:
|
||||
if self.settings.anonymous:
|
||||
return f"{self.settings.endpoint}/{self.bucket}/{key}"
|
||||
return self.client.generate_presigned_url(
|
||||
"get_object",
|
||||
Params={"Bucket": self.bucket, "Key": key},
|
||||
ExpiresIn=expires_in,
|
||||
)
|
||||
|
||||
def upload_excel(
|
||||
self,
|
||||
job_id: str,
|
||||
data: bytes,
|
||||
original_filename: str,
|
||||
content_type: str = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
) -> str:
|
||||
ext = Path(original_filename).suffix.lower() or ".xlsx"
|
||||
canonical = f"input{ext}" if ext in (".xlsx", ".xls", ".csv") else "input.xlsx"
|
||||
key = excel_key(job_id, canonical)
|
||||
self.put_bytes(key, data, content_type=content_type)
|
||||
meta = {
|
||||
"original_filename": original_filename,
|
||||
"size": len(data),
|
||||
"canonical": canonical,
|
||||
}
|
||||
self.put_bytes(meta_key(job_id), json.dumps(meta).encode("utf-8"), "application/json")
|
||||
return key
|
||||
|
||||
def upload_pdf(self, job_id: str, nit: str, radicado: str, local_path: str | Path) -> str:
|
||||
key = pdf_key(job_id, nit, radicado)
|
||||
self.put_file(key, local_path, content_type="application/pdf")
|
||||
return key
|
||||
|
||||
def write_manifest(self, job_id: str, keys: List[str]) -> str:
|
||||
key = manifest_key(job_id)
|
||||
payload = {"job_id": job_id, "files": keys, "count": len(keys)}
|
||||
self.put_bytes(key, json.dumps(payload, ensure_ascii=False).encode("utf-8"), "application/json")
|
||||
return key
|
||||
|
||||
def list_download_files(self, job_id: str, prefix: str | None = None) -> dict:
|
||||
"""Lista PDFs de una corrida (manifest o list_keys)."""
|
||||
prefix = prefix or downloads_prefix(job_id)
|
||||
files: List[dict] = []
|
||||
try:
|
||||
raw = self.get_bytes(manifest_key(job_id))
|
||||
payload = json.loads(raw.decode("utf-8"))
|
||||
for key in payload.get("files") or []:
|
||||
files.append(self._file_entry(str(key)))
|
||||
except Exception:
|
||||
for key in self.list_keys(prefix):
|
||||
if key.endswith("/_manifest.json") or key.endswith("_manifest.json"):
|
||||
continue
|
||||
files.append(self._file_entry(key))
|
||||
return {"job_id": job_id, "prefix": prefix, "files": files, "count": len(files)}
|
||||
|
||||
@staticmethod
|
||||
def _file_entry(key: str) -> dict:
|
||||
# downloads/{job_id}/{nit}/{radicado}.pdf
|
||||
parts = key.strip("/").split("/")
|
||||
nit = parts[-2] if len(parts) >= 3 else None
|
||||
name = parts[-1] if parts else key
|
||||
radicado = name[:-4] if name.lower().endswith(".pdf") else name
|
||||
return {"key": key, "nit": nit, "radicado": radicado}
|
||||
@@ -0,0 +1,466 @@
|
||||
"""Repositorio sobre PostgreSQL (tablas download_* + sync de tasks)."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from domain.models import (
|
||||
Job,
|
||||
JobEventRecord,
|
||||
JobStatus,
|
||||
Worker,
|
||||
WorkerStatus,
|
||||
utc_now,
|
||||
)
|
||||
from infrastructure.database import (
|
||||
DownloadJobEventRow,
|
||||
DownloadJobRow,
|
||||
DownloadWorkerRow,
|
||||
Task,
|
||||
_iso,
|
||||
init_db,
|
||||
session_scope,
|
||||
utcnow,
|
||||
)
|
||||
from infrastructure.storage_minio import downloads_prefix
|
||||
|
||||
HEARTBEAT_OFFLINE_SECONDS = 45
|
||||
|
||||
|
||||
def _parse_iso(value: str | None) -> datetime | None:
|
||||
if not value:
|
||||
return None
|
||||
dt = datetime.fromisoformat(value)
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
return dt
|
||||
|
||||
|
||||
class JobStore:
|
||||
"""API de persistencia compatible con el store SQLite anterior."""
|
||||
|
||||
def __init__(self, db_path: str | None = None):
|
||||
# db_path ignorado (compat); usa DATABASE_URL.
|
||||
init_db()
|
||||
|
||||
def _row_worker(self, row: DownloadWorkerRow) -> Worker:
|
||||
worker = Worker(
|
||||
id=row.id,
|
||||
hostname=row.hostname or "",
|
||||
max_concurrent=int(row.max_concurrent or 1),
|
||||
status=row.status,
|
||||
last_heartbeat=_iso(row.last_heartbeat) or utc_now(),
|
||||
current_job_id=row.current_job_id,
|
||||
)
|
||||
return self._refresh_worker_status(worker)
|
||||
|
||||
def _row_job(self, row: DownloadJobRow) -> Job:
|
||||
return Job(
|
||||
id=row.id,
|
||||
worker_id=row.worker_id,
|
||||
status=row.status,
|
||||
title=row.title or "",
|
||||
excel_key=row.excel_key or "",
|
||||
downloads_prefix=row.downloads_prefix or "",
|
||||
original_filename=row.original_filename or "",
|
||||
cancel_requested=bool(row.cancel_requested),
|
||||
result=dict(row.result or {}),
|
||||
error=row.error,
|
||||
created_at=_iso(row.created_at) or utc_now(),
|
||||
started_at=_iso(row.started_at),
|
||||
finished_at=_iso(row.finished_at),
|
||||
job_id=row.job_id,
|
||||
task_id=row.task_id,
|
||||
)
|
||||
|
||||
def _refresh_worker_status(self, worker: Worker) -> Worker:
|
||||
try:
|
||||
hb = datetime.fromisoformat(worker.last_heartbeat)
|
||||
if hb.tzinfo is None:
|
||||
hb = hb.replace(tzinfo=timezone.utc)
|
||||
age = datetime.now(timezone.utc) - hb.astimezone(timezone.utc)
|
||||
if age > timedelta(seconds=HEARTBEAT_OFFLINE_SECONDS):
|
||||
worker.status = WorkerStatus.OFFLINE.value
|
||||
elif worker.current_job_id:
|
||||
worker.status = WorkerStatus.BUSY.value
|
||||
else:
|
||||
worker.status = WorkerStatus.ONLINE.value
|
||||
except Exception:
|
||||
worker.status = WorkerStatus.OFFLINE.value
|
||||
return worker
|
||||
|
||||
def register_worker(
|
||||
self,
|
||||
worker_id: str,
|
||||
hostname: str = "",
|
||||
max_concurrent: int = 1,
|
||||
) -> Worker:
|
||||
with session_scope() as session:
|
||||
row = session.get(DownloadWorkerRow, worker_id)
|
||||
now = utcnow()
|
||||
if row:
|
||||
row.hostname = hostname
|
||||
row.max_concurrent = max_concurrent
|
||||
row.status = WorkerStatus.ONLINE.value
|
||||
row.last_heartbeat = now
|
||||
row.current_job_id = None
|
||||
else:
|
||||
row = DownloadWorkerRow(
|
||||
id=worker_id,
|
||||
hostname=hostname,
|
||||
max_concurrent=max_concurrent,
|
||||
status=WorkerStatus.ONLINE.value,
|
||||
last_heartbeat=now,
|
||||
current_job_id=None,
|
||||
)
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return self._row_worker(row)
|
||||
|
||||
def heartbeat(self, worker_id: str) -> Optional[Worker]:
|
||||
with session_scope() as session:
|
||||
row = session.get(DownloadWorkerRow, worker_id)
|
||||
if not row:
|
||||
return None
|
||||
row.last_heartbeat = utcnow()
|
||||
row.status = (
|
||||
WorkerStatus.BUSY.value if row.current_job_id else WorkerStatus.ONLINE.value
|
||||
)
|
||||
session.flush()
|
||||
return self._row_worker(row)
|
||||
|
||||
def get_worker(self, worker_id: str) -> Optional[Worker]:
|
||||
with session_scope() as session:
|
||||
row = session.get(DownloadWorkerRow, worker_id)
|
||||
if not row:
|
||||
return None
|
||||
return self._row_worker(row)
|
||||
|
||||
def list_workers(self) -> List[Worker]:
|
||||
with session_scope() as session:
|
||||
rows = session.scalars(
|
||||
select(DownloadWorkerRow).order_by(DownloadWorkerRow.id)
|
||||
).all()
|
||||
return [self._row_worker(r) for r in rows]
|
||||
|
||||
def is_worker_online(self, worker_id: str) -> bool:
|
||||
worker = self.get_worker(worker_id)
|
||||
if not worker:
|
||||
return False
|
||||
return worker.status in (WorkerStatus.ONLINE.value, WorkerStatus.BUSY.value)
|
||||
|
||||
def update_excel_key(self, job_id: str, excel_key: str) -> Optional[Job]:
|
||||
with session_scope() as session:
|
||||
row = session.get(DownloadJobRow, job_id)
|
||||
if not row:
|
||||
return None
|
||||
row.excel_key = excel_key
|
||||
session.flush()
|
||||
return self._row_job(row)
|
||||
|
||||
def create_job(
|
||||
self,
|
||||
worker_id: str,
|
||||
excel_key: str,
|
||||
original_filename: str = "",
|
||||
title: str = "",
|
||||
*,
|
||||
job_id: Optional[str] = None,
|
||||
task_id: Optional[str] = None,
|
||||
) -> Job:
|
||||
import uuid as uuid_mod
|
||||
|
||||
download_id = str(uuid_mod.uuid4())
|
||||
prefix = downloads_prefix(download_id)
|
||||
with session_scope() as session:
|
||||
row = DownloadJobRow(
|
||||
id=download_id,
|
||||
job_id=job_id,
|
||||
task_id=task_id,
|
||||
worker_id=worker_id,
|
||||
status=JobStatus.PENDING.value,
|
||||
title=title or f"Descarga {original_filename}",
|
||||
excel_key=excel_key,
|
||||
downloads_prefix=prefix,
|
||||
original_filename=original_filename,
|
||||
cancel_requested=False,
|
||||
result={},
|
||||
created_at=utcnow(),
|
||||
)
|
||||
session.add(row)
|
||||
session.flush()
|
||||
return self._row_job(row)
|
||||
|
||||
def get_job(self, job_id: str) -> Optional[Job]:
|
||||
with session_scope() as session:
|
||||
row = session.get(DownloadJobRow, job_id)
|
||||
if not row:
|
||||
return None
|
||||
return self._row_job(row)
|
||||
|
||||
def list_jobs(
|
||||
self,
|
||||
limit: int = 100,
|
||||
*,
|
||||
job_id: Optional[str] = None,
|
||||
) -> List[Job]:
|
||||
with session_scope() as session:
|
||||
stmt = select(DownloadJobRow).order_by(DownloadJobRow.created_at.desc()).limit(limit)
|
||||
if job_id:
|
||||
stmt = (
|
||||
select(DownloadJobRow)
|
||||
.where(DownloadJobRow.job_id == job_id)
|
||||
.order_by(DownloadJobRow.created_at.desc())
|
||||
.limit(limit)
|
||||
)
|
||||
rows = session.scalars(stmt).all()
|
||||
return [self._row_job(r) for r in rows]
|
||||
|
||||
def claim_pending(self, worker_id: str) -> Optional[Job]:
|
||||
with session_scope() as session:
|
||||
worker = session.get(DownloadWorkerRow, worker_id)
|
||||
if not worker or worker.current_job_id:
|
||||
return None
|
||||
|
||||
row = session.scalars(
|
||||
select(DownloadJobRow)
|
||||
.where(
|
||||
DownloadJobRow.worker_id == worker_id,
|
||||
DownloadJobRow.status == JobStatus.PENDING.value,
|
||||
)
|
||||
.order_by(DownloadJobRow.created_at.asc())
|
||||
.limit(1)
|
||||
).first()
|
||||
if not row:
|
||||
return None
|
||||
|
||||
now = utcnow()
|
||||
row.status = JobStatus.RUNNING.value
|
||||
row.started_at = now
|
||||
worker.current_job_id = row.id
|
||||
worker.status = WorkerStatus.BUSY.value
|
||||
worker.last_heartbeat = now
|
||||
|
||||
if row.task_id:
|
||||
task = session.get(Task, row.task_id)
|
||||
if task:
|
||||
task.status = JobStatus.RUNNING.value
|
||||
task.started_at = now
|
||||
task.updated_at = now
|
||||
|
||||
session.flush()
|
||||
return self._row_job(row)
|
||||
|
||||
def request_cancel(self, job_id: str) -> Optional[Job]:
|
||||
with session_scope() as session:
|
||||
row = session.get(DownloadJobRow, job_id)
|
||||
if not row:
|
||||
return None
|
||||
if row.status in (
|
||||
JobStatus.COMPLETED.value,
|
||||
JobStatus.FAILED.value,
|
||||
JobStatus.CANCELLED.value,
|
||||
):
|
||||
return self._row_job(row)
|
||||
|
||||
now = utcnow()
|
||||
row.cancel_requested = True
|
||||
if row.status == JobStatus.PENDING.value:
|
||||
row.status = JobStatus.CANCELLED.value
|
||||
row.finished_at = now
|
||||
self._sync_task(session, row, status=JobStatus.CANCELLED.value)
|
||||
session.flush()
|
||||
return self._row_job(row)
|
||||
|
||||
def is_cancel_requested(self, job_id: str) -> bool:
|
||||
job = self.get_job(job_id)
|
||||
return bool(job and job.cancel_requested)
|
||||
|
||||
def _sync_task(
|
||||
self,
|
||||
session,
|
||||
row: DownloadJobRow,
|
||||
*,
|
||||
status: Optional[str] = None,
|
||||
result: Optional[Dict[str, Any]] = None,
|
||||
error: Optional[str] = None,
|
||||
) -> None:
|
||||
if not row.task_id:
|
||||
return
|
||||
task = session.get(Task, row.task_id)
|
||||
if not task:
|
||||
return
|
||||
now = utcnow()
|
||||
if status:
|
||||
task.status = status
|
||||
if result is not None:
|
||||
task.result = {
|
||||
**dict(task.result or {}),
|
||||
"download_job_id": row.id,
|
||||
**result,
|
||||
}
|
||||
if error is not None:
|
||||
task.error = error
|
||||
task.updated_at = now
|
||||
if status in (
|
||||
JobStatus.COMPLETED.value,
|
||||
JobStatus.FAILED.value,
|
||||
JobStatus.CANCELLED.value,
|
||||
):
|
||||
task.finished_at = now
|
||||
|
||||
def patch_job(
|
||||
self,
|
||||
job_id: str,
|
||||
*,
|
||||
status: Optional[str] = None,
|
||||
result: Optional[Dict[str, Any]] = None,
|
||||
error: Optional[str] = None,
|
||||
) -> Optional[Job]:
|
||||
with session_scope() as session:
|
||||
row = session.get(DownloadJobRow, job_id)
|
||||
if not row:
|
||||
return None
|
||||
|
||||
new_status = status or row.status
|
||||
if result is not None:
|
||||
row.result = result
|
||||
if error is not None:
|
||||
row.error = error
|
||||
row.status = new_status
|
||||
|
||||
finished = False
|
||||
if new_status in (
|
||||
JobStatus.COMPLETED.value,
|
||||
JobStatus.FAILED.value,
|
||||
JobStatus.CANCELLED.value,
|
||||
):
|
||||
row.finished_at = row.finished_at or utcnow()
|
||||
finished = True
|
||||
|
||||
self._sync_task(
|
||||
session,
|
||||
row,
|
||||
status=new_status if status else None,
|
||||
result=result,
|
||||
error=error,
|
||||
)
|
||||
|
||||
if finished:
|
||||
worker = session.get(DownloadWorkerRow, row.worker_id)
|
||||
if worker and worker.current_job_id == job_id:
|
||||
worker.current_job_id = None
|
||||
worker.status = WorkerStatus.ONLINE.value
|
||||
|
||||
session.flush()
|
||||
return self._row_job(row)
|
||||
|
||||
def add_event(
|
||||
self,
|
||||
job_id: str,
|
||||
event_type: str,
|
||||
message: str,
|
||||
*,
|
||||
radicado: Optional[str] = None,
|
||||
nit: Optional[str] = None,
|
||||
index: Optional[int] = None,
|
||||
total: Optional[int] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
timestamp: Optional[str] = None,
|
||||
) -> JobEventRecord:
|
||||
with session_scope() as session:
|
||||
ts = _parse_iso(timestamp) or utcnow()
|
||||
ev = DownloadJobEventRow(
|
||||
download_job_id=job_id,
|
||||
event_type=event_type,
|
||||
message=message,
|
||||
timestamp=ts,
|
||||
radicado=radicado,
|
||||
nit=nit,
|
||||
idx=index,
|
||||
total=total,
|
||||
metadata_json=metadata or {},
|
||||
)
|
||||
session.add(ev)
|
||||
session.flush()
|
||||
return JobEventRecord(
|
||||
id=int(ev.id),
|
||||
job_id=job_id,
|
||||
event_type=event_type,
|
||||
message=message,
|
||||
timestamp=_iso(ts) or utc_now(),
|
||||
radicado=radicado,
|
||||
nit=nit,
|
||||
index=index,
|
||||
total=total,
|
||||
metadata=metadata or {},
|
||||
)
|
||||
|
||||
def list_events(self, job_id: str, since_id: int = 0) -> List[JobEventRecord]:
|
||||
with session_scope() as session:
|
||||
rows = session.scalars(
|
||||
select(DownloadJobEventRow)
|
||||
.where(
|
||||
DownloadJobEventRow.download_job_id == job_id,
|
||||
DownloadJobEventRow.id > since_id,
|
||||
)
|
||||
.order_by(DownloadJobEventRow.id.asc())
|
||||
).all()
|
||||
return [
|
||||
JobEventRecord(
|
||||
id=int(r.id),
|
||||
job_id=r.download_job_id,
|
||||
event_type=r.event_type,
|
||||
message=r.message,
|
||||
timestamp=_iso(r.timestamp) or utc_now(),
|
||||
radicado=r.radicado,
|
||||
nit=r.nit,
|
||||
index=r.idx,
|
||||
total=r.total,
|
||||
metadata=dict(r.metadata_json or {}),
|
||||
)
|
||||
for r in rows
|
||||
]
|
||||
|
||||
def create_downloads_task(
|
||||
self,
|
||||
*,
|
||||
parent_job_id: str,
|
||||
download_job_id: str,
|
||||
worker_id: str,
|
||||
title: str,
|
||||
original_filename: str,
|
||||
) -> str:
|
||||
"""Crea Task(type=downloads) y la asocia al download_job. Devuelve task_id."""
|
||||
with session_scope() as session:
|
||||
now = utcnow()
|
||||
task = Task(
|
||||
job_id=parent_job_id,
|
||||
type="downloads",
|
||||
status=JobStatus.PENDING.value,
|
||||
title=title or f"Bot descarga ({original_filename})",
|
||||
request={
|
||||
"worker_id": worker_id,
|
||||
"original_filename": original_filename,
|
||||
"download_job_id": download_job_id,
|
||||
},
|
||||
result={"download_job_id": download_job_id},
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
)
|
||||
session.add(task)
|
||||
session.flush()
|
||||
row = session.get(DownloadJobRow, download_job_id)
|
||||
if row:
|
||||
row.task_id = task.id
|
||||
row.job_id = parent_job_id
|
||||
session.flush()
|
||||
return task.id
|
||||
|
||||
def parent_job_exists(self, parent_job_id: str) -> bool:
|
||||
from infrastructure.database import Job as IpsJob
|
||||
|
||||
with session_scope() as session:
|
||||
return session.get(IpsJob, parent_job_id) is not None
|
||||
@@ -0,0 +1,9 @@
|
||||
fastapi>=0.110.0
|
||||
uvicorn[standard]>=0.27.0
|
||||
python-multipart>=0.0.9
|
||||
boto3>=1.34.0
|
||||
httpx>=0.27.0
|
||||
pydantic>=2.0.0
|
||||
sqlalchemy>=2.0.0
|
||||
psycopg[binary]>=3.1.0
|
||||
pytest>=7.0.0
|
||||
@@ -0,0 +1,151 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from infrastructure.storage_minio import downloads_prefix, excel_key, pdf_key
|
||||
|
||||
|
||||
def test_minio_key_helpers():
|
||||
assert excel_key("abc", "input.xlsx") == "uploads/abc/input.xlsx"
|
||||
assert downloads_prefix("abc") == "downloads/abc/"
|
||||
assert pdf_key("abc", "900", "123") == "downloads/abc/900/123.pdf"
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def store(tmp_path, monkeypatch):
|
||||
db_url = f"sqlite:///{(tmp_path / 'bot.db').as_posix()}"
|
||||
monkeypatch.setenv("DATABASE_URL", db_url)
|
||||
|
||||
# Recargar módulos para que el engine use SQLITE
|
||||
for name in [
|
||||
"infrastructure.database",
|
||||
"infrastructure.store",
|
||||
]:
|
||||
if name in sys.modules:
|
||||
importlib.reload(sys.modules[name])
|
||||
else:
|
||||
importlib.import_module(name)
|
||||
|
||||
from domain.models import JobStatus, WorkerStatus
|
||||
from infrastructure import database as db
|
||||
from infrastructure.store import JobStore
|
||||
|
||||
importlib.reload(db)
|
||||
import infrastructure.store as store_mod
|
||||
|
||||
importlib.reload(store_mod)
|
||||
return store_mod.JobStore(), JobStatus, WorkerStatus
|
||||
|
||||
|
||||
def test_register_heartbeat_and_offline(store):
|
||||
job_store, _JobStatus, WorkerStatus = store
|
||||
w = job_store.register_worker("PC-01", hostname="host1")
|
||||
assert w.status == WorkerStatus.ONLINE.value
|
||||
assert job_store.is_worker_online("PC-01")
|
||||
|
||||
job_store.heartbeat("PC-01")
|
||||
workers = job_store.list_workers()
|
||||
assert len(workers) == 1
|
||||
assert workers[0].id == "PC-01"
|
||||
|
||||
|
||||
def test_create_job_requires_worker_and_claim_is_atomic(store):
|
||||
job_store, JobStatus, _WorkerStatus = store
|
||||
job_store.register_worker("PC-01")
|
||||
|
||||
job = job_store.create_job(
|
||||
worker_id="PC-01",
|
||||
excel_key="uploads/x/input.xlsx",
|
||||
original_filename="datos.xlsx",
|
||||
title="Lote 1",
|
||||
)
|
||||
assert job.status == JobStatus.PENDING.value
|
||||
assert job.worker_id == "PC-01"
|
||||
assert job.downloads_prefix == downloads_prefix(job.id)
|
||||
|
||||
claimed = job_store.claim_pending("PC-01")
|
||||
assert claimed is not None
|
||||
assert claimed.id == job.id
|
||||
assert claimed.status == JobStatus.RUNNING.value
|
||||
|
||||
again = job_store.claim_pending("PC-01")
|
||||
assert again is None
|
||||
|
||||
job_store.register_worker("PC-02")
|
||||
other = job_store.claim_pending("PC-02")
|
||||
assert other is None
|
||||
|
||||
|
||||
def test_cancel_pending_marks_cancelled(store):
|
||||
job_store, JobStatus, _WorkerStatus = store
|
||||
job_store.register_worker("PC-01")
|
||||
job = job_store.create_job("PC-01", "uploads/j/input.xlsx", "a.xlsx")
|
||||
cancelled = job_store.request_cancel(job.id)
|
||||
assert cancelled is not None
|
||||
assert cancelled.status == JobStatus.CANCELLED.value
|
||||
assert cancelled.cancel_requested is True
|
||||
|
||||
|
||||
def test_cancel_running_sets_flag(store):
|
||||
job_store, JobStatus, _WorkerStatus = store
|
||||
job_store.register_worker("PC-01")
|
||||
job = job_store.create_job("PC-01", "uploads/j/input.xlsx", "a.xlsx")
|
||||
job_store.claim_pending("PC-01")
|
||||
flagged = job_store.request_cancel(job.id)
|
||||
assert flagged is not None
|
||||
assert flagged.cancel_requested is True
|
||||
assert flagged.status == JobStatus.RUNNING.value
|
||||
assert job_store.is_cancel_requested(job.id) is True
|
||||
|
||||
|
||||
def test_events_and_patch_complete(store):
|
||||
job_store, JobStatus, _WorkerStatus = store
|
||||
job_store.register_worker("PC-01")
|
||||
job = job_store.create_job("PC-01", "uploads/j/input.xlsx", "a.xlsx")
|
||||
job_store.claim_pending("PC-01")
|
||||
e1 = job_store.add_event(
|
||||
job.id, "radicado_saved", "ok", radicado="1", nit="9", index=1, total=2
|
||||
)
|
||||
assert e1.id >= 1
|
||||
events = job_store.list_events(job.id, since_id=0)
|
||||
assert len(events) == 1
|
||||
|
||||
done = job_store.patch_job(
|
||||
job.id,
|
||||
status=JobStatus.COMPLETED.value,
|
||||
result={"exitosos": 1},
|
||||
)
|
||||
assert done is not None
|
||||
assert done.status == JobStatus.COMPLETED.value
|
||||
worker = job_store.get_worker("PC-01")
|
||||
assert worker is not None
|
||||
assert worker.current_job_id is None
|
||||
|
||||
|
||||
def test_list_jobs_by_parent(store):
|
||||
job_store, _JobStatus, _WorkerStatus = store
|
||||
from infrastructure.database import Job as IpsJob
|
||||
from infrastructure.database import session_scope, utcnow
|
||||
|
||||
job_store.register_worker("PC-01")
|
||||
with session_scope() as session:
|
||||
session.add(
|
||||
IpsJob(
|
||||
id="parent-1",
|
||||
status="pending",
|
||||
title="Trabajo",
|
||||
job_metadata={},
|
||||
created_at=utcnow(),
|
||||
updated_at=utcnow(),
|
||||
)
|
||||
)
|
||||
|
||||
a = job_store.create_job("PC-01", "uploads/a/input.xlsx", "a.xlsx", job_id="parent-1")
|
||||
job_store.create_job("PC-01", "uploads/b/input.xlsx", "b.xlsx", job_id="other")
|
||||
listed = job_store.list_jobs(limit=50, job_id="parent-1")
|
||||
assert len(listed) == 1
|
||||
assert listed[0].id == a.id
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Bot descarga: download_workers, download_jobs, download_job_events.
|
||||
|
||||
Revision ID: 0004
|
||||
Revises: 0003
|
||||
Create Date: 2026-07-12
|
||||
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision: str = "0004"
|
||||
down_revision: Union[str, None] = "0003"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def _table_exists(conn: sa.engine.Connection, table: str) -> bool:
|
||||
return sa.inspect(conn).has_table(table)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
conn = op.get_bind()
|
||||
json_type = (
|
||||
postgresql.JSONB()
|
||||
if conn.dialect.name == "postgresql"
|
||||
else sa.JSON()
|
||||
)
|
||||
|
||||
if not _table_exists(conn, "download_workers"):
|
||||
op.create_table(
|
||||
"download_workers",
|
||||
sa.Column("id", sa.String(36), primary_key=True),
|
||||
sa.Column("hostname", sa.Text(), nullable=False, server_default=""),
|
||||
sa.Column("max_concurrent", sa.Integer(), nullable=False, server_default="1"),
|
||||
sa.Column("status", sa.String(32), nullable=False),
|
||||
sa.Column("last_heartbeat", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("current_job_id", sa.String(36), nullable=True),
|
||||
)
|
||||
op.create_index("ix_download_workers_status", "download_workers", ["status"])
|
||||
|
||||
if not _table_exists(conn, "download_jobs"):
|
||||
op.create_table(
|
||||
"download_jobs",
|
||||
sa.Column("id", sa.String(36), primary_key=True),
|
||||
sa.Column("job_id", sa.String(36), sa.ForeignKey("jobs.id"), nullable=True),
|
||||
sa.Column("task_id", sa.String(36), sa.ForeignKey("tasks.id"), nullable=True),
|
||||
sa.Column("worker_id", sa.String(36), nullable=False),
|
||||
sa.Column("status", sa.String(32), nullable=False, server_default="pending"),
|
||||
sa.Column("title", sa.Text(), nullable=True),
|
||||
sa.Column("excel_key", sa.Text(), nullable=False, server_default=""),
|
||||
sa.Column("downloads_prefix", sa.Text(), nullable=False, server_default=""),
|
||||
sa.Column("original_filename", sa.Text(), nullable=False, server_default=""),
|
||||
sa.Column("cancel_requested", sa.Boolean(), nullable=False, server_default=sa.false()),
|
||||
sa.Column("result", json_type, nullable=False, server_default=sa.text("'{}'::jsonb") if conn.dialect.name == "postgresql" else sa.text("'{}'")),
|
||||
sa.Column("error", sa.Text(), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("started_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
|
||||
)
|
||||
op.create_index("ix_download_jobs_job_id", "download_jobs", ["job_id"])
|
||||
op.create_index("ix_download_jobs_task_id", "download_jobs", ["task_id"])
|
||||
op.create_index("ix_download_jobs_worker_id", "download_jobs", ["worker_id"])
|
||||
op.create_index("ix_download_jobs_status", "download_jobs", ["status"])
|
||||
op.create_index(
|
||||
"ix_download_jobs_worker_status",
|
||||
"download_jobs",
|
||||
["worker_id", "status"],
|
||||
)
|
||||
|
||||
if not _table_exists(conn, "download_job_events"):
|
||||
op.create_table(
|
||||
"download_job_events",
|
||||
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||
sa.Column(
|
||||
"download_job_id",
|
||||
sa.String(36),
|
||||
sa.ForeignKey("download_jobs.id"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column("event_type", sa.String(64), nullable=False),
|
||||
sa.Column("message", sa.Text(), nullable=False),
|
||||
sa.Column("timestamp", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("radicado", sa.String(255), nullable=True),
|
||||
sa.Column("nit", sa.String(64), nullable=True),
|
||||
sa.Column("idx", sa.Integer(), nullable=True),
|
||||
sa.Column("total", sa.Integer(), nullable=True),
|
||||
sa.Column(
|
||||
"metadata",
|
||||
json_type,
|
||||
nullable=False,
|
||||
server_default=sa.text("'{}'::jsonb") if conn.dialect.name == "postgresql" else sa.text("'{}'"),
|
||||
),
|
||||
)
|
||||
op.create_index(
|
||||
"ix_download_job_events_job_id",
|
||||
"download_job_events",
|
||||
["download_job_id", "id"],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
conn = op.get_bind()
|
||||
if _table_exists(conn, "download_job_events"):
|
||||
op.drop_index("ix_download_job_events_job_id", table_name="download_job_events")
|
||||
op.drop_table("download_job_events")
|
||||
if _table_exists(conn, "download_jobs"):
|
||||
op.drop_index("ix_download_jobs_worker_status", table_name="download_jobs")
|
||||
op.drop_index("ix_download_jobs_status", table_name="download_jobs")
|
||||
op.drop_index("ix_download_jobs_worker_id", table_name="download_jobs")
|
||||
op.drop_index("ix_download_jobs_task_id", table_name="download_jobs")
|
||||
op.drop_index("ix_download_jobs_job_id", table_name="download_jobs")
|
||||
op.drop_table("download_jobs")
|
||||
if _table_exists(conn, "download_workers"):
|
||||
op.drop_index("ix_download_workers_status", table_name="download_workers")
|
||||
op.drop_table("download_workers")
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "ipsextractor-desktop",
|
||||
"version": "1.0.3",
|
||||
"version": "1.1.0",
|
||||
"private": true,
|
||||
"description": "Desktop extraction client for IPSExtractor microservices.",
|
||||
"main": "electron/main.cjs",
|
||||
|
||||
@@ -29,6 +29,7 @@ import { UpdatePrompt, CheckUpdatesButton, useAutoUpdateCheck } from "./componen
|
||||
import "./modules/counting/CountingModule";
|
||||
import "./modules/extraction/ExtractionModule";
|
||||
import "./modules/cleaning/CleaningModule";
|
||||
import "./modules/downloads/DownloadsModule";
|
||||
|
||||
type Status = "idle" | "ok" | "error" | "busy";
|
||||
type View = "home" | "job" | "settings";
|
||||
@@ -39,7 +40,7 @@ function App() {
|
||||
const [settingsReturnView, setSettingsReturnView] = useState<"home" | "job">("home");
|
||||
const [settingsSection, setSettingsSection] = useState<SettingsSection>("connection");
|
||||
const [activeJob, setActiveJob] = useState<Job | null>(null);
|
||||
const [jobSection, setJobSection] = useState<JobSection>("counting");
|
||||
const [jobSection, setJobSection] = useState<JobSection>("downloads");
|
||||
const [jobSectionContext, setJobSectionContext] = useState<Record<string, unknown>>({});
|
||||
const [apiBaseUrl, setApiBaseUrl] = useState(loadApiBaseUrl);
|
||||
const [connectionStatus, setConnectionStatus] = useState<Status>("idle");
|
||||
@@ -109,7 +110,7 @@ function App() {
|
||||
|
||||
function openJob(job: Job) {
|
||||
setActiveJob(job);
|
||||
setJobSection("counting");
|
||||
setJobSection("downloads");
|
||||
setJobSectionContext({});
|
||||
setView("job");
|
||||
}
|
||||
@@ -201,6 +202,14 @@ function App() {
|
||||
</div>
|
||||
<div className="side-section-divider" />
|
||||
<nav className="side-nav" aria-label="Módulos del trabajo">
|
||||
<button
|
||||
type="button"
|
||||
className={jobSection === "downloads" ? "active" : ""}
|
||||
onClick={() => handleActivateSection("downloads")}
|
||||
>
|
||||
<Download size={17} />
|
||||
<span>Bot de descarga</span>
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
className={jobSection === "counting" ? "active" : ""}
|
||||
|
||||
@@ -9,6 +9,10 @@ import type {
|
||||
CountSyncResponse,
|
||||
CreateCountJobPayload,
|
||||
CreateJobPayload,
|
||||
DownloadJob,
|
||||
DownloadJobEvent,
|
||||
DownloadJobFilesResponse,
|
||||
DownloadWorker,
|
||||
ExtractionBatch,
|
||||
ExtractionFromCountPayload,
|
||||
Job,
|
||||
@@ -22,6 +26,16 @@ import { DEFAULT_API_BASE_URL } from "./apiDefaults";
|
||||
|
||||
const DEFAULT_TIMEOUT_MS = 30_000;
|
||||
const UPLOAD_TIMEOUT_MS = 600_000;
|
||||
const BOT_TOKEN_KEY = "botApiToken";
|
||||
const DEFAULT_BOT_TOKEN = "dev-token-change-me";
|
||||
|
||||
export function loadBotApiToken(): string {
|
||||
return (localStorage.getItem(BOT_TOKEN_KEY) || DEFAULT_BOT_TOKEN).trim();
|
||||
}
|
||||
|
||||
export function saveBotApiToken(token: string): void {
|
||||
localStorage.setItem(BOT_TOKEN_KEY, token.trim() || DEFAULT_BOT_TOKEN);
|
||||
}
|
||||
|
||||
async function fetchWithTimeout(
|
||||
url: string,
|
||||
@@ -48,19 +62,27 @@ async function fetchWithTimeout(
|
||||
}
|
||||
|
||||
export class ApiClient {
|
||||
constructor(private readonly baseUrl: string) {}
|
||||
constructor(
|
||||
private readonly baseUrl: string,
|
||||
private readonly botToken: string = loadBotApiToken(),
|
||||
) {}
|
||||
|
||||
private url(path: string): string {
|
||||
return `${this.baseUrl.replace(/\/+$/, "")}${path}`;
|
||||
}
|
||||
|
||||
private botHeaders(): HeadersInit {
|
||||
return { Authorization: `Bearer ${this.botToken}` };
|
||||
}
|
||||
|
||||
private isUploadPath(path: string): boolean {
|
||||
return (
|
||||
path.startsWith("/count/") ||
|
||||
path.includes("/ingest") ||
|
||||
path.includes("/upload") ||
|
||||
path.startsWith("/cleaning/analyze-csv") ||
|
||||
path.startsWith("/cleaning/jobs/from-csv")
|
||||
path.startsWith("/cleaning/jobs/from-csv") ||
|
||||
path === "/bot-descarga/jobs"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -194,6 +216,68 @@ export class ApiClient {
|
||||
await this.handle(response);
|
||||
}
|
||||
|
||||
// ── Bot Descarga (control remoto) ──────────────────────────────────────────
|
||||
|
||||
async listDownloadWorkers(): Promise<DownloadWorker[]> {
|
||||
const response = await fetchWithTimeout(
|
||||
this.url("/bot-descarga/workers"),
|
||||
{ headers: this.botHeaders() },
|
||||
this.timeoutFor("/bot-descarga/workers"),
|
||||
);
|
||||
return this.handle<DownloadWorker[]>(response);
|
||||
}
|
||||
|
||||
async createDownloadJob(form: FormData): Promise<DownloadJob> {
|
||||
const response = await fetchWithTimeout(
|
||||
this.url("/bot-descarga/jobs"),
|
||||
{ method: "POST", headers: this.botHeaders(), body: form },
|
||||
this.timeoutFor("/bot-descarga/jobs"),
|
||||
);
|
||||
return this.handle<DownloadJob>(response);
|
||||
}
|
||||
|
||||
async listDownloadJobs(limit = 50, parentJobId?: string): Promise<DownloadJob[]> {
|
||||
const qs = new URLSearchParams({ limit: String(limit) });
|
||||
if (parentJobId) qs.set("job_id", parentJobId);
|
||||
const response = await fetchWithTimeout(
|
||||
this.url(`/bot-descarga/jobs?${qs.toString()}`),
|
||||
{ headers: this.botHeaders() },
|
||||
);
|
||||
return this.handle<DownloadJob[]>(response);
|
||||
}
|
||||
|
||||
async getDownloadJob(jobId: string): Promise<DownloadJob> {
|
||||
const response = await fetchWithTimeout(
|
||||
this.url(`/bot-descarga/jobs/${encodeURIComponent(jobId)}`),
|
||||
{ headers: this.botHeaders() },
|
||||
);
|
||||
return this.handle<DownloadJob>(response);
|
||||
}
|
||||
|
||||
async listDownloadJobFiles(jobId: string): Promise<DownloadJobFilesResponse> {
|
||||
const response = await fetchWithTimeout(
|
||||
this.url(`/bot-descarga/jobs/${encodeURIComponent(jobId)}/files`),
|
||||
{ headers: this.botHeaders() },
|
||||
);
|
||||
return this.handle<DownloadJobFilesResponse>(response);
|
||||
}
|
||||
|
||||
async getDownloadEvents(jobId: string, since = 0): Promise<{ events: DownloadJobEvent[] }> {
|
||||
const response = await fetchWithTimeout(
|
||||
this.url(`/bot-descarga/jobs/${encodeURIComponent(jobId)}/events?since=${since}`),
|
||||
{ headers: this.botHeaders() },
|
||||
);
|
||||
return this.handle<{ events: DownloadJobEvent[] }>(response);
|
||||
}
|
||||
|
||||
async cancelDownloadJob(jobId: string): Promise<DownloadJob> {
|
||||
const response = await fetchWithTimeout(
|
||||
this.url(`/bot-descarga/jobs/${encodeURIComponent(jobId)}/cancel`),
|
||||
{ method: "POST", headers: this.botHeaders() },
|
||||
);
|
||||
return this.handle<DownloadJob>(response);
|
||||
}
|
||||
|
||||
// ── Count Jobs ─────────────────────────────────────────────────────────────
|
||||
|
||||
/** Envía PDFs al servicio de conteo (síncrono). */
|
||||
|
||||
@@ -1 +1,2 @@
|
||||
export type JobSection = "counting" | "extraction" | "cleaning" | "results";
|
||||
export type JobSection = "counting" | "extraction" | "cleaning" | "downloads" | "results";
|
||||
|
||||
|
||||
@@ -17,8 +17,9 @@ import { ExtractionResultsTable } from "../components/ExtractionResultsTable";
|
||||
import { BackgroundProgressBanner } from "../components/BackgroundProgressBanner";
|
||||
import type { ApiClient } from "../lib/apiClient";
|
||||
import { downloadCsv, toCsv, buildExportCsvFilename } from "../lib/csv";
|
||||
import { DownloadsResultsPanel } from "../modules/downloads/DownloadsResultsPanel";
|
||||
|
||||
type ResultTab = "counting" | "extraction" | "cleaning";
|
||||
type ResultTab = "counting" | "extraction" | "cleaning" | "downloads";
|
||||
|
||||
interface Props {
|
||||
client: ApiClient;
|
||||
@@ -31,6 +32,7 @@ interface Props {
|
||||
onStartCounting: () => void;
|
||||
onStartExtraction: (ctx?: Record<string, unknown>) => void;
|
||||
onStartCleaning: (ctx?: Record<string, unknown>) => void;
|
||||
onStartDownload: () => void;
|
||||
}
|
||||
|
||||
export function JobResultsView({
|
||||
@@ -44,6 +46,7 @@ export function JobResultsView({
|
||||
onStartCounting,
|
||||
onStartExtraction,
|
||||
onStartCleaning,
|
||||
onStartDownload,
|
||||
}: Props) {
|
||||
const [activeTab, setActiveTab] = useState<ResultTab>("counting");
|
||||
const [refreshing, setRefreshing] = useState(false);
|
||||
@@ -54,6 +57,8 @@ export function JobResultsView({
|
||||
.sort((a, b) => new Date(b.created_at).getTime() - new Date(a.created_at).getTime());
|
||||
const cleaningTasks = tasks.filter((t) => t.type === "cleaning")
|
||||
.sort((a, b) => new Date(b.created_at).getTime() - new Date(a.created_at).getTime());
|
||||
const downloadTasks = tasks.filter((t) => t.type === "downloads")
|
||||
.sort((a, b) => new Date(b.created_at).getTime() - new Date(a.created_at).getTime());
|
||||
|
||||
async function refresh() {
|
||||
setRefreshing(true);
|
||||
@@ -63,7 +68,7 @@ export function JobResultsView({
|
||||
|
||||
return (
|
||||
<div className="job-results-view">
|
||||
{/* ── Sub-tabs: Conteo | Extracción | Limpieza ── */}
|
||||
{/* ── Sub-tabs: Conteo | Extracción | Limpieza | Bot de descarga ── */}
|
||||
<div className="job-results-tabs">
|
||||
<button
|
||||
type="button"
|
||||
@@ -110,6 +115,17 @@ export function JobResultsView({
|
||||
</span>
|
||||
)}
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
className={`job-results-tab${activeTab === "downloads" ? " job-results-tab--active" : ""}`}
|
||||
onClick={() => setActiveTab("downloads")}
|
||||
>
|
||||
<Download size={15} />
|
||||
Bot de descarga
|
||||
{downloadTasks.length > 0 && (
|
||||
<span className="job-results-tab-count">{downloadTasks.length}</span>
|
||||
)}
|
||||
</button>
|
||||
<div style={{ flex: 1 }} />
|
||||
<button
|
||||
type="button"
|
||||
@@ -155,6 +171,14 @@ export function JobResultsView({
|
||||
onStartCleaning={() => onStartCleaning()}
|
||||
/>
|
||||
)}
|
||||
{activeTab === "downloads" && (
|
||||
<DownloadsResultsPanel
|
||||
client={client}
|
||||
job={job}
|
||||
tasks={downloadTasks}
|
||||
onStartDownload={onStartDownload}
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
|
||||
@@ -7,6 +7,7 @@ import { BackgroundProgressBanner } from "../components/BackgroundProgressBanner
|
||||
import { CountingWizardModule } from "../modules/counting/CountingWizardModule";
|
||||
import { CleaningWizardModule } from "../modules/cleaning/CleaningWizardModule";
|
||||
import { JobExtractionWizard } from "../modules/extraction/JobExtractionWizard";
|
||||
import { DownloadsWizardModule } from "../modules/downloads/DownloadsWizardModule";
|
||||
import { JobResultsView } from "./JobResultsView";
|
||||
|
||||
interface Props {
|
||||
@@ -120,6 +121,7 @@ export function JobView({ client, job, section, sectionContext, onActivateSectio
|
||||
{section === "counting" && "Conteo de servicios"}
|
||||
{section === "extraction" && "Extracción de datos"}
|
||||
{section === "cleaning" && "Limpieza de datos"}
|
||||
{section === "downloads" && "Bot de descarga"}
|
||||
{section === "results" && "Resultados del trabajo"}
|
||||
</div>
|
||||
</header>
|
||||
@@ -179,6 +181,10 @@ export function JobView({ client, job, section, sectionContext, onActivateSectio
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className={`job-section-pane${section === "downloads" ? "" : " job-section-pane--hidden"}`}>
|
||||
<DownloadsWizardModule client={client} job={job} />
|
||||
</div>
|
||||
|
||||
<div className={`job-section-pane${section === "results" ? "" : " job-section-pane--hidden"}`}>
|
||||
<JobResultsView
|
||||
client={client}
|
||||
@@ -191,6 +197,7 @@ export function JobView({ client, job, section, sectionContext, onActivateSectio
|
||||
onStartCounting={() => onActivateSection("counting")}
|
||||
onStartExtraction={(ctx?: Record<string, unknown>) => onActivateSection("extraction", ctx)}
|
||||
onStartCleaning={(ctx?: Record<string, unknown>) => onActivateSection("cleaning", ctx)}
|
||||
onStartDownload={() => onActivateSection("downloads")}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -303,6 +303,62 @@ export interface UpdateProgress {
|
||||
percent?: number;
|
||||
}
|
||||
|
||||
/** Worker del bot de descargas PDF (control remoto). */
|
||||
export interface DownloadWorker {
|
||||
id: string;
|
||||
hostname: string;
|
||||
max_concurrent: number;
|
||||
status: "online" | "offline" | "busy" | string;
|
||||
last_heartbeat: string;
|
||||
current_job_id: string | null;
|
||||
}
|
||||
|
||||
export interface DownloadJob {
|
||||
id: string;
|
||||
worker_id: string;
|
||||
status: "pending" | "running" | "completed" | "failed" | "cancelled" | string;
|
||||
title: string;
|
||||
excel_key: string;
|
||||
downloads_prefix: string;
|
||||
original_filename: string;
|
||||
cancel_requested: boolean;
|
||||
result: Record<string, unknown>;
|
||||
error: string | null;
|
||||
created_at: string;
|
||||
started_at: string | null;
|
||||
finished_at: string | null;
|
||||
/** Job Desktop contenedor (IPSExtractor jobs.id) */
|
||||
job_id?: string | null;
|
||||
/** Task type=downloads */
|
||||
task_id?: string | null;
|
||||
}
|
||||
|
||||
export interface DownloadJobFile {
|
||||
key: string;
|
||||
nit?: string | null;
|
||||
radicado?: string | null;
|
||||
}
|
||||
|
||||
export interface DownloadJobFilesResponse {
|
||||
job_id: string;
|
||||
prefix: string;
|
||||
files: DownloadJobFile[];
|
||||
count: number;
|
||||
}
|
||||
|
||||
export interface DownloadJobEvent {
|
||||
id: number;
|
||||
job_id: string;
|
||||
event_type: string;
|
||||
message: string;
|
||||
timestamp: string;
|
||||
radicado?: string | null;
|
||||
nit?: string | null;
|
||||
index?: number | null;
|
||||
total?: number | null;
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface DesktopBridge {
|
||||
getRuntimeConfig(): Promise<RuntimeConfig>;
|
||||
selectDirectory(): Promise<string | null>;
|
||||
|
||||
Reference in New Issue
Block a user