diff --git a/README.md b/README.md index 5ec29b7..e6fd8bd 100644 --- a/README.md +++ b/README.md @@ -1,117 +1,221 @@ -## IPSExtractor +## IPSExtractor Microservices -Sistema avanzado para la extracción y procesamiento de datos de facturas médicas (IPS) utilizando modelos de Inteligencia Artificial Multimodal (LLMs). Convierte documentos (PDF/TIFF) a imágenes, extrae información estructurada, la normaliza y la exporta a CSV. +Monorepo con servicios separados como si fueran repos independientes. Cada +servicio tiene su propia API, dependencias, Dockerfile, codigo de dominio e +infraestructura local. No hay paquetes compartidos entre servicios. -### 🚀 Características Principales +```text +services/ + extraction/ Repo/servicio de extraccion + api/ API FastAPI + domain/ Dominio propio de extraccion y normalizacion + core/ Utilidades propias para PDF, texto y filas + infrastructure/ Persistencia propia del servicio + requirements.txt + Dockerfile -- **Extracción Multimodal**: Usa modelos de visión (como `qwen2.5vl`) vía API compatible con OpenAI/Ollama para "leer" documentos como imágenes. -- **Procesamiento Inteligente**: - - Conversión automática de PDF/TIFF a imágenes optimizadas. - - Descubrimiento automático de documentos por radicado (prioriza TIFF). - - **Normalización de Datos**: Estandarización de salidas JSON mediante post-procesamiento con LLM. -- **Gestión Robusta**: - - Sistema de **Logs Estructurados** (JSON) y salida en consola coloreada para fácil depuración. - - Estimación de costos y tokens antes de ejecutar. - - Manejo de reintentos y "fallback" automático (reducción de DPI) en caso de fallos. -- **Interfaz Gráfica (GUI)**: - - Selección interactiva de carpetas y configuración. - - Selector visual de campos a extraer (Schema Builder). - - Barra de progreso y logs en tiempo real. -- **Exportación**: Generación incremental de reportes CSV (una fila por ítem de factura). + cleaning/ Repo/servicio de limpieza + api/ API FastAPI + domain/ Reglas propias de limpieza + infrastructure/ Persistencia propia del servicio + requirements.txt + Dockerfile -### 📋 Requisitos + api_gateway/ API central opcional + api/ Proxy HTTP hacia extraction y cleaning + requirements.txt + Dockerfile +``` -- **Python 3.10+** -- **Poppler** (para procesar PDFs): - - Windows: Incluido en `dependencies/poppler-24.08.0/Library/bin`. - - Linux/Mac: Instalar vía gestor de paquetes (`sudo apt install poppler-utils`). -- **Acceso a LLM**: Un servidor Ollama corriendo localmente o acceso a una API compatible con OpenAI. +La base de datos Postgres es compartida como contrato de integracion, pero el +codigo de acceso a datos esta duplicado dentro de cada servicio para que puedan +vivir por separado. -### 🛠️ Instalación +### Puertos -1. **Clonar el repositorio y crear entorno virtual**: - ```bash - python -m venv .venv - # Windows - .venv\Scripts\activate - # Linux/Mac - source .venv/bin/activate - ``` +| Servicio | Puerto | Uso | +| --- | ---: | --- | +| `api_gateway` | `8000` | API central opcional | +| `extraction` | `8001` | API propia de extraccion | +| `cleaning` | `8002` | API propia de limpieza | +| `database` | `5432` | Postgres | -2. **Instalar dependencias**: - ```bash - pip install -r requirements.txt - ``` +Documentacion interactiva: -### ⚙️ Configuración (.env) +- `http://localhost:8000/docs` +- `http://localhost:8001/docs` +- `http://localhost:8002/docs` -La configuración se gestiona mediante variables de entorno. Crea un archivo `.env` en la raíz con los siguientes parámetros (ajusta según tu entorno): +### Variables ```env -# Configuración del Modelo Extractor -EXTRACTOR_BASE_URL=http://10.8.0.4:11434/v1 -EXTRACTOR_API_KEY=ollama -EXTRACTOR_MODEL=qwen2.5vl:32b +IPS_DOCUMENT_ROOT=C:/ruta/a/documentos + +# Ejemplo: extraccion con Grok/xAI +EXTRACTOR_PROVIDER=xai +EXTRACTOR_BASE_URL=https://api.x.ai/v1 +EXTRACTOR_API_KEY=tu_xai_api_key +EXTRACTOR_MODEL=grok-tu-modelo EXTRACTOR_TIMEOUT_S=120 -# Configuración de Procesamiento de Imagen +# Ejemplo: normalizacion con OpenAI +NORMALIZER_PROVIDER=openai +NORMALIZER_BASE_URL=https://api.openai.com/v1 +NORMALIZER_API_KEY=tu_openai_api_key +NORMALIZER_MODEL=gpt-tu-modelo +NORMALIZER_TIMEOUT_S=120 + PDF_DPI=600 PDF_THREAD_COUNT=10 -TIFF_PAGES=(2,3) - -# Configuración de Rutas y Salida -POPPLER_PATH=dependencies/poppler-24.08.0/Library/bin -CSV_DELIMITER=| +CONCURRENCY_LIMIT=4 +ALLOWED_DOCUMENT_TYPES=CC,TI,CE,RC,PA,PPT,PEP,EPT,NUIP ``` -### 🖥️ Uso de la Aplicación +Si `NORMALIZER_BASE_URL` o `NORMALIZER_API_KEY` se dejan vacios, el servicio usa +el mismo backend configurado en `EXTRACTOR_BASE_URL` y `EXTRACTOR_API_KEY`. -Ejecuta el punto de entrada principal: +Dentro de Docker, `IPS_DOCUMENT_ROOT` se monta como `/data`. + +### Levantar ```bash -python gui_main.py +docker compose up --build ``` -#### Flujo de Trabajo en GUI: -1. **Pestaña "Procesar"**: - * Selecciona la carpeta raíz que contiene las subcarpetas de los radicados. - * (Opcional) Define keywords para filtrar archivos específicos. - * Selecciona dónde guardar el CSV de salida. - * Haz clic en **"Procesar"**. - * *Si es la primera vez*: Se abrirá un selector de campos para que elijas qué datos extraer basándose en el primer documento detectado. -2. **Pestaña "Configuración"**: - * Permite ajustar dinámicamente la URL de la API o el modelo sin reiniciar la aplicación. +Health checks: -### 🏗️ Arquitectura del Código +```bash +curl http://localhost:8000/health +curl http://localhost:8001/health +curl http://localhost:8002/health +``` -El proyecto sigue una arquitectura modular para facilitar el mantenimiento y la escalabilidad: +### Flujo Directo Por Servicio -**Capa de Aplicación (`application/`):** -- `service.py`: **Fachada** principal (`InvoiceProcessingService`). Punto de entrada único para la lógica de negocio. -- `batch_processor.py`: Motor de procesamiento masivo. Maneja la iteración de radicados, reportes de progreso y orquestación multi-hilo. -- `estimator.py`: Lógica para calcular costos estimados y uso de tokens antes de lanzar procesos largos. -- `strategies.py`: Implementación del cliente LLM (`OllamaExtractor`) y patrones de extracción. -- `normalizer.py`: Servicio de normalización de datos JSON post-extracción para asegurar consistencia en el esquema. -- `csv_manager.py`: Manejador thread-safe para la escritura incremental de archivos CSV. -- `logging_config.py`: Configuración avanzada de logging (salida JSON para máquinas y colores para humanos). -- `file_discovery.py`: Utilidades para exploración eficiente del sistema de archivos. +1. Ver modelos disponibles en extraccion: -**Capa Core (`core/`):** -- `pdf_utils.py`: Wrapper sobre Poppler para la rasterización de documentos PDF/TIFF. -- `text_utils.py`: Herramientas de limpieza de texto y extracción robusta de bloques JSON. -- `csv_utils.py`: Utilidades puras para aplanamiento de diccionarios anidados y generación de cabeceras. +```bash +curl http://localhost:8001/models +``` -**Capa GUI (`gui/`):** -- Interfaz construida con `customtkinter`, implementando un patrón MVC básico (View, Controller). +2. Descubrir campos: -### 📊 Logs y Depuración +```bash +curl -X POST http://localhost:8001/schemas \ + -H "Content-Type: application/json" \ + -d '{"document_path": "/data/radicado-1/factura.pdf"}' +``` -El sistema genera logs detallados en la carpeta `logs/` para auditoría y depuración: -- `ips_extractor.log`: Log principal de la aplicación (formato JSON estructurado). -- `llm.log`: Trazabilidad específica de las interacciones con el LLM (prompts y respuestas). -- `api_calls.log`: Métricas de uso de API y tokens. +3. Ejecutar extraccion: -Además, la salida por consola está optimizada con colores para una lectura rápida del estado del procesamiento. +```bash +curl -X POST http://localhost:8001/batches \ + -H "Content-Type: application/json" \ + -d '{ + "root_path": "/data", + "keywords": ["factura"], + "fields": ["numero_factura", "items/*/codigo", "items/*/valor"], + "alias_map": { + "numero_factura": "factura", + "items/*/codigo": "codigo", + "items/*/valor": "valor" + }, + "structure_mode": "radicados" + }' +``` -### 📝 Licencia -Uso interno. Ajuste según sus necesidades. +La respuesta devuelve `id`; ese valor es el `batch_id`. + +4. Consultar filas extraidas: + +```bash +curl http://localhost:8001/batches/{batch_id}/rows +``` + +5. Ver reglas de limpieza: + +```bash +curl http://localhost:8002/rules +``` + +6. Analizar columnas extraidas: + +```bash +curl http://localhost:8002/batches/{batch_id}/analysis +``` + +7. Ejecutar limpieza: + +```bash +curl -X POST http://localhost:8002/jobs \ + -H "Content-Type: application/json" \ + -d '{ + "extraction_batch_id": "{batch_id}", + "rules_per_column": { + "valor": ["to_integer"], + "fecha": ["date_to_ddmmyyyy"] + } + }' +``` + +La respuesta devuelve `id`; ese valor es el `job_id`. + +8. Consultar filas limpias: + +```bash +curl http://localhost:8002/jobs/{job_id}/rows +``` + +### Flujo Por API Gateway + +El gateway no importa codigo interno de los servicios; solo enruta HTTP. + +```bash +curl http://localhost:8000/models +curl http://localhost:8000/config +curl http://localhost:8000/extraction/batches/{batch_id}/rows +curl http://localhost:8000/cleaning/jobs/{job_id}/rows +``` + +Para llamar cualquier endpoint propio de un servicio: + +- `http://localhost:8000/extraction/...` +- `http://localhost:8000/cleaning/...` + +### Desarrollo Independiente + +Extraccion como repo aparte: + +```bash +cd services/extraction +pip install -r requirements.txt +uvicorn api.main:app --host 0.0.0.0 --port 8001 +``` + +Limpieza como repo aparte: + +```bash +cd services/cleaning +pip install -r requirements.txt +uvicorn api.main:app --host 0.0.0.0 --port 8002 +``` + +Gateway como repo aparte: + +```bash +cd services/api_gateway +pip install -r requirements.txt +uvicorn api.main:app --host 0.0.0.0 --port 8000 +``` + +### Contrato De Base De Datos + +Los servicios se integran por Postgres: + +- `extraction_batches` +- `extraction_rows` +- `cleaning_jobs` +- `cleaned_rows` + +Cada servicio mantiene su propia copia del modelo SQLAlchemy para no depender +de un paquete comun. diff --git a/application/app_prefs.py b/application/app_prefs.py deleted file mode 100644 index f4b78de..0000000 --- a/application/app_prefs.py +++ /dev/null @@ -1,137 +0,0 @@ -"""Persistencia simple de preferencias de la GUI.""" -from __future__ import annotations - -import json -import logging -import re -from pathlib import Path -from typing import Any, Dict, Optional - -from .config import DEFAULT_ALLOWED_DOCUMENT_TYPES -from .csv_cleaning import DEFAULT_MODERADORA_BY_YEAR, normalize_moderadora_by_year_dict - - -logger = logging.getLogger(__name__) -PREFS_FILE = Path(__file__).resolve().parent.parent / "app_prefs.json" - - -def moderadora_by_year_to_json_dict( - data: dict[int, set[int]], -) -> dict[str, list[int]]: - """Serializa año → conjunto de cuotas a JSON (claves str, listas ordenadas).""" - out: dict[str, list[int]] = {} - for y in sorted(data.keys()): - out[str(y)] = sorted(data[y]) - return out - - -def default_moderadora_by_year_payload() -> dict[str, list[int]]: - """Valores por defecto persistibles para cuota moderadora.""" - return moderadora_by_year_to_json_dict(DEFAULT_MODERADORA_BY_YEAR) - - -def merge_moderadora_prefs(raw: Any) -> dict[str, list[int]]: - """Fusiona prefs guardadas con defaults; devuelve año(str) → lista.""" - base = default_moderadora_by_year_payload() - if not isinstance(raw, dict): - return base - parsed = normalize_moderadora_by_year_dict(raw) - for y, vals in parsed.items(): - base[str(y)] = sorted(vals) - return base - - -def format_moderadora_text_from_prefs(payload: dict[str, list[int]]) -> str: - """Una línea por año: `2024: 4500, 18200, 47700`.""" - lines: list[str] = [] - for ks in sorted(payload.keys(), key=lambda x: int(x)): - nums = payload[ks] - lines.append(f"{ks}: {', '.join(str(n) for n in nums)}") - return "\n".join(lines) - - -def parse_moderadora_text(text: str) -> dict[str, list[int]]: - """Parsea líneas `año: n1, n2` o `año = n1, n2`. Ignora vacías y errores.""" - out: dict[str, list[int]] = {} - for line in (text or "").splitlines(): - line = line.strip() - if not line or line.startswith("#"): - continue - m = re.match(r"^\s*(\d{4})\s*[:=]\s*(.+)$", line) - if not m: - continue - yk = m.group(1) - rest = m.group(2) - nums: list[int] = [] - for part in re.split(r"[,;\s]+", rest): - part = part.strip() - if not part: - continue - try: - nums.append(int(round(float(part)))) - except (TypeError, ValueError): - continue - if nums: - out[yk] = nums - return out - - -def load_cop_mod_preferences() -> tuple[dict[str, list[int]], str]: - """Lee prefs para COP/MOD: mapa año→lista y nombre sugerido columna fecha.""" - prefs = load_app_prefs() - merged = merge_moderadora_prefs(prefs.get("moderadora_by_year")) - fecha_default = str(prefs.get("fecha_servicio_column_default") or "").strip() - return merged, fecha_default - - -def build_cop_mod_config_for_apply( - moderadora_payload: Optional[dict[str, list[int]]] = None, -) -> dict[str, Any]: - """Construye el dict opcional para `csv_cleaning.apply_rules(..., cop_mod_config=)`.""" - if moderadora_payload is None: - prefs = load_app_prefs() - merged = merge_moderadora_prefs(prefs.get("moderadora_by_year")) - else: - merged = merge_moderadora_prefs(moderadora_payload) - return {"moderadora_by_year": merged} - - -def load_app_prefs() -> Dict[str, Any]: - """Carga preferencias desde disco si existen.""" - try: - if not PREFS_FILE.exists(): - return {} - raw = PREFS_FILE.read_text(encoding="utf-8") - data = json.loads(raw) - return data if isinstance(data, dict) else {} - except Exception: - logger.exception("No se pudieron cargar las preferencias de la app") - return {} - - -def save_app_prefs(data: Dict[str, Any]) -> None: - """Guarda preferencias de la app en JSON.""" - try: - PREFS_FILE.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8") - except Exception: - logger.exception("No se pudieron guardar las preferencias de la app") - - -def normalize_document_types(values: Any) -> list[str]: - """Normaliza una lista editable de tipos documentales.""" - raw_values = values if isinstance(values, list) else [] - normalized: list[str] = [] - seen: set[str] = set() - for item in raw_values: - text = str(item or "").strip().upper() - if not text or text == "NIT" or text in seen: - continue - seen.add(text) - normalized.append(text) - return normalized or list(DEFAULT_ALLOWED_DOCUMENT_TYPES) - - -def get_allowed_document_types() -> tuple[str, ...]: - """Devuelve la lista persistida de tipos documentales válidos.""" - prefs = load_app_prefs() - return tuple(normalize_document_types(prefs.get("allowed_document_types"))) diff --git a/application/config.py b/application/config.py deleted file mode 100644 index c2ae9f2..0000000 --- a/application/config.py +++ /dev/null @@ -1,101 +0,0 @@ -"""Configuración centralizada de la aplicación. - -Usa Pydantic BaseSettings para cargar valores por defecto y variables de entorno. -Se lee automáticamente un archivo `.env` en la raíz del proyecto si existe. -""" -from __future__ import annotations - -from typing import Optional, Tuple -import logging -import os - -# Cargar variables desde .env en el entorno (para que otros módulos que usan os.environ las vean) -try: - from dotenv import load_dotenv # type: ignore - # En desarrollo, dar precedencia al archivo .env sobre variables previas del entorno - load_dotenv(override=True) -except Exception: - pass - -from pydantic import Field -from pydantic_settings import BaseSettings, SettingsConfigDict -from pydantic import AliasChoices - - -class ApiSettings(BaseSettings): - """Parámetros del backend compatible con OpenAI/Ollama (usando os.getenv).""" - base_url: str = os.getenv("EXTRACTOR_BASE_URL", "http://10.8.0.4:11434/v1") - api_key: str = os.getenv("EXTRACTOR_API_KEY", "ollama") - model: str = os.getenv("EXTRACTOR_MODEL", "gpt-5.4") - request_timeout_s: int = int(os.getenv("EXTRACTOR_TIMEOUT_S", 120)) - grok_api_key: str = os.getenv("GROK_API_KEY", "") - grok_base_url: str = os.getenv("GROK_BASE_URL", "https://api.x.ai/v1") - model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore") - - -class ImagingSettings(BaseSettings): - """Parámetros de conversión de documentos a imágenes.""" - poppler_path: Optional[str] = Field(None, validation_alias=AliasChoices("POPPLER_PATH")) - pdf_dpi: int = Field(600, validation_alias=AliasChoices("PDF_DPI")) - pdf_thread_count: int = Field(10, validation_alias=AliasChoices("PDF_THREAD_COUNT")) - tiff_pages: Optional[Tuple[int, int]] = Field((2, 3), validation_alias=AliasChoices("TIFF_PAGES")) - model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore") - - -class RetrySettings(BaseSettings): - """Estrategia de reintentos para llamadas al backend.""" - max_attempts: int = Field(3, validation_alias=AliasChoices("RETRY_MAX_ATTEMPTS")) - wait_min_s: float = Field(1.0, validation_alias=AliasChoices("RETRY_WAIT_MIN_S")) - wait_max_s: float = Field(8.0, validation_alias=AliasChoices("RETRY_WAIT_MAX_S")) - model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore") - - -class ProcessingSettings(BaseSettings): - """Configuración de procesamiento y concurrencia.""" - concurrency_limit: int = Field(100, validation_alias=AliasChoices("CONCURRENCY_LIMIT")) - model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore") - - -class AppSettings(BaseSettings): - """Configuración raíz agrupada.""" - api: ApiSettings = ApiSettings() - imaging: ImagingSettings = ImagingSettings() - retry: RetrySettings = RetrySettings() - processing: ProcessingSettings = ProcessingSettings() - csv_delimiter: str = Field("|", validation_alias=AliasChoices("CSV_DELIMITER")) - model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore") - - -settings = AppSettings() - - -def extractor_credentials_for_model(model: str) -> tuple[str, str]: - """Devuelve (api_key, base_url) para el extractor según el id de modelo.""" - m = (model or "").strip() - api = settings.api - if m.lower().startswith("grok-"): - return (str(api.grok_api_key or "").strip(), str(api.grok_base_url or "").strip() or "https://api.x.ai/v1") - return (api.api_key, api.base_url) - - -def is_grok_extractor_model(model: str) -> bool: - """True si el modelo de extracción debe usar la API de xAI (Grok).""" - return (model or "").strip().lower().startswith("grok-") - - -# Tipos documentales válidos por defecto para el campo especial `documento_forzate`. -# NIT se excluye explícitamente porque no debe tratarse como tipo de documento personal. -DEFAULT_ALLOWED_DOCUMENT_TYPES: Tuple[str, ...] = ( - "CC", - "TI", - "CE", - "RC", - "PA", - "PPT", - "PEP", - "EPT", - "NUIP", -) - -# Alias de compatibilidad. -ALLOWED_DOCUMENT_TYPES: Tuple[str, ...] = DEFAULT_ALLOWED_DOCUMENT_TYPES diff --git a/application/database.py b/application/database.py deleted file mode 100644 index be87fc8..0000000 --- a/application/database.py +++ /dev/null @@ -1,281 +0,0 @@ -import sqlite3 -import json -import logging -from datetime import datetime -from typing import List, Dict, Any, Optional -import os - -logger = logging.getLogger(__name__) - -class DatabaseManager: - """ - Gestiona la persistencia de resultados de preclasificación en una base de datos SQLite. - - Estructura de tablas: - - runs: Registra cada ejecución global (fecha, ruta raíz, configuración). - - results: Almacena el estado de procesamiento de cada archivo individual. - - matches: Detalle de campos específicos encontrados en cada archivo. - """ - def __init__(self, db_path: str = "preclassification.db"): - self.db_path = db_path - self._init_db() - - def _init_db(self): - """ - Inicializa el esquema de la base de datos si no existe. - Crea las tablas 'runs', 'results' y 'matches' con sus respectivas llaves foráneas. - """ - try: - conn = sqlite3.connect(self.db_path) - c = conn.cursor() - - # Tabla de ejecuciones - c.execute(''' - CREATE TABLE IF NOT EXISTS runs ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - timestamp TEXT NOT NULL, - root_path TEXT NOT NULL, - config TEXT - ) - ''') - - # Tabla de resultados por archivo/radicado - c.execute(''' - CREATE TABLE IF NOT EXISTS results ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - run_id INTEGER NOT NULL, - radicado TEXT NOT NULL, - file_path TEXT, - status TEXT NOT NULL, -- 'MATCH', 'NO_MATCH', 'ERROR' - raw_response TEXT, - FOREIGN KEY(run_id) REFERENCES runs(id) - ) - ''') - - # Tabla de coincidencias detalladas (campo por campo) - c.execute(''' - CREATE TABLE IF NOT EXISTS matches ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - result_id INTEGER NOT NULL, - key TEXT NOT NULL, - expected_value TEXT, - found_value TEXT, - exists_bool BOOLEAN NOT NULL, - FOREIGN KEY(result_id) REFERENCES results(id) - ) - ''') - - conn.commit() - conn.close() - except Exception as e: - logger.error(f"Error inicializando base de datos: {e}") - - def create_run(self, root_path: str, config: Dict[str, Any]) -> int: - """ - Registra una nueva sesión de procesamiento. - - Args: - root_path: Directorio raíz procesado. - config: Diccionario de configuración usado (se guarda como JSON). - - Returns: - ID de la ejecución creada. - """ - conn = sqlite3.connect(self.db_path) - c = conn.cursor() - timestamp = datetime.now().isoformat() - c.execute('INSERT INTO runs (timestamp, root_path, config) VALUES (?, ?, ?)', - (timestamp, root_path, json.dumps(config))) - run_id = c.lastrowid - conn.commit() - conn.close() - return run_id - - def add_result(self, run_id: int, radicado: str, file_path: str, status: str, raw_response: str) -> int: - """ - Añade el resultado del procesamiento de un archivo. - - Args: - run_id: ID de la ejecución padre. - radicado: Identificador del radicado. - file_path: Ruta al archivo físico. - status: Estado final ('MATCH', 'NO_MATCH', 'ERROR'). - raw_response: Respuesta completa del modelo. - - Returns: - ID del resultado insertado. - """ - conn = sqlite3.connect(self.db_path) - c = conn.cursor() - c.execute('INSERT INTO results (run_id, radicado, file_path, status, raw_response) VALUES (?, ?, ?, ?, ?)', - (run_id, radicado, file_path, status, raw_response)) - result_id = c.lastrowid - conn.commit() - conn.close() - return result_id - - def add_match(self, result_id: int, key: str, expected_value: Optional[str], found_value: Optional[str], exists: bool): - """ - Registra el hallazgo (o ausencia) de un campo específico en un documento. - - Args: - result_id: ID del resultado al que pertenece. - key: Nombre del campo/llave. - expected_value: Valor que se esperaba encontrar (si aplica). - found_value: Valor extraído por el modelo. - exists: Booleano que indica si el campo fue efectivamente localizado. - """ - conn = sqlite3.connect(self.db_path) - c = conn.cursor() - c.execute('INSERT INTO matches (result_id, key, expected_value, found_value, exists_bool) VALUES (?, ?, ?, ?, ?)', - (result_id, key, expected_value, found_value, exists)) - conn.commit() - conn.close() - - def get_run_results(self, run_id: int) -> List[Dict[str, Any]]: - """Obtiene resultados completos de una ejecución.""" - conn = sqlite3.connect(self.db_path) - conn.row_factory = sqlite3.Row - c = conn.cursor() - - c.execute(''' - SELECT r.radicado, r.file_path, r.status, m.key, m.expected_value, m.found_value, m.exists_bool - FROM results r - LEFT JOIN matches m ON r.id = m.result_id - WHERE r.run_id = ? - ''', (run_id,)) - - rows = c.fetchall() - conn.close() - return [dict(row) for row in rows] - - def get_all_runs(self) -> List[Dict[str, Any]]: - """Obtiene lista de todos los runs ordenados por fecha descendente.""" - conn = sqlite3.connect(self.db_path) - conn.row_factory = sqlite3.Row - c = conn.cursor() - c.execute("SELECT * FROM runs ORDER BY timestamp DESC") - rows = c.fetchall() - conn.close() - return [dict(row) for row in rows] - - def _search_results_query( - self, run_id: Optional[int] = None, status: Optional[str] = None, text: Optional[str] = None - ) -> List[Dict[str, Any]]: - """Una búsqueda; todos los criterios activos se combinan con AND. - - Orden de importancia usado al relajar (en `search_results`): - prioridad 1: run, 2: estado, 3: texto. No se suelta el criterio de - `run` al relajar — solo texto y, si aplica, estado. - """ - tx = (text or "").strip() or None - conn = sqlite3.connect(self.db_path) - conn.row_factory = sqlite3.Row - c = conn.cursor() - query = ''' - SELECT r.id, r.run_id, r.radicado, r.file_path, r.status, r.raw_response, runs.timestamp - FROM results r - JOIN runs ON r.run_id = runs.id - WHERE 1=1 - ''' - params: list = [] - if run_id is not None: - query += " AND r.run_id = ?" - params.append(run_id) - if status and status != "Todos": - query += " AND r.status = ?" - params.append(status) - if tx: - query += " AND (r.radicado LIKE ? OR r.file_path LIKE ?)" - wild = f"%{tx}%" - params.extend([wild, wild]) - query += " ORDER BY r.id DESC LIMIT 1000" - c.execute(query, params) - rows = c.fetchall() - conn.close() - return [dict(row) for row in rows] - - def search_results( - self, - run_id: Optional[int] = None, - status: Optional[str] = None, - text: Optional[str] = None, - relax_if_empty: bool = False, - ) -> tuple[List[Dict[str, Any]], Optional[str]]: - """Busca resultados; opcionalmente relaja criterios si no hay filas. - - Prioridad: **run (si aplica) > estado > criterio de búsqueda (texto)**. - Con `relax_if_empty=True`, se omite el texto, luego el estado; el - filtro de run fijado no se omite. - - Devuelve (filas, nota) donde `nota` informa criterios omitidos al - relajar, o `None` si se usó la consulta estricta o no aplica. - """ - had_text = bool((text or "").strip()) - had_status = bool(status and status != "Todos") - - rows = self._search_results_query(run_id, status, text) - if rows or not relax_if_empty: - return (rows, None) - if not had_text and not had_status: - return ([], None) - if had_text: - looser = self._search_results_query(run_id, status, None) - if looser: - return ( - looser, - "Búsqueda ampliada: se omitió el criterio de búsqueda (texto) " - "(el de menor importancia: run > estado > texto; el run no se descarta al relajar).", - ) - if had_status: - looser2 = self._search_results_query(run_id, None, None) - if looser2: - if had_text: - nota = ( - "Búsqueda ampliada: se omitieron el criterio de búsqueda (texto) y el filtro de estado, " - "en ese orden, por prioridad (run > estado > texto; el run no se descarta al relajar)." - ) - else: - nota = ( - "Búsqueda ampliada: se omitió el filtro de estado, " - "criterio más débil al combinar con un run. Prioridad: run > estado > búsqueda; " - "el run no se descarta al relajar." - ) - return (looser2, nota) - return ([], None) - - def update_result_status(self, result_id: int, status: str) -> None: - """Actualiza el status de un resultado existente.""" - conn = sqlite3.connect(self.db_path) - c = conn.cursor() - c.execute("UPDATE results SET status = ? WHERE id = ?", (status, result_id)) - conn.commit() - conn.close() - - def get_matches_for_result(self, result_id: int) -> List[Dict[str, Any]]: - """Obtiene el detalle de matches para un resultado específico.""" - conn = sqlite3.connect(self.db_path) - conn.row_factory = sqlite3.Row - c = conn.cursor() - c.execute("SELECT * FROM matches WHERE result_id = ?", (result_id,)) - rows = c.fetchall() - conn.close() - return [dict(row) for row in rows] - - def get_run(self, run_id: int) -> Optional[Dict[str, Any]]: - """Obtiene un run por ID (incluye config como dict si aplica).""" - conn = sqlite3.connect(self.db_path) - conn.row_factory = sqlite3.Row - c = conn.cursor() - c.execute("SELECT * FROM runs WHERE id = ?", (run_id,)) - row = c.fetchone() - conn.close() - if not row: - return None - d = dict(row) - try: - cfg_raw = d.get("config") - d["config"] = json.loads(cfg_raw) if cfg_raw else {} - except Exception: - d["config"] = {} - return d diff --git a/application/postprocess_agent.py b/application/postprocess_agent.py deleted file mode 100644 index 754540e..0000000 --- a/application/postprocess_agent.py +++ /dev/null @@ -1,254 +0,0 @@ -"""Agente inteligente de post-procesamiento de CSV. - -Genera código Python con un LLM para transformar DataFrames según -instrucciones del usuario en lenguaje natural. -Usa la API de Responses de OpenAI (requerida por modelos codex). -""" -from __future__ import annotations - -import logging -import re -import threading -from typing import Any, Dict, List, Optional, Tuple - -import numpy as np -import pandas as pd -from openai import OpenAI - -logger = logging.getLogger(__name__) - -POSTPROCESS_MODEL = "gpt-5.1-codex-mini" - -SYSTEM_PROMPT = """\ -Eres un agente experto en transformación de datos con pandas. - -REGLAS ESTRICTAS: -1. Recibes una MUESTRA representativa de un CSV y una instrucción del usuario. -2. Debes devolver ÚNICAMENTE un bloque de código Python válido dentro de ```python ... ```. -3. El código opera sobre una variable `df` que es un `pd.DataFrame` ya cargado. -4. El código debe modificar `df` in-place o reasignar a `df`. Al final, `df` debe contener el resultado. -5. Solo puedes usar las siguientes librerías (ya importadas): `pd` (pandas), `np` (numpy), `re`, `datetime`. -6. NO puedes usar: open, exec, eval, __import__, compile, subprocess, os, sys, requests, ni ninguna forma de I/O o acceso al sistema. -7. El código debe ser conciso y eficiente. -8. NO incluyas explicaciones fuera del bloque de código. -""" - -RETRY_TEMPLATE = """\ -El código que generaste produjo el siguiente error al ejecutarse: - -CÓDIGO: -```python -{code} -``` - -ERROR: -{error} - -INSTRUCCIÓN ORIGINAL DEL USUARIO: -{instructions} - -MUESTRA DEL CSV: -{sample} - -Por favor genera un código corregido que solucione el error. Devuelve ÚNICAMENTE el bloque ```python ... ```. -""" - - -def sample_dataframe(df: pd.DataFrame, n: int = 5) -> Tuple[str, Dict[str, Any]]: - """Genera una muestra inteligente del DataFrame. - - Retorna (sample_text, column_info) donde sample_text es una - representación legible y column_info contiene metadatos. - """ - total_rows = len(df) - total_cols = len(df.columns) - - col_info = { - "total_rows": total_rows, - "total_cols": total_cols, - "columns": {col: str(df[col].dtype) for col in df.columns}, - } - - used_indices: set[int] = set() - sections: List[str] = [] - - sections.append( - f"Total filas: {total_rows:,} | Total columnas: {total_cols}\n" - f"Columnas y tipos: {', '.join(f'{c} ({t})' for c, t in col_info['columns'].items())}" - ) - - head_n = min(n, total_rows) - head_idx = list(range(head_n)) - used_indices.update(head_idx) - sections.append(f"--- Primeras {head_n} filas ---\n{df.iloc[head_idx].to_csv(index=False)}") - - if total_rows > n: - tail_start = max(total_rows - n, head_n) - tail_idx = list(range(tail_start, total_rows)) - new_tail = [i for i in tail_idx if i not in used_indices] - if new_tail: - used_indices.update(new_tail) - sections.append(f"--- Últimas {len(new_tail)} filas ---\n{df.iloc[new_tail].to_csv(index=False)}") - - if total_rows > 2 * n: - mid_center = total_rows // 2 - mid_start = max(mid_center - n // 2, 0) - mid_idx = list(range(mid_start, min(mid_start + n, total_rows))) - new_mid = [i for i in mid_idx if i not in used_indices] - if new_mid: - used_indices.update(new_mid) - sections.append(f"--- Filas del medio ---\n{df.iloc[new_mid].to_csv(index=False)}") - - remaining = [i for i in range(total_rows) if i not in used_indices] - if remaining: - rand_n = min(n, len(remaining)) - rng = np.random.default_rng(42) - rand_idx = sorted(rng.choice(remaining, size=rand_n, replace=False).tolist()) - used_indices.update(rand_idx) - sections.append(f"--- {rand_n} filas aleatorias ---\n{df.iloc[rand_idx].to_csv(index=False)}") - - sample_text = "\n\n".join(sections) - return sample_text, col_info - - -def _extract_python_code(text: str) -> str: - """Extrae el bloque de código Python de la respuesta del LLM.""" - pattern = r"```python\s*\n(.*?)```" - match = re.search(pattern, text, re.DOTALL) - if match: - return match.group(1).strip() - pattern2 = r"```\s*\n(.*?)```" - match2 = re.search(pattern2, text, re.DOTALL) - if match2: - return match2.group(1).strip() - return text.strip() - - -def _call_responses_api( - client: OpenAI, - instructions: str, - user_input: str, -) -> Tuple[str, Optional[Dict[str, Any]]]: - """Llama a la API de Responses (requerida por modelos codex).""" - response = client.responses.create( - model=POSTPROCESS_MODEL, - instructions=instructions, - input=user_input, - ) - usage = None - raw_usage = getattr(response, "usage", None) - if raw_usage is not None: - usage = { - "input_tokens": getattr(raw_usage, "input_tokens", None), - "output_tokens": getattr(raw_usage, "output_tokens", None), - "total_tokens": getattr(raw_usage, "total_tokens", None), - } - return response.output_text, usage - - -def generate_code( - client: OpenAI, - csv_sample: str, - user_instructions: str, -) -> Tuple[str, Optional[Dict[str, Any]]]: - """Genera código Python invocando al LLM via Responses API. - - Retorna (code_string, usage_dict). - """ - user_input = ( - f"MUESTRA DEL CSV:\n{csv_sample}\n\n" - f"INSTRUCCIONES:\n{user_instructions}" - ) - raw_text, usage = _call_responses_api(client, SYSTEM_PROMPT, user_input) - code = _extract_python_code(raw_text) - return code, usage - - -def _build_safe_builtins() -> dict: - import builtins as _b - blocked = {"open", "exec", "eval", "__import__", "compile", - "exit", "quit", "breakpoint", "input"} - return {k: v for k, v in vars(_b).items() if k not in blocked} - -_SAFE_BUILTINS = _build_safe_builtins() - - -def execute_code( - code: str, - df: pd.DataFrame, - timeout_s: float = 60, -) -> Tuple[Optional[pd.DataFrame], Optional[str]]: - """Ejecuta código Python generado en un entorno restringido. - - Retorna (df_result, error_message). Si error_message es None, la - ejecución fue exitosa. - """ - import datetime as _dt - - df_copy = df.copy() - namespace: Dict[str, Any] = { - "__builtins__": _SAFE_BUILTINS, - "pd": pd, - "np": np, - "re": re, - "datetime": _dt, - "df": df_copy, - } - - result_holder: Dict[str, Any] = {"df": None, "error": None} - - def _run(): - try: - exec(code, namespace) # noqa: S102 - result_holder["df"] = namespace.get("df", df_copy) - except Exception as exc: - result_holder["error"] = f"{type(exc).__name__}: {exc}" - - thread = threading.Thread(target=_run, daemon=True) - thread.start() - thread.join(timeout=timeout_s) - - if thread.is_alive(): - return None, "Timeout: la ejecución excedió el límite de tiempo." - - if result_holder["error"]: - return None, result_holder["error"] - - result_df = result_holder["df"] - if not isinstance(result_df, pd.DataFrame): - return None, "El código no produjo un DataFrame válido en la variable `df`." - - return result_df, None - - -def retry_with_error( - client: OpenAI, - csv_sample: str, - user_instructions: str, - failed_code: str, - error_msg: str, -) -> Tuple[str, Optional[Dict[str, Any]]]: - """Re-envía al LLM el código fallido y el error para corrección.""" - retry_content = RETRY_TEMPLATE.format( - code=failed_code, - error=error_msg, - instructions=user_instructions, - sample=csv_sample, - ) - raw_text, usage = _call_responses_api(client, SYSTEM_PROMPT, retry_content) - code = _extract_python_code(raw_text) - return code, usage - - -def detect_delimiter(filepath: str) -> str: - """Detecta el delimitador de un archivo CSV leyendo las primeras líneas.""" - import csv as _csv - - with open(filepath, "r", encoding="utf-8", errors="replace") as f: - sample = f.read(8192) - - try: - dialect = _csv.Sniffer().sniff(sample, delimiters=",;\t|") - return dialect.delimiter - except _csv.Error: - return "," diff --git a/application/preclassification.py b/application/preclassification.py deleted file mode 100644 index b761ecd..0000000 --- a/application/preclassification.py +++ /dev/null @@ -1,448 +0,0 @@ -import fnmatch -import json -import logging -import os -import re -import shutil -import time -from typing import List, Dict, Any, Optional, Tuple, Callable - -import pandas as pd - -from application.strategies import OllamaExtractor -from application.database import DatabaseManager -from core.pdf_utils import document_to_images_bs64 - -logger = logging.getLogger(__name__) - -class PreclassificationService: - """ - Servicio para la preclasificación masiva de documentos. - - Permite identificar si un documento cumple con ciertos criterios (patterns) - y extraer información clave para organizar los archivos físicamente en carpetas. - """ - def __init__(self, extractor: OllamaExtractor): - self.extractor = extractor - self.db = DatabaseManager() - - def export_to_excel(self, run_id: int, output_path: str) -> bool: - """ - Exporta los resultados de una ejecución de preclasificación a Excel o CSV. - - Args: - run_id: ID de la ejecución en la base de datos. - output_path: Ruta del archivo de salida (.xlsx o .csv). - - Returns: - True si la exportación fue exitosa, False si no hay datos. - """ - try: - results = self.db.get_run_results(run_id) - if not results: - return False - - # Pivotar para tener filas únicas por archivo - # get_run_results devuelve una fila por cada key encontrada en cada archivo - # Necesitamos agrupar por (radicado, file_path) - - data_map = {} - keys_seen = set() - - for row in results: - file_id = f"{row['radicado']}|{row['file_path']}" - if file_id not in data_map: - data_map[file_id] = { - "Radicado": row["radicado"], - "Archivo": row["file_path"], - "Status": row["status"] - } - - k = row["key"] - if k: - # found_value puede ser None - val = row["found_value"] - exists = row["exists_bool"] - # Podemos poner el valor o un indicador - data_map[file_id][k] = val if exists else "NO_FOUND" - keys_seen.add(k) - - # Convertir a lista - final_data = list(data_map.values()) - - df = pd.DataFrame(final_data) - - # Asegurar todas las columnas de keys - for k in keys_seen: - if k not in df.columns: - df[k] = None - - if output_path.endswith(".csv"): - df.to_csv(output_path, index=False, encoding="utf-8-sig", sep=";") - else: - df.to_excel(output_path, index=False) - - return True - except Exception as e: - logger.exception("Error exportando a Excel") - raise e - - def organize_files(self, run_id: int, base_output_path: str) -> tuple[int, int]: - """ - Copia físicamente los archivos que hicieron 'MATCH' a una nueva estructura de carpetas. - La jerarquía se determina automáticamente basada en los valores extraídos. - - Args: - run_id: ID de la ejecución. - base_output_path: Directorio donde se creará la nueva estructura. - - Returns: - Tupla (total_copiados, total_errores). - """ - run = self.db.get_run(run_id) - mode = "radicados" - if run: - cfg = run.get("config") or {} - mode = str(cfg.get("mode", "radicados") or "radicados").strip().lower() - if mode not in ("radicados", "general"): - mode = "radicados" - - results = self.db.get_run_results(run_id) - # Filtrar solo MATCH - match_rows = [r for r in results if r["status"] == "MATCH"] - - if not match_rows: - return 0, 0 - - # Agrupar por archivo para tener el "set" de datos de cada uno - files_data = {} # { file_path: {key: val, ...}, 'radicado': ... } - key_values_all = {} # { key: set(values) } - - for row in match_rows: - fpath = row["file_path"] - if not fpath: continue - - if fpath not in files_data: - files_data[fpath] = {"radicado": row["radicado"], "keys": {}} - - k = row["key"] - v = row["found_value"] - if k and v: - files_data[fpath]["keys"][k] = v - - if k not in key_values_all: - key_values_all[k] = set() - key_values_all[k].add(v) - - # Determinar orden de jerarquía por cardinalidad - # Lista de (key, cardinalidad) - hierarchy = [] - for k, val_set in key_values_all.items(): - hierarchy.append((k, len(val_set))) - - # Ordenar ascendente: primero los que tienen menos valores (ej. Categorias generales) - hierarchy.sort(key=lambda x: x[1]) - ordered_keys = [x[0] for x in hierarchy] - - logger.info(f"Jerarquía de carpetas calculada: {ordered_keys}") - - count_ok = 0 - count_err = 0 - - for fpath, info in files_data.items(): - try: - # Construir ruta relativa - # Sanitizar valores para que sean nombres de carpeta válidos - path_parts = [] - for k in ordered_keys: - val = info["keys"].get(k, "DESCONOCIDO") - safe_val = self._sanitize_filename(str(val)) - - # KeyName / ValueName - safe_key = self._sanitize_filename(k) - path_parts.append(safe_key) - path_parts.append(safe_val) - - # Añadir radicado al final solo si NO es modo general (en general el radicado es el archivo) - # Si mode es radicados: ... / Radicado / Archivo - # Si mode es general: ... / Archivo - if mode != "general": - path_parts.append(self._sanitize_filename(info["radicado"])) - - dest_dir = os.path.join(base_output_path, *path_parts) - os.makedirs(dest_dir, exist_ok=True) - - fname = os.path.basename(fpath) - dest_file = os.path.join(dest_dir, fname) - # Evitar sobrescrituras (especialmente en modo general sin radicado) - if os.path.exists(dest_file): - base, ext = os.path.splitext(fname) - n = 1 - while True: - candidate = os.path.join(dest_dir, f"{base}_{n}{ext}") - if not os.path.exists(candidate): - dest_file = candidate - break - n += 1 - - # Copiar - shutil.copy2(fpath, dest_file) - count_ok += 1 - except Exception as e: - logger.error(f"Error organizando {fpath}: {e}") - count_err += 1 - - return count_ok, count_err - - def _sanitize_filename(self, name: str) -> str: - """Limpia caracteres inválidos para rutas de archivo en Windows/Linux.""" - # Reemplazar caracteres no alfanuméricos (excepto espacio, guion, punto) - clean = re.sub(r'[\\/*?:"<>|]', "", name) - clean = clean.replace("\n", " ").replace("\r", "").strip() - if not clean: - return "UNTITLED" - return clean[:50] # Limitar longitud - - def run_preclassification( - self, - root_path: str, - patterns: List[Dict[str, str]], # [{key, mode, value?}, ...] - target_info: Dict[str, Any], # {enabled, definition, pattern} - page_range: Tuple[bool, int, int], # (enabled, start, end) - on_progress: Optional[Callable[[str, int, int], None]] = None, - mode: str = "radicados" # "radicados" | "general" - ) -> int: - """ - Ejecuta el flujo completo de preclasificación sobre un directorio. - - Args: - root_path: Directorio con los documentos. - patterns: Reglas de validación (ej. buscar NIT, buscar Fecha). - target_info: Filtro de archivos por nombre/patrón. - page_range: Rango de páginas a procesar por documento. - on_progress: Callback para reportar avance a la GUI. - mode: 'radicados' (carpetas) o 'general' (archivos sueltos). - - Returns: - ID de la ejecución (run_id) generada. - """ - # 1. Crear Run en BD - mode = str(mode or "radicados").strip().lower() - if mode not in ("radicados", "general"): - mode = "radicados" - - run_config = { - "patterns": patterns, - "target": target_info, - "range": page_range, - "mode": mode - } - run_id = self.db.create_run(root_path, run_config) - - # 2. Descubrir items a procesar - radicados = [] - if os.path.isdir(root_path): - if mode == "general": - # En modo general, consideramos que la raiz es el único "contenedor" - # o iteramos sobre una lista dummy. - # Para simplificar el loop principal, usaremos una lista con un elemento "." - radicados = ["."] - else: - radicados = sorted([ - d for d in os.listdir(root_path) - if os.path.isdir(os.path.join(root_path, d)) - ]) - - total = len(radicados) - - # Preparar prompt - keys_to_extract = [p["key"] for p in patterns] - prompt = self._build_prompt(keys_to_extract) - - for idx, rad in enumerate(radicados, start=1): - # Si es general, rad es ".", no es util para el usuario ver "Procesando ." - # Ajustaremos el reporte dentro del loop de archivos si es general - if mode != "general" and on_progress: - on_progress(rad, idx, total) - - rad_path = os.path.join(root_path, rad) - - # 3. Identificar archivos a procesar - files_to_process = self._identify_files(rad_path, target_info, mode) - logger.info(f"Radicado/Carpeta: {rad} - Archivos identificados: {len(files_to_process)}") - - # Ordenar archivos para consistencia - files_to_process.sort() - - if not files_to_process: - if mode == "general": - # Registrar una sola vez que no había archivos - self.db.add_result(run_id, "GENERAL", None, "NO_FILES", "") - return run_id - else: - self.db.add_result(run_id, rad, None, "NO_FILES", "") - continue - - total_files = len(files_to_process) - for f_idx, file_path in enumerate(files_to_process, 1): - # En modo general, el "radicado" lógico será el nombre del archivo sin extensión - # En modo radicados, es el nombre de la carpeta (variable rad) - current_radicado = rad - - # Definir radicado para modo general ANTES de cualquier lógica - if mode == "general": - base_name = os.path.basename(file_path) - name_no_ext = os.path.splitext(base_name)[0] - # Validar que no quede vacio ni sea un punto - if name_no_ext and name_no_ext.strip() not in (".", ""): - current_radicado = name_no_ext.strip() - else: - current_radicado = base_name - - if str(current_radicado).strip() in (".", ""): - current_radicado = f"FILE_{int(time.time()*1000)}_{f_idx}" - - logger.info(f"Procesando archivo [{f_idx}/{total_files}]: {file_path} -> Radicado: {current_radicado}") - if on_progress: - try: - # Asegurar que pasamos un string válido - safe_rad = str(current_radicado) - on_progress(safe_rad, f_idx, total_files) - except Exception: - pass # Evitar romper el loop si la GUI falla - - try: - # 4. Convertir a imágenes (con soporte de rango si está habilitado) - images_b64 = self._get_images(file_path, page_range) - - if not images_b64: - self.db.add_result(run_id, rad, file_path, "ERROR_IMG", "No images generated") - continue - - # 5. LLM - response = self.extractor.extract(images_b64, prompt) - raw_text = response.get("raw", "") - - # 6. Parseo y Validación - parsed = self._parse_json_response(raw_text) - - # Guardar resultado cabecera - result_id = self.db.add_result(run_id, current_radicado, file_path, "PROCESSED", raw_text) - - # Guardar detalles y calcular match global - global_match = True - if not parsed: - global_match = False - # Guardar error de parseo? - else: - for pattern in patterns: - key = pattern["key"] - pat_mode = pattern.get("mode", "key") - expected_val = pattern.get("value", "").lower().strip() - - # Buscar en respuesta - # Normalizamos claves para busqueda insensible a mayusculas - found_item = None - for k, v in parsed.items(): - if k.lower() == key.lower(): - found_item = v - break - - exists = False - found_val = None - - if found_item and isinstance(found_item, dict): - exists = bool(found_item.get("exists", False)) - found_val = str(found_item.get("value", "") or "") - - # Regla - rule_ok = False - if pat_mode == "key": - rule_ok = exists - elif pat_mode == "key+value": - # Comparacion laxa - rule_ok = exists and (expected_val in found_val.lower()) - - if not rule_ok: - global_match = False - - self.db.add_match(result_id, key, expected_val if pat_mode=="key+value" else None, found_val, exists) - - # Actualizar status final - final_status = "MATCH" if global_match else "NO_MATCH" - self._update_result_status(result_id, final_status) - - except Exception as e: - logger.exception(f"Error procesando {file_path}") - self.db.add_result(run_id, current_radicado, file_path, f"ERROR: {str(e)}", "") - - return run_id - - def _identify_files(self, rad_path: str, target_info: Dict[str, Any], mode: str = "radicados") -> List[str]: - """Devuelve lista de rutas de archivos a procesar.""" - if not os.path.exists(rad_path): - return [] - - # Si mode es general, rad_path es la raiz y buscamos archivos directos. - # Si mode es radicados, rad_path es la subcarpeta del radicado. - # La logica es la misma: listar archivos en el directorio dado. - - all_files = [] - try: - for f in os.listdir(rad_path): - full_path = os.path.join(rad_path, f) - if os.path.isfile(full_path) and f.lower().endswith(('.pdf', '.tif', '.tiff')): - all_files.append(full_path) - except Exception: - pass - - if not target_info.get("enabled"): - return all_files - - pattern = target_info.get("pattern", "*").lower() - filtered = [f for f in all_files if fnmatch.fnmatch(os.path.basename(f).lower(), pattern)] - return filtered - - def _get_images(self, file_path: str, page_range: Tuple[bool, int, int]) -> List[str]: - """Convierte archivo a imagenes, aplicando rango si es necesario.""" - enabled, start, end = page_range - - if enabled: - from core.pdf_utils import pdf_pages_to_images_bs64 - ext = os.path.splitext(file_path)[1].lower() - s, e = max(1, start), max(1, end) - if s > e: - s, e = e, s - if ext == ".pdf": - return pdf_pages_to_images_bs64(file_path, s, e, dpi=300) - elif ext in (".tif", ".tiff"): - return document_to_images_bs64(file_path, tiff_page_range=(s, e), dpi=300) - - return document_to_images_bs64(file_path, dpi=300) - - def _build_prompt(self, keys: List[str]) -> str: - keys_str = ", ".join(keys) - return f"""Responde con true o false si tiene los siguientes campos, si existe extrae cuales son. -Responde ÚNICAMENTE en JSON con el siguiente formato, sin bloques de código markdown: -{{ - "nombre_campo": {{ - "exists": true, - "value": "valor extraído" - }}, - ... -}} - -Campos a buscar: {keys_str}""" - - def _parse_json_response(self, text: str) -> Dict[str, Any]: - try: - # Limpiar bloques markdown si el modelo los pone - clean = text.replace("```json", "").replace("```", "").strip() - return json.loads(clean) - except Exception: - # Intento de rescate simple con regex si falla json puro - # Por ahora retornamos vacio para indicar fallo - return {} - - def _update_result_status(self, result_id: int, status: str): - self.db.update_result_status(result_id, status) diff --git a/application/tracking_manager.py b/application/tracking_manager.py deleted file mode 100644 index ae1a5b0..0000000 --- a/application/tracking_manager.py +++ /dev/null @@ -1,107 +0,0 @@ -import requests -import logging -import os -import socket -import getpass -from typing import Optional, Dict, Any -from datetime import datetime - -logger = logging.getLogger(__name__) - -class TrackingManager: - """Gestiona la comunicación con el servidor de seguimiento web.""" - - def __init__(self, base_url: str = "http://localhost:8000"): - self.base_url = base_url - self.current_batch_id: Optional[str] = None - self.is_enabled = True - self._session = requests.Session() # Usar sesión para reutilizar conexiones - - def start_batch(self, root_path: str, total_radicados: int, config: Optional[Dict] = None, - csv_output_path: Optional[str] = None): - """Notifica el inicio de un nuevo lote al servidor.""" - if not self.is_enabled: - return - - try: - payload = { - "root_path": root_path, - "total_radicados": total_radicados, - "csv_output_path": csv_output_path, - "hostname": socket.gethostname(), - "username": getpass.getuser(), - "config": config or {} - } - response = self._session.post(f"{self.base_url}/batches", json=payload, timeout=5) - if response.status_code == 200: - self.current_batch_id = response.json().get("id") - logger.info(f"Tracking iniciado: Batch ID {self.current_batch_id}") - else: - logger.warning(f"Error al iniciar tracking: {response.status_code}") - except Exception as e: - logger.error(f"Error de conexión con el servidor de tracking: {e}") - - def update_progress(self, current_index: int, success_count: int, error_count: int, - total_tokens: int, total_cost_usd: float, status: str = "processing"): - """Actualiza el progreso del lote actual.""" - if not self.is_enabled or not self.current_batch_id: - return - - try: - payload = { - "current_radicado_index": current_index, - "success_count": success_count, - "error_count": error_count, - "total_tokens": total_tokens, - "total_cost_usd": total_cost_usd, - "status": status - } - self._session.patch(f"{self.base_url}/batches/{self.current_batch_id}", json=payload, timeout=1) - except Exception as e: - logger.debug(f"Error al actualizar tracking: {e}") - - def log(self, message: str, level: str = "info", radicado: str = None): - """Envía una entrada de log al servidor de tracking.""" - if not self.is_enabled or not self.current_batch_id: - return - try: - payload = {"message": message, "level": level, "radicado": radicado} - self._session.post( - f"{self.base_url}/batches/{self.current_batch_id}/logs", - json=payload, timeout=1 - ) - except Exception as e: - logger.debug(f"Error al enviar log a tracking: {e}") - - def report_phase(self, phase: str, radicado: str = None): - """Reporta la fase actual de procesamiento (extract, normalize, csv, completed).""" - if not self.is_enabled or not self.current_batch_id: - return - try: - payload = {"current_phase": phase, "current_radicado": radicado} - self._session.patch(f"{self.base_url}/batches/{self.current_batch_id}", json=payload, timeout=1) - except Exception as e: - logger.debug(f"Error al reportar fase a tracking: {e}") - - def report_error(self, error_message: str): - """Reporta un error crítico al servidor.""" - if not self.is_enabled or not self.current_batch_id: - return - - try: - payload = {"last_error": error_message} - self._session.patch(f"{self.base_url}/batches/{self.current_batch_id}", json=payload, timeout=1) - except Exception as e: - logger.debug(f"Error al reportar error de tracking: {e}") - - def complete_batch(self, success_count: int, error_count: int, total_tokens: int, total_cost_usd: float): - """Notifica la finalización exitosa del lote.""" - self.update_progress( - current_index=-1, # Indica finalización si se desea - success_count=success_count, - error_count=error_count, - total_tokens=total_tokens, - total_cost_usd=total_cost_usd, - status="completed" - ) - self.current_batch_id = None diff --git a/build_installer.bat b/build_installer.bat deleted file mode 100644 index e95103f..0000000 --- a/build_installer.bat +++ /dev/null @@ -1,97 +0,0 @@ -@echo off -REM ============================================================================ -REM Build script para IPSExtractor -REM Genera el ejecutable con PyInstaller y opcionalmente el instalador con -REM Inno Setup. -REM -REM Uso: -REM build_installer.bat (solo PyInstaller) -REM build_installer.bat --inno (PyInstaller + Inno Setup) -REM ============================================================================ - -setlocal - -set "ROOT=%~dp0" -cd /d "%ROOT%" - -echo. -echo ======================================== -echo IPSExtractor - Build -echo ======================================== -echo. - -REM -- Verificar Python --- -python --version >nul 2>&1 -if errorlevel 1 ( - echo ERROR: Python no encontrado en PATH. - pause - exit /b 1 -) - -REM -- Verificar PyInstaller --- -python -c "import PyInstaller" >nul 2>&1 -if errorlevel 1 ( - echo Instalando PyInstaller... - pip install pyinstaller -) - -REM -- Limpiar builds anteriores --- -echo Limpiando builds anteriores... -if exist "build" rmdir /s /q "build" -if exist "dist\IPSExtractor" rmdir /s /q "dist\IPSExtractor" - -REM -- Ejecutar PyInstaller --- -echo. -echo Ejecutando PyInstaller... -echo. -pyinstaller ipsextractor.spec --noconfirm - -if errorlevel 1 ( - echo. - echo ERROR: PyInstaller fallo. Revise los errores arriba. - pause - exit /b 1 -) - -echo. -echo ======================================== -echo Build exitoso! -echo Output: dist\IPSExtractor\ -echo ======================================== -echo. - -REM -- Inno Setup (opcional) --- -if /i "%~1"=="--inno" ( - echo Buscando Inno Setup... - - set "ISCC=" - if exist "C:\Program Files (x86)\Inno Setup 6\ISCC.exe" ( - set "ISCC=C:\Program Files (x86)\Inno Setup 6\ISCC.exe" - ) - if exist "C:\Program Files\Inno Setup 6\ISCC.exe" ( - set "ISCC=C:\Program Files\Inno Setup 6\ISCC.exe" - ) - - if not defined ISCC ( - echo AVISO: Inno Setup no encontrado. Instale Inno Setup 6 para generar el instalador. - echo Descargue en: https://jrsoftware.org/isdl.php - pause - exit /b 0 - ) - - echo Generando instalador con Inno Setup... - "%ISCC%" "%ROOT%installer.iss" - - if errorlevel 1 ( - echo ERROR: Inno Setup fallo. - pause - exit /b 1 - ) - - echo. - echo ======================================== - echo Instalador generado en: dist\ - echo ======================================== -) - -pause diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..e264763 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,77 @@ +services: + database: + image: postgres:16-alpine + container_name: ipsextractor-db + environment: + POSTGRES_DB: ipsextractor + POSTGRES_USER: ipsextractor + POSTGRES_PASSWORD: ipsextractor + ports: + - "5432:5432" + volumes: + - ipsextractor_db:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U ipsextractor -d ipsextractor"] + interval: 5s + timeout: 5s + retries: 10 + + extraction: + build: + context: ./services/extraction + dockerfile: Dockerfile + container_name: ipsextractor-extraction + depends_on: + database: + condition: service_healthy + environment: + DATABASE_URL: postgresql+psycopg://ipsextractor:ipsextractor@database:5432/ipsextractor + EXTRACTOR_PROVIDER: ${EXTRACTOR_PROVIDER:-} + EXTRACTOR_BASE_URL: ${EXTRACTOR_BASE_URL:-} + EXTRACTOR_API_KEY: ${EXTRACTOR_API_KEY:-} + EXTRACTOR_MODEL: ${EXTRACTOR_MODEL:-} + NORMALIZER_PROVIDER: ${NORMALIZER_PROVIDER:-} + NORMALIZER_BASE_URL: ${NORMALIZER_BASE_URL:-} + NORMALIZER_API_KEY: ${NORMALIZER_API_KEY:-} + NORMALIZER_MODEL: ${NORMALIZER_MODEL:-} + NORMALIZER_TIMEOUT_S: ${NORMALIZER_TIMEOUT_S:-0} + EXTRACTOR_TIMEOUT_S: ${EXTRACTOR_TIMEOUT_S:-120} + PDF_DPI: ${PDF_DPI:-600} + PDF_THREAD_COUNT: ${PDF_THREAD_COUNT:-10} + POPPLER_PATH: ${POPPLER_PATH:-} + CONCURRENCY_LIMIT: ${CONCURRENCY_LIMIT:-4} + ALLOWED_DOCUMENT_TYPES: ${ALLOWED_DOCUMENT_TYPES:-CC,TI,CE,RC,PA,PPT,PEP,EPT,NUIP} + ports: + - "8001:8001" + volumes: + - ${IPS_DOCUMENT_ROOT:-./data}:/data + + cleaning: + build: + context: ./services/cleaning + dockerfile: Dockerfile + container_name: ipsextractor-cleaning + depends_on: + database: + condition: service_healthy + environment: + DATABASE_URL: postgresql+psycopg://ipsextractor:ipsextractor@database:5432/ipsextractor + ports: + - "8002:8002" + + api_gateway: + build: + context: ./services/api_gateway + dockerfile: Dockerfile + container_name: ipsextractor-api + depends_on: + - extraction + - cleaning + environment: + EXTRACTION_URL: http://extraction:8001 + CLEANING_URL: http://cleaning:8002 + ports: + - "8000:8000" + +volumes: + ipsextractor_db: diff --git a/gui/__init__.py b/gui/__init__.py deleted file mode 100644 index f30ec26..0000000 --- a/gui/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -"""Módulo de GUI.""" -from gui.view import InvoiceAppView -from gui.styles import AppStyles - -__all__ = ["InvoiceAppView", "AppStyles"] diff --git a/gui/controller.py b/gui/controller.py deleted file mode 100644 index 658bdbf..0000000 --- a/gui/controller.py +++ /dev/null @@ -1,1410 +0,0 @@ -"""Controlador de la aplicación GUI: coordina vista y servicio. - -Gestiona selección de carpetas, ejecución en hilo de procesamiento, selector -interactivo de campos y actualización de configuración del extractor. -""" -import json -import logging -import threading -import asyncio -import tkinter.filedialog as fd -import tkinter.messagebox as mb -import os - -from core.utils import parse_comma_separated, collect_radicado_docs, collect_documents_flat -from application.logging_config import configure_logging -from application.app_prefs import ( - build_cop_mod_config_for_apply, - load_app_prefs, - merge_moderadora_prefs, - normalize_document_types, - parse_moderadora_text, - save_app_prefs, -) -from gui.field_selector import FieldSelectorDialog -from application.preclassification import PreclassificationService -from application.strategies import OllamaExtractor -from application.database import DatabaseManager -from application.config import settings, extractor_credentials_for_model, is_grok_extractor_model -from application.costs import MODEL_PRICING -from application import postprocess_agent -from application import csv_cleaning -from openai import OpenAI -import pandas as pd - - -logger = logging.getLogger(__name__) - -# Etiquetas antiguas del combo → etiquetas actuales (prefs / estado previo) -_LEGACY_SETTINGS_MODEL_LABEL = { - "5.4": "gpt-5.4", - "5.4-mini": "gpt-5.4-mini", - "5.4-nano": "gpt-5.4-nano", - "5.2": "gpt-5.4", - "gpt-5.2": "gpt-5.4", - "4.1-mini": "gpt-4.1-mini", -} - - -def _canonical_settings_model_label(name: str) -> str: - s = (name or "").strip() - return _LEGACY_SETTINGS_MODEL_LABEL.get(s, s) - - -class AppController: - """Controlador principal de la GUI que orquesta la vista y el servicio. - - Se encarga de: - - Vincular callbacks de la vista. - - Disparar el flujo de procesamiento de facturas. - - Abrir el diálogo de selección de campos cuando el servicio lo solicite. - """ - def __init__(self, view, service): - """Inicializa el controlador y vincula handlers de la vista. - - Args: - view: Instancia de vista (por ejemplo, `InvoiceAppView`). - service: Servicio de facturas compatible con `process_root_with_field_selection`. - """ - # asegurar logging configurado - try: - configure_logging() - except Exception: - pass - self.view = view - self.service = service - # Acumuladores de uso - self._total_prompt_tokens = 0 - self._total_cached_tokens = 0 - self._total_completion_tokens = 0 - self._total_usd = 0.0 - # Estado para dashboard - self._hist_requests = [] # list[dict]: {radicado, phase, prompt, cached, completion, cost_usd} - self._per_doc_costs = {} # radicado -> {"extract": float|None, "normalize": float|None} - self._completed_doc_count = 0 - self._phase_tokens = { - "extract": {"input": 0, "cached": 0, "output": 0, "usd": 0.0}, - "normalize": {"input": 0, "cached": 0, "output": 0, "usd": 0.0}, - } - # Guard para evitar doble lanzamiento de worker2 (por doble click o condiciones de carrera) - self._worker2_running = False - - # Estado Preclasificación - self._last_run_id = None - self.db_manager = DatabaseManager() # Para pestaña resultados - - self.view.set_on_select_root(self.on_select_root) - try: - self.view.set_on_select_preclass_root(self.on_select_preclass_root) - self.view.set_on_run_preclassification(self.on_run_preclassification) - self.view.set_on_export_excel(self.on_export_excel) - self.view.set_on_organize_files(self.on_organize_files) - - # Resultados - self.view.set_on_res_search(self.on_res_search) - self.view.set_on_res_export(self.on_res_export) - self.view.set_on_res_refresh(self.on_res_refresh) - self.view.set_on_res_organize(self.on_res_organize_run) - self.view.set_on_res_select(self.on_res_select) - - # Carga inicial - self.on_res_refresh() - except Exception: - pass - self.view.set_on_process(self.on_process) - self.view.set_on_select_csv(self.on_select_csv) - self.view.set_on_save_settings(self.on_save_settings) - - # Post-procesamiento - self._pp_df: pd.DataFrame | None = None - self._pp_result_df: pd.DataFrame | None = None - self._pp_csv_sample: str = "" - self._pp_client = OpenAI(api_key=settings.api.api_key, base_url=settings.api.base_url) - # Métricas acumuladas de post-procesamiento - self._pp_total_usd = 0.0 - self._pp_total_calls = 0 - self._pp_total_retries = 0 - self._pp_total_input_tokens = 0 - self._pp_total_output_tokens = 0 - try: - self.view.set_on_postprocess_select_csv(self.on_pp_select_csv) - self.view.set_on_postprocess_run(self.on_pp_run) - self.view.set_on_postprocess_save(self.on_pp_save) - except Exception: - pass - - # Proceso de limpieza (sin LLM) - self._cleaning_df: pd.DataFrame | None = None - self._cleaning_result_df: pd.DataFrame | None = None - try: - self.view.set_on_cleaning_select_csv(self.on_cleaning_select_csv) - self.view.set_on_cleaning_apply(self.on_cleaning_apply) - self.view.set_on_cleaning_save(self.on_cleaning_save) - except Exception: - pass - - # Mapeo de nombres de UI a IDs de modelo (y viceversa) - self.ui_to_model = { - "gpt-5.4": "gpt-5.4", - "gpt-5.4-mini": "gpt-5.4-mini", - "gpt-5.4-nano": "gpt-5.4-nano", - "gpt-4.1-mini": "gpt-4.1-mini", - "grok-4.3": "grok-4.3", - } - self.model_to_ui = {v: k for k, v in self.ui_to_model.items()} - - # Inicializar vista con configuración actual - try: - # Llenar tabla de precios de modelos - pricing_data = self._get_model_pricing_data() - self.view.update_comparison_table(pricing_data) - - current_model = getattr(self.service.extractor, "model", "") - # Intentar mapear modelo backend -> nombre UI, si no existe usar raw - ui_name = self.model_to_ui.get(current_model, current_model) - if ui_name: - self.view.set_selected_model(ui_name) - - # Cargar concurrencia - self.view.set_concurrency_limit(settings.processing.concurrency_limit) - except Exception: - pass - - self._load_persisted_preferences() - try: - self.view.protocol("WM_DELETE_WINDOW", self.on_app_close) - except Exception: - pass - - def _get_model_pricing_data(self): - """Obtiene las tarifas por 1M de tokens para cada modelo.""" - results = [] - for model_id, rates in MODEL_PRICING.items(): - results.append({ - "model": self.model_to_ui.get(model_id, model_id), - "input": rates["input"], - "cached_input": rates["cached_input"], - "output": rates["output"] - }) - # Ordenar por precio de input descendente - results.sort(key=lambda x: x["input"], reverse=True) - return results - - def _apply_extractor_model(self, real_model_id: str, *, interactive: bool = False) -> bool: - """Aplica modelo y credenciales del extractor (Grok usa GROK_API_KEY / GROK_BASE_URL).""" - if not hasattr(self.service.extractor, "update_config"): - return True - if is_grok_extractor_model(real_model_id): - api_key, base_url = extractor_credentials_for_model(real_model_id) - if not api_key: - msg = "Define GROK_API_KEY en el entorno o en .env para usar modelos Grok." - if interactive: - mb.showwarning("Configuración", msg) - else: - logger.warning("Modelo persistido %s no aplicado: %s", real_model_id, msg) - return False - self.service.extractor.update_config(api_key=api_key, base_url=base_url, model=real_model_id) - return True - api_key, base_url = extractor_credentials_for_model(real_model_id) - self.service.extractor.update_config(api_key=api_key, base_url=base_url, model=real_model_id) - return True - - def on_save_settings(self): - """Actualiza la configuración del servicio (modelo LLM y concurrencia).""" - selected_name = self.view.get_selected_model() - concurrency = self.view.get_concurrency_limit() - - if not selected_name: - mb.showwarning("Configuración", "El modelo no puede estar vacío.") - return - - if concurrency <= 0: - mb.showwarning("Configuración", "El límite de concurrencia debe ser mayor a 0.") - return - - # Traducir nombre UI -> ID modelo real - canonical_ui = _canonical_settings_model_label(selected_name) - real_model_id = self.ui_to_model.get(canonical_ui, canonical_ui) - - try: - if not self._apply_extractor_model(real_model_id, interactive=True): - return - - # Actualizar settings globales - settings.processing.concurrency_limit = concurrency - self.view.set_allowed_document_types_hint(self.view.get_allowed_document_types()) - self._save_preferences() - - mb.showinfo("Configuración", f"Configuración actualizada.\nModelo: {canonical_ui}\nConcurrencia: {concurrency}") - self.view.append_output_safe(f"Configuración actualizada: Modelo={real_model_id}, Concurrencia={concurrency}") - except Exception as e: - logger.exception("Error actualizando configuración") - mb.showerror("Error", f"No se pudo actualizar la configuración: {e}") - - def on_select_root(self): - """Abre un selector de carpeta y establece la ruta raíz en la vista. - - Además, si el primer documento detectado es TIFF, muestra los controles - de rango TIFF para que el usuario pueda ajustar el índice antes de procesar. - """ - selected = fd.askdirectory() - if not selected: - return - self.view.set_root_path(selected) - self._save_preferences() - - # Detectar si el primer documento a procesar es TIFF para esta carpeta - try: - keywords_str = self.view.get_keywords() - keywords = parse_comma_separated(keywords_str) or ["Factura_O_Documento_Equivalente", "factura"] - except Exception: - keywords = ["Factura_O_Documento_Equivalente", "factura"] - has_tiff_first = False - try: - preview_entries = collect_radicado_docs(selected, keywords) - if preview_entries: - first_path = str(preview_entries[0].get("path") or "").lower() - has_tiff_first = first_path.endswith((".tif", ".tiff")) - except Exception: - has_tiff_first = False - try: - self.view.show_tiff_range_controls(has_tiff_first) - except Exception: - pass - - def on_select_preclass_root(self): - """Abre un selector de carpeta y establece la ruta raíz para preclasificación.""" - selected = fd.askdirectory() - if not selected: - return - self.view.set_preclass_root_path(selected) - - def on_run_preclassification(self): - """Ejecuta la preclasificación en un hilo separado.""" - # 1. Leer inputs de la UI - root = self.view.get_preclass_root_path() - patterns = self.view.get_preclass_patterns() - target_info = self.view.get_preclass_target_info() - page_range = self.view.get_preclass_range() - mode = self.view.get_preclass_mode() - - # Validaciones básicas - if not root or not os.path.isdir(root): - mb.showwarning("Error", "Seleccione una carpeta raíz válida.") - return - - if not patterns: - mb.showwarning("Error", "Debe agregar al menos un patrón.") - return - - if target_info["enabled"] and not target_info["pattern"]: - mb.showwarning("Error", "Si habilita archivo objetivo, debe especificar un patrón de archivo.") - return - - # Bloquear UI - self.view.set_processing(True) - self.view.enable_preclass_actions(False) - self.view.append_preclass_output("Iniciando preclasificación...") - self.view.append_preclass_output(f"Raíz: {root}") - self.view.append_preclass_output(f"Modo: {mode}") - self.view.append_preclass_output(f"Patrones: {len(patterns)}") - if target_info["enabled"]: - self.view.append_preclass_output(f"Filtro: {target_info['pattern']}") - - # Instanciar servicio (usando el extractor actual) - # Nota: PreclassificationService maneja su propia instancia de DB - pre_service = PreclassificationService(self.service.extractor) - - def worker(): - try: - def on_progress(radicado, idx, total): - pct = int((idx / total) * 100) - msg = f"[{idx}/{total}] Procesando radicado: {radicado} ({pct}%)" - self.view.append_preclass_output_safe(msg) - - run_id = pre_service.run_preclassification( - root_path=root, - mode=mode, - patterns=patterns, - target_info=target_info, - page_range=page_range, - on_progress=on_progress - ) - - self.view.append_preclass_output_safe(f"\n--- Ejecución finalizada (ID: {run_id}) ---") - - # Guardar ID - self._last_run_id = run_id - def _refresh_results_tab(): - # Recargar runs, seleccionar el run recién ejecutado y listar sus resultados - try: - self.on_res_refresh() - self.view.set_res_run(run_id) - self.on_res_search() - except Exception: - pass - try: - self.view.enable_preclass_actions(True) - except Exception: - pass - self.view.after(0, _refresh_results_tab) - - # Mostrar resumen rápido - # Usar tabla results (1 fila = 1 documento), no el join con matches - docs, _ = self.db_manager.search_results( - run_id=run_id, status="Todos", text=None, relax_if_empty=False - ) - match_count = sum(1 for r in docs if r.get("status") == "MATCH") - self.view.append_preclass_output_safe(f"Total procesados: {len(docs)}") - self.view.append_preclass_output_safe(f"Coincidencias (MATCH): {match_count}") - - # Detalle opcional - for res in docs[:10]: # mostrar primeros 10 - status_icon = "✅" if res.get("status") == "MATCH" else "❌" - self.view.append_preclass_output_safe(f"{status_icon} {res.get('radicado')} ({res.get('status')})") - - if len(docs) > 10: - self.view.append_preclass_output_safe("... ver base de datos para más detalles.") - - self.view.after(0, lambda: mb.showinfo("Completado", f"Preclasificación finalizada.\nCoincidencias: {match_count}/{len(docs)}")) - - except Exception as e: - logger.exception("Error en preclasificación") - self.view.append_preclass_output_safe(f"ERROR CRÍTICO: {str(e)}") - self.view.after(0, lambda: mb.showerror("Error", f"Ocurrió un error: {e}")) - finally: - self.view.set_processing(False) - - threading.Thread(target=worker, daemon=True).start() - - def on_export_excel(self): - """Exporta los resultados de la última preclasificación a Excel.""" - if not self._last_run_id: - mb.showwarning("Aviso", "No hay resultados de preclasificación recientes para exportar.") - return - - path = fd.asksaveasfilename( - defaultextension=".xlsx", - filetypes=[("Excel", "*.xlsx"), ("CSV", "*.csv"), ("Todos", "*.*")] - ) - if not path: - return - - try: - pre_service = PreclassificationService(self.service.extractor) - ok = pre_service.export_to_excel(self._last_run_id, path) - if ok: - mb.showinfo("Exportar", "Archivo exportado correctamente.") - else: - mb.showerror("Exportar", "No se encontraron datos para exportar.") - except Exception as e: - logger.exception("Error exportando excel") - mb.showerror("Error", f"Error al exportar: {e}") - - def on_organize_files(self): - """Organiza los archivos MATCH en carpetas (Run actual).""" - if not self._last_run_id: - mb.showwarning("Aviso", "No hay resultados para organizar.") - return - self._organize_run_generic(self._last_run_id) - - def _organize_run_generic(self, run_id: int): - target_dir = fd.askdirectory(title="Seleccionar carpeta de destino para organizar") - if not target_dir: - return - - if not mb.askyesno("Confirmar", "¿Está seguro de copiar y organizar los archivos en la carpeta seleccionada?\nEsta acción creará una estructura de carpetas basada en los datos extraídos."): - return - - try: - # Usamos append_preclass_output si estamos en esa pestaña, pero desde Resultados? - # El metodo de view inserta en el widget de log. Si estamos en tab Resultados, no se ve. - # Pero no importa, mostramos mb.showinfo al final. - pre_service = PreclassificationService(self.service.extractor) - ok_count, err_count = pre_service.organize_files(run_id, target_dir) - - msg = f"Organización finalizada.\nCopiados exitosos: {ok_count}\nErrores: {err_count}" - mb.showinfo("Organizar", msg) - except Exception as e: - logger.exception("Error organizando archivos") - mb.showerror("Error", f"Error al organizar: {e}") - - # --- Pestaña Resultados --- - def on_res_select(self, event): - """Muestra detalles del resultado seleccionado.""" - item_vals = self.view.get_selected_result_values() - if not item_vals: - return - try: - res_id = int(item_vals[0]) - matches = self.db_manager.get_matches_for_result(res_id) - self.view.update_res_matches_table(matches) - except Exception as e: - logger.error(f"Error cargando detalle: {e}") - - def on_res_organize_run(self): - """Organiza archivos del Run seleccionado en el filtro.""" - filters = self.view.get_res_filters() - run_val = filters["run_id"] - - if run_val == "Todos" or not run_val: - mb.showwarning("Organizar", "Por favor seleccione un 'Run ID' específico en el filtro para organizar.") - return - - try: - run_id = int(run_val) - self._organize_run_generic(run_id) - except ValueError: - mb.showerror("Error", "ID de Run inválido.") - - def on_res_refresh(self): - """Refresca la lista de Runs en el combo.""" - try: - runs = self.db_manager.get_all_runs() - ids = [r["id"] for r in runs] - self.view.update_res_runs_combo(ids) - # Tambien refrescar tabla - self.on_res_search() - except Exception as e: - logger.error(f"Error refrescando runs: {e}") - - def on_res_search(self): - """Busca resultados con los filtros actuales.""" - self.view.clear_res_filter_note() - filters = self.view.get_res_filters() - run_id = None - if filters["run_id"] != "Todos": - try: - run_id = int(filters["run_id"]) - except: pass - - try: - rows, relax_note = self.db_manager.search_results( - run_id=run_id, - status=filters["status"], - text=filters["text"], - relax_if_empty=bool(filters.get("relax_on_empty", False)), - ) - self.view.update_res_table(rows) - if relax_note: - self.view.set_res_filter_note(relax_note) - except Exception as e: - logger.error(f"Error buscando resultados: {e}") - mb.showerror("Error", f"Error en búsqueda: {e}") - - def on_res_export(self): - """Exporta la vista actual de resultados a Excel.""" - path = fd.asksaveasfilename( - defaultextension=".xlsx", - filetypes=[("Excel", "*.xlsx"), ("CSV", "*.csv")] - ) - if not path: - return - - # Obtener los datos actuales (repite la busqueda) - # Podriamos optimizar guardando el ultimo result set - try: - filters = self.view.get_res_filters() - run_id = None - if filters["run_id"] != "Todos": - try: run_id = int(filters["run_id"]) - except: pass - - rows, _ = self.db_manager.search_results( - run_id=run_id, - status=filters["status"], - text=filters["text"], - relax_if_empty=bool(filters.get("relax_on_empty", False)), - ) - - if not rows: - mb.showwarning("Exportar", "No hay datos para exportar.") - return - - df = pd.DataFrame(rows) - if path.endswith(".csv"): - df.to_csv(path, index=False, sep=";") - else: - df.to_excel(path, index=False) - - mb.showinfo("Exportar", "Datos exportados correctamente.") - except Exception as e: - logger.exception("Error exportando vista") - mb.showerror("Error", f"Error al exportar: {e}") - - def on_select_csv(self): - """Abre un diálogo de guardado y establece la ruta del CSV de salida.""" - selected = fd.asksaveasfilename(defaultextension=".csv", filetypes=[("CSV", "*.csv"), ("Todos", "*.*")]) - if selected: - self.view.set_csv_path(selected) - self._save_preferences() - - def _collect_preferences(self) -> dict: - """Toma una foto de la configuración visible de la app.""" - try: - doc_types = self.view.get_doc_types() - except Exception: - doc_types = [] - - return { - "selected_model": self.view.get_selected_model(), - "concurrency_limit": self.view.get_concurrency_limit(), - "allowed_document_types": self.view.get_allowed_document_types(), - "retry_on_empty": self.view.get_retry_empty_enabled(), - "process_mode": self.view.get_process_mode(), - "root_path": self.view.get_root_path(), - "csv_path": self.view.get_csv_path(), - "doc_types": doc_types, - "moderadora_by_year": parse_moderadora_text(self.view.get_moderadora_cuotas_text()), - "fecha_servicio_column_default": self.view.get_fecha_servicio_column_hint(), - } - - def _save_preferences(self) -> None: - """Guarda preferencias de la GUI en disco.""" - save_app_prefs(self._collect_preferences()) - - def _load_persisted_preferences(self) -> None: - """Carga preferencias persistidas y las aplica a la vista/servicio.""" - prefs = load_app_prefs() - if not prefs: - return - - try: - selected_model = str(prefs.get("selected_model") or "").strip() - if selected_model: - canonical_ui = _canonical_settings_model_label(selected_model) - self.view.set_selected_model(canonical_ui) - real_model_id = self.ui_to_model.get(canonical_ui, canonical_ui) - self._apply_extractor_model(real_model_id, interactive=False) - except Exception: - logger.exception("No se pudo restaurar el modelo persistido") - - try: - concurrency = int(prefs.get("concurrency_limit") or settings.processing.concurrency_limit) - self.view.set_concurrency_limit(concurrency) - settings.processing.concurrency_limit = concurrency - except Exception: - pass - - try: - allowed_document_types = normalize_document_types(prefs.get("allowed_document_types")) - self.view.set_allowed_document_types(allowed_document_types) - self.view.set_allowed_document_types_hint(allowed_document_types) - except Exception: - pass - - try: - self.view.set_retry_empty_enabled(bool(prefs.get("retry_on_empty", False))) - except Exception: - pass - - try: - self.view.set_process_mode(str(prefs.get("process_mode") or "radicados")) - except Exception: - pass - - try: - root_path = str(prefs.get("root_path") or "").strip() - if root_path: - self.view.set_root_path(root_path) - except Exception: - pass - - try: - csv_path = str(prefs.get("csv_path") or "").strip() - if csv_path: - self.view.set_csv_path(csv_path) - except Exception: - pass - - try: - doc_types = prefs.get("doc_types") - if isinstance(doc_types, list): - self.view.set_doc_types(doc_types) - except Exception: - pass - - try: - merged = merge_moderadora_prefs(prefs.get("moderadora_by_year")) - self.view.set_moderadora_cuotas_ui( - merged, - str(prefs.get("fecha_servicio_column_default") or ""), - ) - except Exception: - pass - - def on_app_close(self): - """Guarda preferencias antes de cerrar la ventana.""" - try: - self._save_preferences() - finally: - self.view.destroy() - - def on_process(self): - """Inicia el flujo de procesamiento por radicado (solo facturas) en un hilo de fondo. - - - Si el servicio requiere selección de campos, abre `FieldSelectorDialog` y relanza. - - Emite resultados en el área de salida y opcionalmente escribe CSV incremental. - """ - self._save_preferences() - root = self.view.get_root_path() - # Tipos de Documento (multi-doc) - doc_types = [] - try: - doc_types = self.view.get_doc_types() - except Exception: - doc_types = [] - if doc_types: - try: - ok, msg = self.view.validate_doc_types() - except Exception: - ok, msg = True, None - if not ok: - self.view.append_output_safe(msg or "Tipos de Documento inválidos.") - return - keywords_str = self.view.get_keywords() - fields_str = self.view.get_fields() - structure_mode = self.view.get_process_mode() - retry_on_empty = self.view.get_retry_empty_enabled() - - # Filtro de radicados - radicado_filter = None - try: - if self.view.get_radicado_filter_enabled(): - radicado_filter = self.view.get_radicado_filter_list() or None - except Exception: - radicado_filter = None - - if structure_mode == "general": - # En modo general, si está vacío o es el default, procesar todo (None) - # Si el usuario escribió algo específico, usarlo. - kws = parse_comma_separated(keywords_str) - # Check if it matches default legacy keywords to avoid accidental filtering - if kws == ["Factura_O_Documento_Equivalente", "factura"]: - keywords = None - else: - keywords = kws if kws else None - else: - keywords = parse_comma_separated(keywords_str) or ["Factura_O_Documento_Equivalente", "factura"] - - fields = parse_comma_separated(fields_str) if fields_str else None - - # Validar ruta antes de deshabilitar la UI - if not root or not os.path.isdir(root): - self.view.append_output_safe(f"Error: La ruta raíz no es un directorio: {root or ''}") - return - - # Si es modo general, ignorar doc_types - if structure_mode == "general": - doc_types = [] - - # Determinar si el primer documento a procesar es TIFF; si no lo es, ocultar - # los controles de rango TIFF y no aplicar ningún recorte. - tiff_page_range = None - has_tiff_first = False - try: - if structure_mode == "general": - preview_entries = collect_documents_flat(root, keywords) - else: - preview_entries = collect_radicado_docs(root, keywords) - - if preview_entries: - first_path = str(preview_entries[0].get("path") or "").lower() - has_tiff_first = first_path.endswith((".tif", ".tiff")) - except Exception: - has_tiff_first = False - try: - # Mostrar/ocultar controles en función de si el primer doc es TIFF - self.view.show_tiff_range_controls(has_tiff_first) - except Exception: - pass - - if has_tiff_first: - # Rango opcional de páginas TIFF (1-based; 0 = sin límite) - try: - tiff_start_raw = int(getattr(self.view, "get_tiff_start_page", lambda: 0)() or 0) - except Exception: - tiff_start_raw = 0 - try: - tiff_end_raw = int(getattr(self.view, "get_tiff_end_page", lambda: 0)() or 0) - except Exception: - tiff_end_raw = 0 - - if tiff_start_raw < 0 or tiff_end_raw < 0: - self.view.append_output_safe("Error: Las páginas TIFF deben ser mayores o iguales a 0.") - return - - tiff_start = tiff_start_raw if tiff_start_raw > 0 else None - tiff_end = tiff_end_raw if tiff_end_raw > 0 else None - if tiff_start is not None and tiff_end is not None and tiff_start > tiff_end: - self.view.append_output_safe("Error: La página de inicio TIFF no puede ser mayor que la página de fin.") - return - # Si ambos son 0 (None/None), el usuario quiere "todas las páginas": - # usamos el centinela (None, None) para indicar "no recortar TIFF". - if tiff_start is None and tiff_end is None: - tiff_page_range = (None, None) - else: - tiff_page_range = (tiff_start, tiff_end) - - def worker(): - resultados = None - if doc_types: - try: - self.view.append_output_safe(f"Procesando raíz: {root} | tipos: {[(dt.get('name'), dt.get('keyword')) for dt in doc_types]}") - except Exception: - self.view.append_output_safe(f"Procesando raíz: {root} | tipos: {len(doc_types)}") - else: - self.view.append_output_safe(f"Procesando raíz: {root} | keywords: {keywords}") - # limpiar pestaña Uso - try: - self._total_prompt_tokens = 0 - self._total_cached_tokens = 0 - self._total_completion_tokens = 0 - self._total_usd = 0.0 - self._hist_requests.clear() - self._per_doc_costs.clear() - self._completed_doc_count = 0 - self._phase_tokens = { - "extract": {"input": 0, "cached": 0, "output": 0, "usd": 0.0}, - "normalize": {"input": 0, "cached": 0, "output": 0, "usd": 0.0}, - } - self.view.clear_usage() - except Exception: - pass - try: - def on_progress(radicado, idx, total): - pct = int((idx / total) * 100) - self.view.append_output_safe(f"Progreso: {idx}/{total} ({pct}%) - radicado {radicado}") - - csv_path = self.view.get_csv_path() or None - - # Primera ejecución: si fields es None, el servicio devolverá _needs_fields - if doc_types: - resultados = self.service.process_root_with_field_selection( - root_path=root, - doc_types=doc_types, - fields=fields, - on_progress=on_progress, - csv_output_path=csv_path, - tiff_page_range=tiff_page_range, - structure_mode=structure_mode, - retry_on_empty=retry_on_empty, - radicado_filter=radicado_filter, - ) - else: - resultados = self.service.process_root_with_field_selection( - root_path=root, - keywords=keywords, - fields=fields, - on_progress=on_progress, - csv_output_path=csv_path, - tiff_page_range=tiff_page_range, - structure_mode=structure_mode, - retry_on_empty=retry_on_empty, - radicado_filter=radicado_filter, - ) - - # Si requiere selección de campos abrir diálogo; de lo contrario, mostrar resultados - if resultados and isinstance(resultados, list) and isinstance(resultados[0], dict) and resultados[0].get('_needs_fields'): - schema = resultados[0].get('schema') - sample = resultados[0].get('first_full') - first_pdf_path = resultados[0].get('_source_pdf') or "" - schema_by_type = resultados[0].get('schema_by_type') - first_full_by_type = resultados[0].get('first_full_by_type') - first_radicado = resultados[0].get('first_radicado') or "" - combined_schema = None - combined_sample = None - if isinstance(schema_by_type, dict) and schema_by_type: - combined_schema = {} - try: - for tname, sch in schema_by_type.items(): - if not isinstance(sch, dict): - continue - for path, typ in sch.items(): - combined_schema[f"{tname}/{path}"] = typ - except Exception: - combined_schema = {} - # Construir muestra combinada para el botón "Ver muestra" - try: - combined_sample = {"radicado": first_radicado} - if isinstance(first_full_by_type, dict): - for tname, s in first_full_by_type.items(): - combined_sample[tname] = s or {} - except Exception: - combined_sample = None - - def on_submit(selected_fields, alias_map): - if not selected_fields: - self.view.append_output_safe("No se seleccionaron campos.") - self.view.set_processing(False) - return - # Estimar tokens/costos para toda la carpeta en hilo de fondo y pedir confirmación en hilo principal - def do_estimate_and_prompt(): - try: - if doc_types and schema_by_type: - est = self.service.estimate_tokens_and_cost( - root_path=root, - doc_types=doc_types, - schema_by_type=schema_by_type, - selected_paths=selected_fields, - structure_mode=structure_mode, - ) - else: - est = self.service.estimate_tokens_and_cost( - root_path=root, - keywords=keywords, - schema=schema, - selected_fields=selected_fields, - structure_mode=structure_mode, - ) - total_docs = est.get("total_docs", 0) - per_doc_cost = est.get("per_doc_cost", {}) or {} - estimated_total = est.get("estimated_total_cost", {}) or {} - msg = ( - f"Se procesarán {total_docs} documentos.\n\n" - f"Costo estimado por documento (USD): {per_doc_cost.get('total_usd', 'N/D')}\n" - f"Costo estimado total (USD): {estimated_total.get('total_usd', 'N/D')}\n\n" - f"¿Desea continuar?" - ) - except Exception as e_est: - msg = f"No se pudo calcular la estimación ({e_est}). ¿Desea continuar sin estimación?" - - def prompt_and_continue(): - ok = mb.askyesno("Confirmar estimación", msg) - if not ok: - self.view.set_processing(False) - return - # Evitar doble arranque - if self._worker2_running: - return - self._worker2_running = True - # Si acepta, lanzar procesamiento en otro hilo - threading.Thread(target=worker2, daemon=True).start() - - # Volver al hilo principal para mostrar el cuadro de diálogo - self.view.after(0, prompt_and_continue) - - threading.Thread(target=do_estimate_and_prompt, daemon=True).start() - def worker2(): - try: - loop = asyncio.new_event_loop() - asyncio.set_event_loop(loop) - - async def on_error_retry(error): - # Este callback corre en el loop de asyncio (worker thread) - future = loop.create_future() - - def ask(): - msg = ( - f"⚠️ Error de Cuota o Límite (429):\n\n{error}\n\n" - "¿Desea PAUSAR el proceso, añadir fondos/esperar y REINTENTAR este radicado?\n\n" - "Si selecciona 'No', el proceso se detendrá y se guardará lo avanzado hasta ahora." - ) - res = mb.askyesno("Límite de Cuota Excedido", msg) - # Regresar el resultado al loop de asyncio - loop.call_soon_threadsafe(future.set_result, res) - - self.view.after(0, ask) - return await future - - # callback para uso por petición - def on_usage(phase, radicado, usage, cost): - try: - cached = 0 - if isinstance(usage, dict): - pt = int(usage.get("prompt_tokens") or 0) - ct = int(usage.get("completion_tokens") or 0) - details = usage.get("prompt_tokens_details") or {} - cached = int(details.get("cached_tokens") or 0) - self._total_prompt_tokens += pt - self._total_cached_tokens += cached - self._total_completion_tokens += ct - if isinstance(cost, dict): - self._total_usd += float(cost.get("total_usd") or 0.0) - # Registrar histórico - per_cost = float((cost or {}).get("total_usd") or 0.0) - self._hist_requests.append({ - "radicado": radicado, "phase": phase, - "prompt": int((usage or {}).get("prompt_tokens") or 0), - "cached": cached, - "completion": int((usage or {}).get("completion_tokens") or 0), - "cost_usd": per_cost, - }) - # Actualizar per-doc y graficar cuando tengamos ambas fases - bucket = self._per_doc_costs.get(radicado) or {"extract": None, "normalize": None} - bucket[phase] = per_cost - self._per_doc_costs[radicado] = bucket - if bucket.get("extract") is not None and bucket.get("normalize") is not None: - self._completed_doc_count += 1 - total_cost_doc = float(bucket["extract"] or 0.0) + float(bucket["normalize"] or 0.0) - self.view.after(0, lambda idx=self._completed_doc_count, val=total_cost_doc: self.view.update_cost_line(idx, val)) - avg_cost = self._total_usd / max(self._completed_doc_count, 1) - self.view.after(0, lambda p=self._completed_doc_count, a=avg_cost: self.view.update_doc_kpi(p, a)) - # Actualizar totales por fase - if phase in self._phase_tokens: - self._phase_tokens[phase]["input"] += int((usage or {}).get("prompt_tokens") or 0) - self._phase_tokens[phase]["cached"] += cached - self._phase_tokens[phase]["output"] += int((usage or {}).get("completion_tokens") or 0) - self._phase_tokens[phase]["usd"] += per_cost - ex = self._phase_tokens["extract"] - nz = self._phase_tokens["normalize"] - pp_in = self._pp_total_input_tokens - pp_out = self._pp_total_output_tokens - self.view.after(0, lambda: self.view.update_token_bars( - ex["input"], ex["cached"], ex["output"], - nz["input"], nz["cached"], nz["output"], - pp_in, pp_out, - )) - # Actualizar KPIs - overall = { - "input": self._total_prompt_tokens, - "cached": self._total_cached_tokens, - "output": self._total_completion_tokens, - "usd": self._total_usd, - } - self.view.after(0, lambda ov=overall, exd=ex.copy(), nzd=nz.copy(): self.view.update_kpis(ov, exd, nzd)) - # Añadir fila textual - self.view.after(0, lambda p=phase, r=radicado, u=usage, c=cost: self.view.append_usage_row(p, r, u, c)) - # actualizar totales - self.view.set_usage_totals( - prompt_tokens=self._total_prompt_tokens, - cached_tokens=self._total_cached_tokens, - completion_tokens=self._total_completion_tokens, - total_usd=self._total_usd, - ) - except Exception: - pass - if doc_types: - res = self.service.process_root_with_field_selection( - root_path=root, - doc_types=doc_types, - fields=selected_fields, - on_progress=on_progress, - csv_output_path=csv_path, - on_usage=on_usage, - on_error_retry=on_error_retry, - alias_map=alias_map, - tiff_page_range=tiff_page_range, - structure_mode=structure_mode, - retry_on_empty=retry_on_empty, - radicado_filter=radicado_filter, - ) - else: - res = self.service.process_root_with_field_selection( - root_path=root, - keywords=keywords, - fields=selected_fields, - on_progress=on_progress, - csv_output_path=csv_path, - on_usage=on_usage, - on_error_retry=on_error_retry, - alias_map=alias_map, - tiff_page_range=tiff_page_range, - structure_mode=structure_mode, - retry_on_empty=retry_on_empty, - radicado_filter=radicado_filter, - ) - self.view.append_output_safe(json.dumps(res, ensure_ascii=False, indent=2)) - except Exception as e2: - logging.exception("Error en reproceso") - self.view.append_output_safe(f"Error: {e2}") - finally: - # Liberar guard al finalizar - self._worker2_running = False - self.view.set_processing(False) - self.view.after(0, lambda: mb.showinfo("Proceso Finalizado", "El procesamiento ha finalizado.")) - - # Preparar callback de reprocesamiento - reprocess_cb = None - if doc_types: - def _reprocess_multi(): - s_by_type, f_full_by_type, f_rad = self.service.build_schema_from_first_for_types( - root, doc_types, tiff_page_range=tiff_page_range, structure_mode=structure_mode - ) - c_schema = {} - try: - for tname, sch in s_by_type.items(): - if not isinstance(sch, dict): continue - for p, typ in sch.items(): - c_schema[f"{tname}/{p}"] = typ - except Exception: pass - - c_sample = {"radicado": f_rad} - try: - for tname, s in f_full_by_type.items(): - c_sample[tname] = s or {} - except Exception: pass - return c_schema, c_sample - reprocess_cb = _reprocess_multi - elif first_pdf_path: - def _reprocess_single(): - return self.service.build_schema_from_first(first_pdf_path, tiff_page_range=tiff_page_range) - reprocess_cb = _reprocess_single - - # Asegurar que siempre pasamos un schema dict (aunque sea vacío) al selector - effective_schema = combined_schema if combined_schema is not None else (schema or {}) - effective_sample = combined_sample if combined_sample is not None else sample - - self.view.after(0, lambda: self._open_field_selector(effective_schema, effective_sample, on_submit, reprocess_cb)) - else: - self.view.append_output_safe(json.dumps(resultados, ensure_ascii=False, indent=2)) - except Exception as e: - logger.exception("Error en procesamiento") - self.view.append_output_safe(f"Error: {e}") - finally: - # Si se abrió selector, se reactivará al terminar worker2 - if not (isinstance(resultados, list) and resultados and isinstance(resultados[0], dict) and resultados[0].get('_needs_fields')): - self.view.set_processing(False) - self.view.after(0, lambda: mb.showinfo("Proceso Finalizado", "El procesamiento ha finalizado.")) - - self.view.set_processing(True) - threading.Thread(target=worker, daemon=True).start() - - def _open_field_selector(self, schema, sample, on_submit, on_reprocess=None): - """Abre el diálogo de selección de campos. - - Args: - schema: Mapa path->tipo que ofrece el servicio para ayudar a seleccionar. - sample: JSON de muestra del primer documento. - on_submit: Callback `(selected_fields, alias_map)` a invocar al aceptar. - on_reprocess: Callback opcional para reprocesar muestra y schema. - """ - dlg = FieldSelectorDialog(self.view, schema=schema, sample=sample) - dlg.set_on_submit(on_submit) - if on_reprocess: - dlg.set_on_reprocess(on_reprocess) - - # ========================================================================= - # --- POST-PROCESAMIENTO --- - # ========================================================================= - - def on_pp_select_csv(self): - """Abre un selector de archivo CSV y muestra la vista previa.""" - filepath = fd.askopenfilename( - filetypes=[("CSV", "*.csv"), ("Todos", "*.*")], - title="Seleccionar CSV para post-procesamiento", - ) - if not filepath: - return - - try: - delimiter = postprocess_agent.detect_delimiter(filepath) - df = pd.read_csv(filepath, delimiter=delimiter, dtype=str) - except Exception as e: - mb.showerror("Error", f"No se pudo leer el CSV:\n{e}") - return - - self._pp_df = df - self._pp_result_df = None - self.view.set_postprocess_csv_path(filepath) - self.view.enable_postprocess_save(False) - self.view.set_postprocess_code("") - self.view.set_postprocess_result("") - self.view.clear_postprocess_log() - - sample_text, _ = postprocess_agent.sample_dataframe(df) - self._pp_csv_sample = sample_text - self.view.set_postprocess_preview(sample_text) - - def on_pp_run(self): - """Genera y ejecuta código de transformación con el agente LLM.""" - if self._pp_df is None: - mb.showwarning("Aviso", "Primero seleccione un archivo CSV.") - return - - instructions = self.view.get_postprocess_instructions() - if not instructions: - mb.showwarning("Aviso", "Escriba las instrucciones de transformación.") - return - - self.view.enable_postprocess_controls(False) - self.view.enable_postprocess_save(False) - self.view.clear_postprocess_log() - self.view.set_postprocess_code("") - self.view.set_postprocess_result("") - self.view.set_postprocess_status("Procesando...") - - csv_sample = self._pp_csv_sample - df = self._pp_df - - def track_usage(usage, is_retry=False): - self._pp_total_calls += 1 - if is_retry: - self._pp_total_retries += 1 - if isinstance(usage, dict): - inp = int(usage.get("input_tokens") or 0) - out = int(usage.get("output_tokens") or 0) - self._pp_total_input_tokens += inp - self._pp_total_output_tokens += out - cost = (inp / 1_000_000.0) * 0.25 + (out / 1_000_000.0) * 2.00 - self._pp_total_usd += cost - self.view.append_postprocess_log_safe( - f" Tokens: in={inp:,} out={out:,} | Costo: ${cost:.4f}" - ) - - def worker(): - max_retries = 3 - code = None - error_msg = None - - try: - self.view.append_postprocess_log_safe("Generando código con el agente...") - code, usage = postprocess_agent.generate_code( - self._pp_client, csv_sample, instructions, - ) - track_usage(usage) - self.view.after(0, lambda c=code: self.view.set_postprocess_code(c)) - self.view.append_postprocess_log_safe("Código generado. Ejecutando...") - - result_df, error_msg = postprocess_agent.execute_code(code, df) - - attempt = 1 - while error_msg and attempt < max_retries: - attempt += 1 - self.view.append_postprocess_log_safe( - f"Error en ejecución: {error_msg}\nReintentando ({attempt}/{max_retries})..." - ) - code, usage = postprocess_agent.retry_with_error( - self._pp_client, csv_sample, instructions, code, error_msg, - ) - track_usage(usage, is_retry=True) - self.view.after(0, lambda c=code: self.view.set_postprocess_code(c)) - self.view.append_postprocess_log_safe("Código corregido. Ejecutando...") - result_df, error_msg = postprocess_agent.execute_code(code, df) - - # Actualizar dashboard de PP - short_instr = instructions[:60] + ("..." if len(instructions) > 60 else "") - self.view.after(0, lambda: self.view.update_pp_kpis( - self._pp_total_usd, self._pp_total_calls, self._pp_total_retries, - self._pp_total_input_tokens, self._pp_total_output_tokens, - short_instr, - )) - self.view.after(0, lambda: self.view.append_usage_row( - "post-proc", short_instr, - {"prompt_tokens": self._pp_total_input_tokens, "completion_tokens": self._pp_total_output_tokens}, - {"total_usd": self._pp_total_usd}, - )) - - if error_msg: - self.view.append_postprocess_log_safe( - f"Error final tras {max_retries} intentos: {error_msg}" - ) - self.view.after(0, lambda: self.view.set_postprocess_status("Error")) - final_err = error_msg - self.view.after(0, lambda: mb.showerror( - "Error", f"No se pudo aplicar la transformación tras {max_retries} intentos:\n{final_err}", - )) - return - - self._pp_result_df = result_df - result_sample, _ = postprocess_agent.sample_dataframe(result_df) - self.view.after(0, lambda s=result_sample: self.view.set_postprocess_result(s)) - self.view.append_postprocess_log_safe( - f"Ejecución exitosa. Filas: {len(result_df):,} | Columnas: {len(result_df.columns)}" - ) - self.view.after(0, lambda: self.view.enable_postprocess_save(True)) - self.view.after(0, lambda: self.view.set_postprocess_status("Completado")) - - except Exception as exc: - logger.exception("Error en post-procesamiento") - err_str = str(exc) - self.view.append_postprocess_log_safe(f"Error inesperado: {err_str}") - self.view.after(0, lambda: self.view.set_postprocess_status("Error")) - self.view.after(0, lambda m=err_str: mb.showerror("Error", f"Error inesperado:\n{m}")) - finally: - self.view.after(0, lambda: self.view.enable_postprocess_controls(True)) - - threading.Thread(target=worker, daemon=True).start() - - def on_pp_save(self): - """Guarda el DataFrame resultante del post-procesamiento.""" - if self._pp_result_df is None: - mb.showwarning("Aviso", "No hay resultado para guardar.") - return - - path = fd.asksaveasfilename( - defaultextension=".csv", - filetypes=[("CSV", "*.csv"), ("Excel", "*.xlsx"), ("Todos", "*.*")], - title="Guardar CSV post-procesado", - ) - if not path: - return - - try: - if path.endswith(".xlsx"): - self._pp_result_df.to_excel(path, index=False) - else: - self._pp_result_df.to_csv(path, index=False) - mb.showinfo("Guardado", f"Archivo guardado correctamente:\n{path}") - except Exception as e: - logger.exception("Error guardando CSV post-procesado") - mb.showerror("Error", f"No se pudo guardar el archivo:\n{e}") - - # ========================================================================= - # --- PROCESO DE LIMPIEZA --- - # ========================================================================= - - def on_cleaning_select_csv(self): - """Abre un selector de CSV y prepara el resumen de columnas.""" - filepath = fd.askopenfilename( - filetypes=[("CSV", "*.csv"), ("Todos", "*.*")], - title="Seleccionar CSV para limpieza", - ) - if not filepath: - return - - self._cleaning_result_df = None - self.view.set_cleaning_csv_path(filepath) - self.view.enable_cleaning_controls(False) - self.view.enable_cleaning_save(False) - self.view.set_cleaning_result(None) - self.view.set_cleaning_preview(None) - self.view.set_cleaning_status("Cargando CSV...") - try: - self.view.set_cleaning_loading(True, "Leyendo archivo CSV...") - except Exception: - pass - - def worker(): - try: - df, delimiter = csv_cleaning.load_csv(filepath) - self.view.after( - 0, - lambda d=df, delim=delimiter: self._on_cleaning_csv_loaded( - d, delim - ), - ) - summaries = csv_cleaning.analyze_dataframe(df) - self._cleaning_df = df - self.view.after( - 0, - lambda d=df, s=summaries: self._finish_cleaning_csv_load(d, s), - ) - except Exception as e: - logger.exception("Error leyendo o analizando CSV para limpieza") - err = str(e) - self.view.after(0, lambda m=err: self._fail_cleaning_csv_load(m)) - - threading.Thread(target=worker, daemon=True).start() - - def _on_cleaning_csv_loaded(self, df: pd.DataFrame, delimiter: str): - self.view.set_cleaning_status( - f"CSV leído: {len(df):,} filas x {len(df.columns)} columnas. Analizando columnas..." - ) - try: - self.view.set_cleaning_loading(True, "Analizando columnas...") - self.view.set_cleaning_delimiter_chip(delimiter) - except Exception: - pass - - def _finish_cleaning_csv_load(self, df: pd.DataFrame, summaries): - self.view.set_cleaning_preview(df) - self.view.populate_cleaning_columns(summaries) - self.view.set_cleaning_status( - f"CSV cargado: {len(df):,} filas x {len(df.columns)} columnas." - ) - try: - self.view.set_cleaning_loading(False) - except Exception: - pass - self.view.enable_cleaning_controls(True) - - def _fail_cleaning_csv_load(self, message: str): - self._cleaning_df = None - self.view.set_cleaning_status("Error al cargar CSV") - try: - self.view.set_cleaning_loading(False) - except Exception: - pass - self.view.enable_cleaning_controls(True) - mb.showerror("Error", f"No se pudo cargar o analizar el CSV:\n{message}") - - def on_cleaning_apply(self): - """Aplica las reglas seleccionadas en un hilo y actualiza el resultado.""" - if self._cleaning_df is None: - mb.showwarning("Aviso", "Primero seleccione un archivo CSV.") - return - - rules = self.view.get_cleaning_selected_rules() - if not rules: - mb.showwarning( - "Aviso", - "No hay reglas seleccionadas. Marque al menos una regla para alguna columna.", - ) - return - - fecha_map = self.view.get_cleaning_cop_mod_fecha_columns() - for col_name, rule_ids in rules.items(): - if "cop_mod_split" in rule_ids and col_name not in fecha_map: - mb.showwarning( - "COP/MOD", - f"Marque «Separa COP/MOD» y elija la columna «Fecha Servicio» para «{col_name}».", - ) - return - - self.view.enable_cleaning_controls(False) - self.view.enable_cleaning_save(False) - self.view.set_cleaning_status("Aplicando reglas...") - self.view.set_cleaning_result(None) - - df = self._cleaning_df - - def worker(): - try: - cop_cfg = build_cop_mod_config_for_apply() - fecha_map = self.view.get_cleaning_cop_mod_fecha_columns() - result_df, log = csv_cleaning.apply_rules( - df, - rules, - cop_mod_fecha_column=fecha_map or None, - cop_mod_config=cop_cfg, - ) - self._cleaning_result_df = result_df - ok_count = sum(1 for line in log if line.startswith("[OK]")) - err_count = sum(1 for line in log if line.startswith("[ERROR]")) - warn_count = sum(1 for line in log if line.startswith("[WARN]")) - info = f"{ok_count} reglas aplicadas" - if err_count or warn_count: - info += f" · {err_count} errores · {warn_count} avisos" - self.view.after(0, lambda d=result_df, t=info: self.view.set_cleaning_result(d, t)) - self.view.after(0, lambda: self.view.enable_cleaning_save(True)) - status = ( - f"Completado: {ok_count} reglas aplicadas correctamente" - + (f", {err_count} errores" if err_count else "") - + (f", {warn_count} avisos" if warn_count else "") - + "." - ) - self.view.after(0, lambda s=status: self.view.set_cleaning_status(s)) - except Exception as exc: - logger.exception("Error aplicando reglas de limpieza") - err = str(exc) - self.view.after(0, lambda: self.view.set_cleaning_status("Error")) - self.view.after(0, lambda m=err: mb.showerror("Error", f"Error aplicando reglas:\n{m}")) - finally: - self.view.after(0, lambda: self.view.enable_cleaning_controls(True)) - - threading.Thread(target=worker, daemon=True).start() - - def on_cleaning_save(self): - """Guarda el DataFrame limpio en disco.""" - if self._cleaning_result_df is None: - mb.showwarning("Aviso", "No hay resultado para guardar.") - return - - path = fd.asksaveasfilename( - defaultextension=".csv", - filetypes=[("CSV", "*.csv"), ("Excel", "*.xlsx"), ("Todos", "*.*")], - title="Guardar CSV limpio", - ) - if not path: - return - - try: - if path.lower().endswith(".xlsx"): - self._cleaning_result_df.to_excel(path, index=False) - else: - self._cleaning_result_df.to_csv(path, index=False, sep="|") - mb.showinfo("Guardado", f"Archivo guardado correctamente:\n{path}") - except Exception as e: - logger.exception("Error guardando CSV limpio") - mb.showerror("Error", f"No se pudo guardar el archivo:\n{e}") diff --git a/gui/field_selector.py b/gui/field_selector.py deleted file mode 100644 index 3a12011..0000000 --- a/gui/field_selector.py +++ /dev/null @@ -1,386 +0,0 @@ -"""Diálogo de selección de campos basado en `customtkinter`. - -Permite buscar, seleccionar/deseleccionar y revisar una muestra de la primera -factura para facilitar la elección de paths del schema. -""" -from __future__ import annotations - -import json -from typing import Dict, List, Callable, Optional - -import customtkinter as ctk -import threading -import tkinter.messagebox as mb -from application.app_prefs import get_allowed_document_types - -DOCUMENT_TYPE_HINT_ALIASES = { - "CC", "TI", "CE", "RC", "PA", "PPT", "PEP", "EPT", "NUIP", - "PT", "PAS", "PASS", -} - - -class FieldSelectorDialog(ctk.CTkToplevel): - """Ventana modal para elegir campos del schema a extraer.""" - def __init__(self, master, schema: Dict[str, str], sample: Optional[dict] = None, initial_selected: Optional[List[str]] = None): - """Crea el diálogo con lista scrolleable y búsqueda. - - Args: - master: Ventana padre. - schema: Mapa path->tipo (ayuda visual en etiqueta del checkbox). - sample: JSON de muestra para vista previa. - initial_selected: Paths seleccionados inicialmente. - """ - super().__init__(master) - self.title("Seleccionar campos del schema") - self.geometry("800x600") - self.transient(master) - self.grab_set() - - self._schema = schema - self._sample = sample - self._on_submit: Optional[Callable[[List[str], Dict[str, str]], None]] = None - self._on_reprocess: Optional[Callable[[], tuple[Dict[str, str], dict]]] = None - - self.search_entry = ctk.CTkEntry(self, placeholder_text="Buscar campo...") - self.search_entry.grid(row=0, column=0, columnspan=3, sticky="we", padx=10, pady=(10, 5)) - self.search_entry.bind("", self._on_search) - - self.scroll = ctk.CTkScrollableFrame(self) - self.scroll.grid(row=1, column=0, columnspan=3, sticky="nsew", padx=10, pady=5) - - self.select_all_btn = ctk.CTkButton(self, text="Seleccionar todo", command=self._select_all) - self.select_all_btn.grid(row=2, column=0, padx=10, pady=10, sticky="w") - self.clear_btn = ctk.CTkButton(self, text="Limpiar", command=self._clear_all) - self.clear_btn.grid(row=2, column=1, padx=10, pady=10, sticky="w") - self._right_btns = ctk.CTkFrame(self) - self._right_btns.grid(row=2, column=2, padx=10, pady=10, sticky="e") - self.sample_btn = ctk.CTkButton(self._right_btns, text="Ver muestra", command=self._show_sample) - self.sample_btn.pack(side="left", padx=(0, 8)) - self.reprocess_btn = ctk.CTkButton(self._right_btns, text="Reprocesar", command=self._reprocess) - self.reprocess_btn.pack(side="left") - - self.cancel_btn = ctk.CTkButton(self, text="Cancelar", command=self._cancel) - self.cancel_btn.grid(row=3, column=0, padx=10, pady=(0, 10), sticky="w") - self.ok_btn = ctk.CTkButton(self, text="Aceptar", command=self._accept) - self.ok_btn.grid(row=3, column=2, padx=10, pady=(0, 10), sticky="e") - - self.grid_columnconfigure(0, weight=1) - self.grid_columnconfigure(1, weight=0) - self.grid_columnconfigure(2, weight=1) - self.grid_rowconfigure(1, weight=1) - - self._vars: Dict[str, ctk.BooleanVar] = {} - self._rows: Dict[str, ctk.CTkFrame] = {} - self._alias_entries: Dict[str, ctk.CTkEntry] = {} - self._group_labels: Dict[str, ctk.CTkLabel] = {} - self._group_paths: Dict[str, list[str]] = {} - - initial_set = set(initial_selected or []) - # Construir grupos por prefijo (Tipo) cuando el path contiene '/' - grouped: Dict[str, list[tuple[str, str]]] = {} - for path, typ in sorted(self._schema.items(), key=lambda x: x[0]): - if '/' in path: - group, sub = path.split('/', 1) - else: - group, sub = "General", path - grouped.setdefault(group, []).append((path, typ)) - # Crear filas agrupadas - rows_container = self.scroll - for group_name, items in grouped.items(): - # Encabezado del grupo - glabel = ctk.CTkLabel(rows_container, text=group_name, anchor="w") - glabel.pack(anchor="w", fill="x", padx=8, pady=(6, 2)) - self._group_labels[group_name] = glabel - self._group_paths[group_name] = [] - # Filas del grupo - for path, typ in items: - var = ctk.BooleanVar(value=(path in initial_set)) - row = ctk.CTkFrame(rows_container) - row.pack(anchor="w", fill="x", padx=8, pady=2) - cb = ctk.CTkCheckBox(row, text=f"{path} ({typ})", variable=var) - cb.grid(row=0, column=0, sticky="w") - alias_entry = ctk.CTkEntry(row, width=260, placeholder_text="Nombre en CSV") - try: - alias_entry.insert(0, path) - except Exception: - pass - alias_entry.grid(row=0, column=1, padx=(12, 0)) - self._add_document_type_hint(row, path) - try: - alias_entry.bind("", lambda _e: self._validate_aliases()) - except Exception: - pass - def _toggle(ent: ctk.CTkEntry, v: ctk.BooleanVar = var, p: str = path): - try: - if p == 'radicado': - ent.grid_remove() - else: - if v.get(): - ent.grid(row=0, column=1, padx=(12, 0)) - else: - ent.grid_remove() - except Exception: - pass - self._validate_aliases() - _toggle(alias_entry) - cb.configure(command=lambda e=alias_entry, v=var, p=path: _toggle(e, v, p)) - self._vars[path] = var - self._rows[path] = row - self._alias_entries[path] = alias_entry - self._group_paths[group_name].append(path) - self._validate_aliases() - - def set_on_submit(self, callback: Callable[[List[str], Dict[str, str]], None]) -> None: - """Registra el callback a invocar con (selected_fields, alias_map).""" - self._on_submit = callback - - def set_on_reprocess(self, callback: Callable[[], tuple[Dict[str, str], dict]]) -> None: - """Registra el callback a invocar para reprocesar y actualizar (schema, sample).""" - self._on_reprocess = callback - - def _on_search(self, _event=None): - """Filtra filas visibles por coincidencia de texto en el path.""" - query = self.search_entry.get().strip().lower() - - # Lista ordenada de grupos según se crearon - ordered_groups = list(self._group_labels.keys()) - - for group_name in ordered_groups: - paths = self._group_paths.get(group_name, []) - # Determinar qué filas de este grupo coinciden - visible_paths = [] - for p in paths: - if query in p.lower(): - visible_paths.append(p) - - # Si hay filas visibles, mostrar grupo y esas filas; si no, ocultar todo el grupo - lbl = self._group_labels.get(group_name) - if visible_paths: - if lbl: - lbl.pack(anchor="w", fill="x", padx=8, pady=(6, 2)) - for p in paths: - row = self._rows.get(p) - if not row: continue - if p in visible_paths: - row.pack(anchor="w", fill="x", padx=8, pady=2) - else: - row.pack_forget() - else: - if lbl: - lbl.pack_forget() - for p in paths: - row = self._rows.get(p) - if row: - row.pack_forget() - - def _select_all(self): - """Marca todos los campos y muestra las entradas de alias (excepto radicado).""" - for path, var in self._vars.items(): - var.set(True) - try: - ent = self._alias_entries.get(path) - if ent is not None: - if path == 'radicado': - ent.grid_remove() - else: - ent.grid(row=0, column=1, padx=(12, 0)) - except Exception: - pass - self._validate_aliases() - - def _clear_all(self): - """Desmarca todos los campos y oculta entradas de alias.""" - for path, var in self._vars.items(): - var.set(False) - try: - ent = self._alias_entries.get(path) - if ent is not None: - ent.grid_remove() - except Exception: - pass - self._validate_aliases() - - def _show_sample(self): - """Muestra la muestra JSON en un toplevel de solo lectura (si existe).""" - if self._sample is None: - return - win = ctk.CTkToplevel(self) - win.title("Muestra de la primera factura") - win.geometry("700x500") - win.transient(self) - win.lift() - try: - win.attributes("-topmost", True) - # mantener al frente un instante y luego permitir comportamiento normal - win.after(300, lambda: win.attributes("-topmost", False)) - except Exception: - pass - win.focus_force() - txt = ctk.CTkTextbox(win) - txt.pack(expand=True, fill="both", padx=10, pady=10) - txt.insert("end", json.dumps(self._sample, ensure_ascii=False, indent=2)) - txt.configure(state="disabled") - - def _accept(self): - """Construye selección y alias y cierra devolviendo datos al callback.""" - selected = [p for p, v in self._vars.items() if v.get()] - alias_map: Dict[str, str] = {} - for p in selected: - ent = self._alias_entries.get(p) - alias = None - try: - alias = (ent.get().strip() if ent is not None else "") - except Exception: - alias = "" - if not alias: - alias = p - # 'radicado' permanece fijo si aparece - if p == 'radicado': - alias = 'radicado' - alias_map[p] = alias - if self._on_submit: - self._on_submit(selected, alias_map) - self.destroy() - - def _cancel(self): - """Cancela la selección (devuelve listas vacías) y cierra el diálogo.""" - if self._on_submit: - self._on_submit([], {}) - self.destroy() - - def _validate_aliases(self): - """Valida alias. (Validación de duplicados eliminada para permitir fusión de columnas).""" - # Siempre permitir aceptar, ya que los duplicados se fusionarán en el backend. - try: - self.ok_btn.configure(state="normal") - except Exception: - pass - - - # --- Reprocesar (reextraer primer documento y refrescar lista) --- - def _set_busy(self, busy: bool) -> None: - """Habilita/deshabilita controles y cambia texto del botón Reprocesar.""" - try: - state = "disabled" if busy else "normal" - self.select_all_btn.configure(state=state) - self.clear_btn.configure(state=state) - self.sample_btn.configure(state=state) - self.ok_btn.configure(state=state) - self.cancel_btn.configure(state=state) - self.search_entry.configure(state=state) - self.reprocess_btn.configure(text=("Reprocesando…" if busy else "Reprocesar"), state=("disabled" if busy else "normal")) - except Exception: - pass - - def _reprocess(self): - """Ejecuta reproceso en hilo de fondo y actualiza schema/muestra preservando selección y alias existentes.""" - if self._on_reprocess is None: - return - self._set_busy(True) - - def worker(): - try: - new_schema, new_sample = self._on_reprocess() - def apply_updates(): - self._apply_new_schema_sample(new_schema, new_sample) - self._set_busy(False) - self.after(0, apply_updates) - except Exception as e: - def on_err(): - try: - mb.showerror("Error de reproceso", f"No se pudo reprocesar: {e}") - except Exception: - pass - self._set_busy(False) - self.after(0, on_err) - - threading.Thread(target=worker, daemon=True).start() - - def _apply_new_schema_sample(self, new_schema: Dict[str, str], new_sample: dict) -> None: - """Reconstruye la lista de campos preservando selección y alias cuando sea posible.""" - # Guardar selección y alias actuales - prev_selected = {p for p, v in self._vars.items() if v.get()} - prev_alias: Dict[str, str] = {} - for p, ent in self._alias_entries.items(): - try: - val = (ent.get().strip() if ent is not None else "") or p - except Exception: - val = p - if p == 'radicado': - val = 'radicado' - prev_alias[p] = val - - # Limpiar filas actuales - try: - for _, frame in self._rows.items(): - frame.destroy() - except Exception: - pass - self._vars.clear() - self._rows.clear() - self._alias_entries.clear() - - # Asignar nuevos datos - self._schema = new_schema or {} - self._sample = new_sample - - rows_container = self.scroll - for path in sorted(self._schema.keys()): - var = ctk.BooleanVar(value=(path in prev_selected)) - row = ctk.CTkFrame(rows_container) - row.pack(anchor="w", fill="x", padx=8, pady=2) - cb = ctk.CTkCheckBox(row, text=f"{path} ({self._schema[path]})", variable=var) - cb.grid(row=0, column=0, sticky="w") - alias_entry = ctk.CTkEntry(row, width=260, placeholder_text="Nombre en CSV") - # Establecer alias previo si existe, si no por defecto al path - alias_value = prev_alias.get(path, path) - try: - alias_entry.insert(0, alias_value) - except Exception: - pass - alias_entry.grid(row=0, column=1, padx=(12, 0)) - self._add_document_type_hint(row, path) - try: - alias_entry.bind("", lambda _e: self._validate_aliases()) - except Exception: - pass - def _toggle(ent: ctk.CTkEntry, v: ctk.BooleanVar = var, p: str = path): - try: - if p == 'radicado': - ent.grid_remove() - else: - if v.get(): - ent.grid(row=0, column=1, padx=(12, 0)) - else: - ent.grid_remove() - except Exception: - pass - self._validate_aliases() - _toggle(alias_entry) - cb.configure(command=lambda e=alias_entry, v=var, p=path: _toggle(e, v, p)) - self._vars[path] = var - self._rows[path] = row - self._alias_entries[path] = alias_entry - - # Revalidar y re-aplicar filtro de búsqueda actual - self._validate_aliases() - try: - self._on_search() - except Exception: - pass - - def _add_document_type_hint(self, row: ctk.CTkFrame, path: str) -> None: - """Muestra una ayuda breve bajo claves tipo CC, TI, CE, PAS, etc.""" - key_tail = str(path or "").split("/")[-1].strip().upper() - valid_hint_keys = set(get_allowed_document_types()) | DOCUMENT_TYPE_HINT_ALIASES - if key_tail not in valid_hint_keys: - return - - hint = ctk.CTkLabel( - row, - text="Si a este campo le pones el alias 'documento_forzate', se guardará con la sigla exacta detectada, por ejemplo 'CC 123456789' o 'EPT 123456789'.", - anchor="w", - justify="left", - ) - hint.grid(row=1, column=0, columnspan=2, sticky="w", padx=(28, 0), pady=(2, 0)) - diff --git a/gui/styles.py b/gui/styles.py deleted file mode 100644 index 4bcf3e5..0000000 --- a/gui/styles.py +++ /dev/null @@ -1,58 +0,0 @@ -"""Configuración centralizada de estilos, temas y fuentes para la GUI.""" -import customtkinter as ctk - - -class AppStyles: - """Clase con configuración de estilos y fuentes de la aplicación.""" - - @staticmethod - def configure_theme(): - """Configura el tema y apariencia general de la aplicación.""" - ctk.set_appearance_mode("System") - ctk.set_default_color_theme("blue") - - @staticmethod - def get_fonts(): - """Devuelve un diccionario con las fuentes personalizadas.""" - return { - "title": ctk.CTkFont(family="Roboto", size=22, weight="bold"), - "section": ctk.CTkFont(family="Roboto", size=16, weight="bold"), - "ui": ctk.CTkFont(family="Roboto", size=13), - "mono": ctk.CTkFont(family="Consolas", size=12), - "mono_small": ctk.CTkFont(family="Consolas", size=10), - "italic_small": ctk.CTkFont(size=12, slant="italic"), - "bold_large": ctk.CTkFont(size=16, weight="bold"), - "button_large": ctk.CTkFont(size=14, weight="bold"), - } - - @staticmethod - def get_colors(): - """Devuelve un diccionario con los colores personalizados.""" - return { - "primary_button": ("#106BA3", "#106BA3"), - "primary_hover": ("#0D5582", "#0D5582"), - "secondary_button": ("#2E86C1", "#2E86C1"), - "secondary_hover": ("#21618C", "#21618C"), - "success": ("#27AE60", "#27AE60"), - "success_hover": ("#1E8449", "#1E8449"), - "warning": ("#D35400", "#D35400"), - "warning_hover": ("#A04000", "#A04000"), - "danger": ("#C0392B", "#C0392B"), - "danger_hover": ("#A93226", "#A93226"), - "gray_light": "gray60", - "gray_dark": "gray70", - "text_muted": ("gray50", "gray70"), - } - - @staticmethod - def get_spacing(): - """Devuelve un diccionario con espaciados y padding estándar.""" - return { - "padding_small": 2, - "padding_medium": 5, - "padding_large": 10, - "padding_section": 15, - "margin_small": (0, 2), - "margin_medium": (5, 5), - "margin_large": (10, 15), - } diff --git a/gui/tabs/__init__.py b/gui/tabs/__init__.py deleted file mode 100644 index 9c80cab..0000000 --- a/gui/tabs/__init__.py +++ /dev/null @@ -1,18 +0,0 @@ -"""Módulo de pestañas de la GUI.""" -from gui.tabs.settings_tab import SettingsTab -from gui.tabs.results_tab import ResultsTab -from gui.tabs.main_tab import MainProcessingTab -from gui.tabs.preclass_tab import PreclassificationTab -from gui.tabs.metrics_tab import MetricsTab -from gui.tabs.postprocess_tab import PostProcessTab -from gui.tabs.cleaning_tab import CleaningTab - -__all__ = [ - "SettingsTab", - "ResultsTab", - "MainProcessingTab", - "PreclassificationTab", - "MetricsTab", - "PostProcessTab", - "CleaningTab", -] diff --git a/gui/tabs/cleaning_tab.py b/gui/tabs/cleaning_tab.py deleted file mode 100644 index c7e2684..0000000 --- a/gui/tabs/cleaning_tab.py +++ /dev/null @@ -1,1336 +0,0 @@ -"""Pestaña de Proceso de Limpieza de CSV. - -Permite al usuario cargar un CSV, ver un resumen inteligente por columna y -aplicar reglas de limpieza seleccionables antes de exportar el resultado. -""" -from __future__ import annotations - -from dataclasses import dataclass -from tkinter import ttk -from typing import Optional - -import customtkinter as ctk -import pandas as pd - -from application.csv_cleaning import ( - CLEANING_RULES, - ColumnSummary, - applicable_rule_ids_for_type, -) -from application.app_prefs import load_cop_mod_preferences -from gui.styles import AppStyles - - -_PREVIEW_MAX_ROWS = 50 -_PREVIEW_MAX_CELL_LEN = 80 - - -_TYPE_LABELS = { - "integer": "Entero", - "decimal": "Decimal", - "date": "Fecha", - "boolean": "Booleano", - "categorical": "Categórico", - "text": "Texto", - "mixed": "Mixto", - "empty": "Vacío", -} - -# Color por tipo (badge). Tuplas (light, dark) compatibles con CTk. -_TYPE_COLORS: dict[str, tuple[str, str]] = { - "integer": ("#1F6FEB", "#1F6FEB"), - "decimal": ("#0EA5E9", "#0EA5E9"), - "date": ("#D97706", "#D97706"), - "boolean": ("#7C3AED", "#7C3AED"), - "categorical": ("#16A34A", "#16A34A"), - "text": ("#475569", "#64748B"), - "mixed": ("#DC2626", "#DC2626"), - "empty": ("#6B7280", "#6B7280"), -} - -# Orden fijo en el desplegable de anulación manual -_USER_EDITABLE_TYPE_IDS: tuple[str, ...] = ( - "integer", "decimal", "date", "boolean", "categorical", "text", "mixed", "empty", -) -_TYPE_MENU_VALUES: list[str] = [ - _TYPE_LABELS.get(t, t) for t in _USER_EDITABLE_TYPE_IDS -] - -_RULE_SHORT_LABELS: dict[str, str] = { - "strip_special_spaces": "Sin especiales/espacios (no antes de fecha+hora)", - "strip_special_keep_date_seps": "Símbolos: conserva /, -, :, espacios", - "strip_spaces": "Sin espacios", - "to_integer": "A entero", - "date_to_ddmmyyyy": "Normalizar fecha → DD/MM/AAAA", - "date_numeric_to_ddmmyyyy": "Fecha numérica → DD/MM/AAAA", - "date_alpha_to_ddmmyyyy": "Fecha alfanumérica → DD/MM/AAAA", - "date_messy_text_to_ddmmyyyy": "Fecha con ruido (texto) → DD/MM/AAAA", - "round_decimals_custom": "Aprox. decimales (>0.5 sube)", - "doc_id_split_tipo_numero": "Dividir doc (tipo + cifras)", - "doc_type_homologate": "Homologar tipo doc", - "cop_mod_split": "Separa COP/MOD", - "nota_paciente_autorizacion": "Extraer doc. y autorización (notas)", -} - - -def _fmt_number(x) -> str: - """Formatea un número para mostrar en chip (entero limpio o 2 decimales).""" - if x is None: - return "—" - try: - f = float(x) - except (TypeError, ValueError): - return str(x) - if f.is_integer(): - return f"{int(f):,}" - return f"{f:,.2f}" - - -@dataclass -class _ColumnUI: - """Referencias a los widgets de una tarjeta de columna.""" - card: ctk.CTkFrame - body: ctk.CTkFrame - toggle_btn: ctk.CTkButton - type_menu: ctk.CTkOptionMenu - count_label: ctk.CTkLabel - stats_frame: ctk.CTkFrame - rules_frame: ctk.CTkFrame - rule_vars: list[tuple[str, ctk.CTkCheckBox, ctk.BooleanVar]] - summary: ColumnSummary - expanded: bool = True - cop_mod_fecha_frame: Optional[ctk.CTkFrame] = None - cop_mod_fecha_menu: Optional[ctk.CTkOptionMenu] = None - cop_mod_fecha_var: Optional[ctk.StringVar] = None - - -class CleaningTab: - """UI de la pestaña Proceso de Limpieza.""" - - def __init__(self, parent_tab): - self.tab = parent_tab - self.fonts = AppStyles.get_fonts() - self.colors = AppStyles.get_colors() - self.spacing = AppStyles.get_spacing() - - self._column_uis: dict[str, _ColumnUI] = {} - self._search_var: Optional[ctk.StringVar] = None - self._csv_column_names: list[str] = [] - # Tipo de dato forzado por el usuario (id interno) por columna; vacío = inferido - self._type_overrides: dict[str, str] = {} - self._syncing_rule_vars = False - - self._setup_layout() - self._create_file_section() - self._create_preview_section() - self._create_bulk_actions_section() - self._create_columns_section() - self._create_action_section() - self._create_result_section() - - # -- Layout ------------------------------------------------------------------ - - def _setup_layout(self): - self.tab.grid_columnconfigure(0, weight=1) - self.tab.grid_rowconfigure(0, weight=1) - - self.scroll = ctk.CTkScrollableFrame(self.tab, fg_color="transparent") - self.scroll.grid(row=0, column=0, sticky="nsew", padx=10, pady=10) - self.scroll.grid_columnconfigure(0, weight=1) - - # -- Sección archivo --------------------------------------------------------- - - def _create_file_section(self): - frame = ctk.CTkFrame(self.scroll, corner_radius=8) - frame.grid(row=0, column=0, sticky="ew", padx=5, pady=(5, 4)) - frame.grid_columnconfigure(2, weight=1) - - lbl_title = ctk.CTkLabel( - frame, text="Archivo CSV", font=self.fonts["section"], - ) - lbl_title.grid(row=0, column=0, columnspan=4, sticky="w", padx=15, pady=(10, 4)) - - self.btn_select_csv = ctk.CTkButton( - frame, text="📂 Seleccionar CSV", font=self.fonts["ui"], width=170, - fg_color=self.colors["primary_button"], hover_color=self.colors["primary_hover"], - ) - self.btn_select_csv.grid(row=1, column=0, padx=(15, 8), pady=(0, 12), sticky="w") - - self.lbl_csv_path = ctk.CTkLabel( - frame, text="Ningún archivo seleccionado", - font=self.fonts["italic_small"], text_color=self.colors["gray_light"], - anchor="w", - ) - self.lbl_csv_path.grid(row=1, column=1, columnspan=2, padx=4, pady=(0, 12), sticky="ew") - - # Chips informativos: filas, columnas, delimitador - chips_frame = ctk.CTkFrame(frame, fg_color="transparent") - chips_frame.grid(row=1, column=3, padx=(8, 15), pady=(0, 12), sticky="e") - self.chip_rows = self._make_chip(chips_frame, "Filas: —") - self.chip_rows.pack(side="left", padx=(0, 6)) - self.chip_cols = self._make_chip(chips_frame, "Columnas: —") - self.chip_cols.pack(side="left", padx=(0, 6)) - self.chip_delim = self._make_chip(chips_frame, "Delim: —") - self.chip_delim.pack(side="left") - - self.loading_frame = ctk.CTkFrame(frame, fg_color="transparent") - self.loading_frame.grid( - row=2, column=0, columnspan=4, sticky="ew", padx=15, pady=(0, 12) - ) - self.loading_frame.grid_columnconfigure(1, weight=1) - self.loading_label = ctk.CTkLabel( - self.loading_frame, - text="", - font=self.fonts["italic_small"], - text_color=self.colors["gray_light"], - anchor="w", - ) - self.loading_label.grid(row=0, column=0, sticky="w", padx=(0, 10)) - self.loading_progress = ctk.CTkProgressBar( - self.loading_frame, - mode="indeterminate", - height=8, - ) - self.loading_progress.grid(row=0, column=1, sticky="ew") - self.loading_frame.grid_remove() - - def _make_chip(self, parent, text: str) -> ctk.CTkLabel: - return ctk.CTkLabel( - parent, text=text, font=self.fonts["italic_small"], - fg_color=("gray85", "gray25"), corner_radius=12, - padx=10, pady=2, - ) - - # -- Vista previa ------------------------------------------------------------ - - def _create_preview_section(self): - frame = ctk.CTkFrame(self.scroll, corner_radius=8) - frame.grid(row=1, column=0, sticky="ew", padx=5, pady=4) - frame.grid_columnconfigure(0, weight=1) - - header = ctk.CTkFrame(frame, fg_color="transparent") - header.grid(row=0, column=0, sticky="ew", padx=15, pady=(10, 4)) - header.grid_columnconfigure(0, weight=1) - - lbl = ctk.CTkLabel(header, text="Vista previa", font=self.fonts["section"]) - lbl.pack(side="left") - - self.lbl_preview_info = ctk.CTkLabel( - header, text="(sin datos)", font=self.fonts["italic_small"], - text_color=self.colors["gray_light"], - ) - self.lbl_preview_info.pack(side="left", padx=(12, 0)) - - self.btn_toggle_preview = ctk.CTkButton( - header, text="Ocultar", width=80, height=24, - font=self.fonts["italic_small"], - fg_color="transparent", border_width=1, - command=self._on_toggle_preview, - ) - self.btn_toggle_preview.pack(side="right") - - self.preview_table_holder = ctk.CTkFrame( - frame, fg_color=("gray92", "gray18"), corner_radius=6, height=220, - ) - self.preview_table_holder.grid(row=1, column=0, sticky="ew", padx=15, pady=(0, 10)) - self.preview_table_holder.grid_propagate(False) - self.preview_table_holder.grid_columnconfigure(0, weight=1) - self.preview_table_holder.grid_rowconfigure(0, weight=1) - - self._preview_tree: Optional[ttk.Treeview] = None - self._preview_visible = True - self._render_table_placeholder( - self.preview_table_holder, "Selecciona un CSV para ver la vista previa.", - ) - - def _on_toggle_preview(self): - if self._preview_visible: - self.preview_table_holder.grid_remove() - self.btn_toggle_preview.configure(text="Mostrar") - else: - self.preview_table_holder.grid() - self.btn_toggle_preview.configure(text="Ocultar") - self._preview_visible = not self._preview_visible - - # -- Acciones masivas -------------------------------------------------------- - - def _create_bulk_actions_section(self): - frame = ctk.CTkFrame(self.scroll, corner_radius=8) - frame.grid(row=2, column=0, sticky="ew", padx=5, pady=4) - frame.grid_columnconfigure(0, weight=1) - - header = ctk.CTkFrame(frame, fg_color="transparent") - header.grid(row=0, column=0, sticky="ew", padx=15, pady=(10, 4)) - header.grid_columnconfigure(0, weight=1) - - lbl = ctk.CTkLabel( - header, text="Acciones rápidas", font=self.fonts["section"], - ) - lbl.pack(side="left") - - body = ctk.CTkFrame(frame, fg_color="transparent") - body.grid(row=1, column=0, sticky="ew", padx=15, pady=(0, 10)) - body.grid_columnconfigure(2, weight=1) - - # Buscador - lbl_search = ctk.CTkLabel(body, text="🔍 Buscar columna:", font=self.fonts["ui"]) - lbl_search.grid(row=0, column=0, padx=(0, 6), pady=4, sticky="w") - self._search_var = ctk.StringVar(value="") - self._search_var.trace_add("write", lambda *_: self._apply_search_filter()) - self.entry_search = ctk.CTkEntry( - body, textvariable=self._search_var, width=240, - placeholder_text="nombre de columna...", - ) - self.entry_search.grid(row=0, column=1, padx=(0, 12), pady=4, sticky="w") - - # Botones colapsar/expandir todas - btns_right = ctk.CTkFrame(body, fg_color="transparent") - btns_right.grid(row=0, column=2, sticky="e") - - self.btn_expand_all = ctk.CTkButton( - btns_right, text="Expandir todas", width=130, height=28, - font=self.fonts["ui"], fg_color="transparent", border_width=1, - command=lambda: self._set_all_expanded(True), - ) - self.btn_expand_all.pack(side="left", padx=4) - self.btn_collapse_all = ctk.CTkButton( - btns_right, text="Colapsar todas", width=130, height=28, - font=self.fonts["ui"], fg_color="transparent", border_width=1, - command=lambda: self._set_all_expanded(False), - ) - self.btn_collapse_all.pack(side="left", padx=4) - self.btn_clear_rules = ctk.CTkButton( - btns_right, text="Limpiar selección", width=140, height=28, - font=self.fonts["ui"], - fg_color=self.colors["danger"], hover_color=self.colors["danger_hover"], - command=self._clear_all_rules, - ) - self.btn_clear_rules.pack(side="left", padx=4) - - # Aplicar regla a todas las de un tipo - bulk_row = ctk.CTkFrame(body, fg_color="transparent") - bulk_row.grid(row=1, column=0, columnspan=3, sticky="ew", pady=(8, 0)) - - lbl_bulk = ctk.CTkLabel( - bulk_row, text="Aplicar regla a todas las columnas de tipo:", - font=self.fonts["ui"], - ) - lbl_bulk.pack(side="left", padx=(0, 6)) - - type_options = list(_TYPE_LABELS.values()) - self._bulk_type_var = ctk.StringVar(value=type_options[0]) - self.bulk_type_menu = ctk.CTkOptionMenu( - bulk_row, values=type_options, variable=self._bulk_type_var, - width=130, font=self.fonts["ui"], - ) - self.bulk_type_menu.pack(side="left", padx=(0, 6)) - - self._bulk_rule_var = ctk.StringVar(value=CLEANING_RULES[0]["name"]) - self.bulk_rule_menu = ctk.CTkOptionMenu( - bulk_row, values=[r["name"] for r in CLEANING_RULES], variable=self._bulk_rule_var, - width=380, font=self.fonts["ui"], - ) - self.bulk_rule_menu.pack(side="left", padx=(0, 6)) - self._sync_bulk_rule_menu() - self._bulk_type_var.trace_add("write", lambda *_: self._sync_bulk_rule_menu()) - - self.btn_bulk_apply = ctk.CTkButton( - bulk_row, text="Aplicar a todas", width=140, height=28, - font=self.fonts["ui"], - fg_color=self.colors["secondary_button"], - hover_color=self.colors["secondary_hover"], - command=self._on_bulk_apply, - ) - self.btn_bulk_apply.pack(side="left", padx=4) - - # -- Columnas y reglas ------------------------------------------------------- - - def _create_columns_section(self): - frame = ctk.CTkFrame(self.scroll, corner_radius=8) - frame.grid(row=3, column=0, sticky="ew", padx=5, pady=4) - frame.grid_columnconfigure(0, weight=1) - - header = ctk.CTkFrame(frame, fg_color="transparent") - header.grid(row=0, column=0, sticky="ew", padx=15, pady=(10, 4)) - header.grid_columnconfigure(0, weight=1) - - lbl = ctk.CTkLabel( - header, text="Resumen y reglas por columna", font=self.fonts["section"], - ) - lbl.pack(side="left") - - self.lbl_global_count = ctk.CTkLabel( - header, text="0 reglas seleccionadas", - font=self.fonts["italic_small"], text_color=self.colors["gray_light"], - ) - self.lbl_global_count.pack(side="right") - - # Frame plano: el scroll único es el de `self.scroll` (evita doble rueda del ratón) - self.columns_container = ctk.CTkFrame( - frame, fg_color="transparent", - ) - self.columns_container.grid(row=1, column=0, sticky="ew", padx=10, pady=(0, 10)) - self.columns_container.grid_columnconfigure(0, weight=1) - - self._empty_label = ctk.CTkLabel( - self.columns_container, - text="Selecciona un CSV para ver el resumen de columnas.", - font=self.fonts["italic_small"], text_color=self.colors["gray_light"], - ) - self._empty_label.grid(row=0, column=0, sticky="w", padx=10, pady=20) - - # -- Acción ------------------------------------------------------------------ - - def _create_action_section(self): - frame = ctk.CTkFrame(self.scroll, corner_radius=8) - frame.grid(row=4, column=0, sticky="ew", padx=5, pady=4) - frame.grid_columnconfigure(1, weight=1) - - self.btn_apply = ctk.CTkButton( - frame, text="✨ Aplicar limpieza", font=self.fonts["button_large"], - fg_color=self.colors["primary_button"], hover_color=self.colors["primary_hover"], - height=42, width=220, - ) - self.btn_apply.grid(row=0, column=0, padx=15, pady=12, sticky="w") - - self.lbl_status = ctk.CTkLabel( - frame, text="", font=self.fonts["italic_small"], - text_color=self.colors["gray_light"], anchor="w", - ) - self.lbl_status.grid(row=0, column=1, padx=10, pady=12, sticky="ew") - - # -- Resultado --------------------------------------------------------------- - - def _create_result_section(self): - frame = ctk.CTkFrame(self.scroll, corner_radius=8) - frame.grid(row=5, column=0, sticky="ew", padx=5, pady=(4, 6)) - frame.grid_columnconfigure(0, weight=1) - - top = ctk.CTkFrame(frame, fg_color="transparent") - top.grid(row=0, column=0, sticky="ew", padx=15, pady=(10, 4)) - top.grid_columnconfigure(0, weight=1) - - lbl = ctk.CTkLabel( - top, text="Vista previa del resultado", font=self.fonts["section"], - ) - lbl.pack(side="left") - - self.lbl_result_info = ctk.CTkLabel( - top, text="", font=self.fonts["italic_small"], - text_color=self.colors["gray_light"], - ) - self.lbl_result_info.pack(side="left", padx=(12, 0)) - - self.btn_save = ctk.CTkButton( - top, text="💾 Guardar CSV", font=self.fonts["ui"], width=160, height=32, - fg_color=self.colors["success"], hover_color=self.colors["success_hover"], - state="disabled", - ) - self.btn_save.pack(side="right") - - self.result_table_holder = ctk.CTkFrame( - frame, fg_color=("gray92", "gray18"), corner_radius=6, height=260, - ) - self.result_table_holder.grid(row=1, column=0, sticky="ew", padx=15, pady=(0, 10)) - self.result_table_holder.grid_propagate(False) - self.result_table_holder.grid_columnconfigure(0, weight=1) - self.result_table_holder.grid_rowconfigure(0, weight=1) - - self._result_tree: Optional[ttk.Treeview] = None - self._render_table_placeholder( - self.result_table_holder, "Aplica reglas para ver el resultado.", - ) - - # =========================================================================== - # API pública - # =========================================================================== - - def set_on_select_csv(self, callback): - self.btn_select_csv.configure(command=callback) - - def set_on_apply(self, callback): - self.btn_apply.configure(command=callback) - - def set_on_save(self, callback): - self.btn_save.configure(command=callback) - - def set_csv_path(self, path: str): - self.lbl_csv_path.configure(text=path or "Ningún archivo seleccionado") - - def set_status(self, text: str): - self.lbl_status.configure(text=text) - - def set_loading(self, loading: bool, text: str = ""): - """Muestra u oculta una barra indeterminada para operaciones de CSV.""" - if loading: - self.loading_label.configure(text=text or "Procesando CSV...") - self.loading_frame.grid() - try: - self.loading_progress.start() - except Exception: - pass - else: - try: - self.loading_progress.stop() - except Exception: - pass - self.loading_label.configure(text="") - self.loading_frame.grid_remove() - - def set_preview(self, df: Optional[pd.DataFrame]): - """Renderiza el DataFrame original en la tabla de vista previa.""" - if df is None or len(df) == 0: - self._csv_column_names = [] - self._render_table_placeholder( - self.preview_table_holder, "(sin datos)", - ) - self.lbl_preview_info.configure(text="(sin datos)") - self._preview_tree = None - return - info = ( - f"{len(df):,} filas · {len(df.columns)} columnas · " - f"mostrando primeras {min(_PREVIEW_MAX_ROWS, len(df))}" - ) - self.lbl_preview_info.configure(text=info) - self._preview_tree = self._render_dataframe_table( - self.preview_table_holder, df, - ) - self.chip_rows.configure(text=f"Filas: {len(df):,}") - self.chip_cols.configure(text=f"Columnas: {len(df.columns)}") - self._csv_column_names = list(df.columns) - - def set_result(self, df: Optional[pd.DataFrame], info_text: str = ""): - """Renderiza el DataFrame resultante en la tabla de resultado.""" - if df is None or len(df) == 0: - self._render_table_placeholder( - self.result_table_holder, "(sin datos)", - ) - self.lbl_result_info.configure(text=info_text) - self._result_tree = None - return - summary = ( - f"{len(df):,} filas · {len(df.columns)} columnas · " - f"mostrando primeras {min(_PREVIEW_MAX_ROWS, len(df))}" - ) - if info_text: - summary = f"{summary} · {info_text}" - self.lbl_result_info.configure(text=summary) - self._result_tree = self._render_dataframe_table( - self.result_table_holder, df, - ) - - def enable_save(self, enabled: bool): - self.btn_save.configure(state="normal" if enabled else "disabled") - - def enable_controls(self, enabled: bool): - state = "normal" if enabled else "disabled" - self.btn_select_csv.configure(state=state) - self.btn_apply.configure(state=state) - self.btn_bulk_apply.configure(state=state) - self.btn_clear_rules.configure(state=state) - self.btn_expand_all.configure(state=state) - self.btn_collapse_all.configure(state=state) - - def _sync_bulk_rule_menu(self) -> None: - """Acota las reglas del desplegable según el tipo (p. ej. entero: sin a entero/approx).""" - if not hasattr(self, "bulk_rule_menu") or not hasattr( - self, "_bulk_type_var", - ): - return - type_label = self._bulk_type_var.get() - type_id = next( - (k for k, v in _TYPE_LABELS.items() if v == type_label), "text" - ) - want = set(applicable_rule_ids_for_type(type_id)) - names = [r["name"] for r in CLEANING_RULES if r["id"] in want] - if not names: - return - cur = self._bulk_rule_var.get() - self.bulk_rule_menu.configure(values=names) - if cur in names: - self._bulk_rule_var.set(cur) - else: - self._bulk_rule_var.set(names[0]) - - def populate_columns(self, summaries: list[ColumnSummary]): - """Renderiza la lista de columnas con sus reglas aplicables.""" - for child in list(self.columns_container.winfo_children()): - child.destroy() - self._column_uis.clear() - self._type_overrides.clear() - if self._search_var is not None: - try: - self._search_var.set("") - except Exception: - pass - - if not summaries: - empty = ctk.CTkLabel( - self.columns_container, - text="El CSV no tiene columnas.", - font=self.fonts["italic_small"], text_color=self.colors["gray_light"], - ) - empty.grid(row=0, column=0, sticky="w", padx=10, pady=20) - self._update_global_count() - self.chip_cols.configure(text=f"Columnas: 0") - return - - rules_by_id = {r["id"]: r for r in CLEANING_RULES} - - for row_idx, summary in enumerate(summaries): - ui = self._build_column_card(row_idx, summary, rules_by_id) - self._column_uis[summary.name] = ui - - self.chip_cols.configure(text=f"Columnas: {len(summaries)}") - self._update_global_count() - self._refresh_cop_mod_menus() - - def get_cop_mod_fecha_columns(self) -> dict[str, str]: - """Mapa columna de importe -> columna Fecha Servicio (solo si COP/MOD está marcado).""" - out: dict[str, str] = {} - for col_name, ui in self._column_uis.items(): - if ui.cop_mod_fecha_var is None: - continue - cop_on = any( - rid == "cop_mod_split" and bool(var.get()) - for rid, _chk, var in ui.rule_vars - ) - if not cop_on: - continue - sel = str(ui.cop_mod_fecha_var.get() or "").strip() - if sel: - out[col_name] = sel - return out - - def _date_columns_for_cop_mod_menu(self) -> list[str]: - cols = self._csv_column_names or [] - return cols if cols else [""] - - def _suggest_fecha_column_option(self) -> str: - cols = self._csv_column_names or [] - if not cols: - return "" - _, hint = load_cop_mod_preferences() - if hint and hint in cols: - return hint - for c in cols: - low = c.lower().replace("_", " ") - if "fecha" in low and "servicio" in low: - return c - return cols[0] - - def _refresh_cop_mod_menus(self) -> None: - cols = self._date_columns_for_cop_mod_menu() - suggest = self._suggest_fecha_column_option() - for ui in self._column_uis.values(): - if ui.cop_mod_fecha_menu is None or ui.cop_mod_fecha_var is None: - continue - cur = ui.cop_mod_fecha_var.get() - ui.cop_mod_fecha_menu.configure(values=cols) - if cur and cur in cols: - ui.cop_mod_fecha_var.set(cur) - elif suggest and suggest in cols: - ui.cop_mod_fecha_var.set(suggest) - elif cols and cols[0]: - ui.cop_mod_fecha_var.set(cols[0]) - - def get_selected_rules(self) -> dict[str, list[str]]: - """Devuelve un mapa columna -> lista de rule_ids en el orden visual.""" - result: dict[str, list[str]] = {} - for col, ui in self._column_uis.items(): - selected = [rid for rid, _chk, var in ui.rule_vars if bool(var.get())] - if selected: - result[col] = selected - return result - - # =========================================================================== - # Construcción de tarjeta de columna - # =========================================================================== - - def _build_column_card( - self, row_idx: int, summary: ColumnSummary, rules_by_id: dict, - ) -> _ColumnUI: - card = ctk.CTkFrame(self.columns_container, corner_radius=8, border_width=1) - card.grid(row=row_idx, column=0, sticky="ew", padx=2, pady=4) - card.grid_columnconfigure(0, weight=1) - - # ---- Header ---- - header = ctk.CTkFrame(card, fg_color="transparent") - header.grid(row=0, column=0, sticky="ew", padx=10, pady=(8, 4)) - header.grid_columnconfigure(2, weight=1) - - toggle_btn = ctk.CTkButton( - header, text="▼", width=28, height=28, - font=self.fonts["ui"], fg_color="transparent", - border_width=0, hover=True, - ) - toggle_btn.grid(row=0, column=0, padx=(0, 6), sticky="w") - - eff0 = self._type_overrides.get(summary.name) or summary.inferred_type - if eff0 not in _TYPE_LABELS: - eff0 = "text" - tcol = _TYPE_COLORS.get(eff0, ("#64748B", "#64748B")) - tmenu = ctk.CTkOptionMenu( - header, - values=_TYPE_MENU_VALUES, - width=150, - height=28, - font=self.fonts["ui"], - corner_radius=10, - button_color=tcol[0] if tcol[0] else tcol[1], - button_hover_color=tcol[0], - fg_color=tcol[0] if tcol[0] else tcol[1], - ) - tmenu.set(_TYPE_LABELS.get(eff0, _TYPE_LABELS["text"])) - tmenu.grid(row=0, column=1, padx=(0, 6), sticky="w") - tmenu.configure( - command=lambda v, c=summary.name, m=tmenu: self._on_user_column_type_change( - c, m, v - ) - ) - - name_lbl = ctk.CTkLabel( - header, text=summary.name, font=self.fonts["bold_large"], anchor="w", - ) - name_lbl.grid(row=0, column=2, sticky="w") - - count_lbl = ctk.CTkLabel( - header, text="0 reglas", - font=self.fonts["italic_small"], text_color=self.colors["gray_light"], - ) - count_lbl.grid(row=0, column=3, padx=(8, 0), sticky="e") - - # ---- Body (info + reglas) ---- - body = ctk.CTkFrame(card, fg_color="transparent") - body.grid(row=1, column=0, sticky="ew", padx=10, pady=(0, 10)) - body.grid_columnconfigure(0, weight=1) - - example = summary.example or "(sin datos)" - if len(example) > 100: - example = example[:97] + "..." - lbl_example = ctk.CTkLabel( - body, text=f'Ejemplo: "{example}"', - font=self.fonts["mono_small"], anchor="w", - text_color=self.colors["gray_dark"], justify="left", - ) - lbl_example.grid(row=0, column=0, sticky="ew", padx=4, pady=(0, 4)) - - stats_frame = self._build_stats_chips(body, summary, eff0) - stats_frame.grid( - row=1, column=0, sticky="ew", padx=4, pady=(0, 4), - ) - - if summary.top_values: - self._build_top_values_panel(body, summary.top_values).grid( - row=2, column=0, sticky="ew", padx=4, pady=(0, 6), - ) - - rules_frame = ctk.CTkFrame(body, fg_color=("gray92", "gray18"), corner_radius=6) - rules_frame.grid(row=3, column=0, sticky="ew", padx=0, pady=(2, 0)) - rules_frame.grid_columnconfigure((0, 1), weight=1, uniform="rules") - - eff_rules = self._type_overrides.get(summary.name) or summary.inferred_type - if eff_rules not in _TYPE_LABELS: - eff_rules = "text" - applic = set(applicable_rule_ids_for_type(eff_rules)) - ordered_ids = [r["id"] for r in CLEANING_RULES if r["id"] in applic] - - vars_list: list[tuple[str, ctk.CTkCheckBox, ctk.BooleanVar]] = [] - col_in_row = 0 - row_in_rules = 0 - for rid in ordered_ids: - rule = rules_by_id.get(rid) - if rule is None: - continue - var = ctk.BooleanVar(value=False) - var.trace_add( - "write", - lambda *_a, c=summary.name, r=rid: self._on_rule_toggle(c, r), - ) - label = _RULE_SHORT_LABELS.get(rid, rule["name"]) - chk = ctk.CTkCheckBox( - rules_frame, text=label, variable=var, - font=self.fonts["ui"], checkbox_width=18, checkbox_height=18, - ) - chk.grid(row=row_in_rules, column=col_in_row, sticky="w", padx=10, pady=4) - vars_list.append((rid, chk, var)) - col_in_row += 1 - if col_in_row >= 2: - col_in_row = 0 - row_in_rules += 1 - - fec_frame = None - fec_menu = None - fec_var = None - if "cop_mod_split" in ordered_ids: - fec_frame = ctk.CTkFrame(body, fg_color="transparent") - fec_frame.grid(row=4, column=0, sticky="ew", padx=4, pady=(4, 0)) - fec_frame.grid_columnconfigure(1, weight=1) - ctk.CTkLabel( - fec_frame, - text="Fecha Servicio (columna):", - font=self.fonts["italic_small"], - text_color=self.colors["gray_dark"], - ).grid(row=0, column=0, sticky="w", padx=(0, 8)) - fec_var = ctk.StringVar(value=self._suggest_fecha_column_option()) - fec_menu = ctk.CTkOptionMenu( - fec_frame, - values=self._date_columns_for_cop_mod_menu(), - variable=fec_var, - font=self.fonts["ui"], - width=260, - ) - fec_menu.grid(row=0, column=1, sticky="ew") - fec_menu.configure(state="disabled") - - ui = _ColumnUI( - card=card, - body=body, - toggle_btn=toggle_btn, - type_menu=tmenu, - count_label=count_lbl, - stats_frame=stats_frame, - rules_frame=rules_frame, - rule_vars=vars_list, - summary=summary, - cop_mod_fecha_frame=fec_frame, - cop_mod_fecha_menu=fec_menu, - cop_mod_fecha_var=fec_var, - ) - toggle_btn.configure(command=lambda c=summary.name: self._toggle_column(c)) - return ui - - # =========================================================================== - # Stats visuales por columna - # =========================================================================== - - def _build_stats_chips( - self, parent, summary: ColumnSummary, display_type: Optional[str] = None - ) -> ctk.CTkFrame: - """Construye una fila de chips con la información clave de la columna.""" - row = ctk.CTkFrame(parent, fg_color="transparent") - dtype = (display_type or summary.inferred_type) or "text" - if dtype not in _TYPE_LABELS: - dtype = "text" - - # Sin datos - nulls_color = ( - ("#FEE2E2", "#7F1D1D") if summary.null_count > 0 - else ("gray88", "gray22") - ) - nulls_text_color = ( - ("#991B1B", "#FECACA") if summary.null_count > 0 - else ("gray40", "gray70") - ) - self._make_stat_chip( - row, - label="Sin datos", - value=f"{summary.null_count} ({summary.null_pct:.0f}%)", - bg=nulls_color, - text_color=nulls_text_color, - ).pack(side="left", padx=(0, 6), pady=2) - - # Únicos - self._make_stat_chip( - row, - label="Únicos", - value=f"{summary.unique_count}", - bg=("#DBEAFE", "#1E3A8A"), - text_color=("#1E40AF", "#BFDBFE"), - ).pack(side="left", padx=(0, 6), pady=2) - - # Stats por tipo - if dtype in ("integer", "decimal"): - if summary.min is not None: - self._make_stat_chip( - row, label="Mín", value=_fmt_number(summary.min), - bg=("#DCFCE7", "#14532D"), text_color=("#166534", "#BBF7D0"), - ).pack(side="left", padx=(0, 6), pady=2) - if summary.max is not None: - self._make_stat_chip( - row, label="Máx", value=_fmt_number(summary.max), - bg=("#DCFCE7", "#14532D"), text_color=("#166534", "#BBF7D0"), - ).pack(side="left", padx=(0, 6), pady=2) - if summary.mean is not None: - self._make_stat_chip( - row, label="Promedio", value=_fmt_number(summary.mean), - bg=("#DCFCE7", "#14532D"), text_color=("#166534", "#BBF7D0"), - ).pack(side="left", padx=(0, 6), pady=2) - elif dtype == "date": - if summary.detected_date_format: - self._make_stat_chip( - row, label="Formato detectado", - value=summary.detected_date_format, - bg=("#FFEDD5", "#7C2D12"), - text_color=("#9A3412", "#FED7AA"), - ).pack(side="left", padx=(0, 6), pady=2) - elif dtype in ("text", "categorical"): - if summary.avg_length is not None: - self._make_stat_chip( - row, label="Longitud promedio", - value=f"{summary.avg_length:.0f} caracteres", - bg=("gray88", "gray22"), text_color=("gray35", "gray75"), - ).pack(side="left", padx=(0, 6), pady=2) - - return row - - def _make_stat_chip( - self, - parent, - *, - label: str, - value: str, - bg, - text_color, - ) -> ctk.CTkFrame: - """Chip con etiqueta pequeña + valor en negrita.""" - chip = ctk.CTkFrame(parent, fg_color=bg, corner_radius=10) - inner = ctk.CTkFrame(chip, fg_color="transparent") - inner.pack(padx=8, pady=2) - ctk.CTkLabel( - inner, text=label, font=ctk.CTkFont(size=10), - text_color=text_color, - ).pack(side="left", padx=(0, 4)) - ctk.CTkLabel( - inner, text=value, font=ctk.CTkFont(size=11, weight="bold"), - text_color=text_color, - ).pack(side="left") - return chip - - def _build_top_values_panel( - self, parent, top_values: list[tuple[str, int]], - ) -> ctk.CTkFrame: - """Panel con los valores más frecuentes en formato chip.""" - container = ctk.CTkFrame(parent, fg_color="transparent") - container.grid_columnconfigure(1, weight=1) - - ctk.CTkLabel( - container, text="Más frecuentes:", - font=ctk.CTkFont(size=11, weight="bold"), - text_color=self.colors["text_muted"], anchor="w", - ).grid(row=0, column=0, padx=(0, 8), sticky="nw") - - chips_row = ctk.CTkFrame(container, fg_color="transparent") - chips_row.grid(row=0, column=1, sticky="ew") - - for value, count in top_values[:5]: - display = str(value) - if len(display) > 60: - display = display[:57] + "..." - chip = ctk.CTkFrame( - chips_row, fg_color=("gray90", "gray20"), corner_radius=10, - ) - chip.pack(side="left", padx=(0, 6), pady=2) - ctk.CTkLabel( - chip, text=f'"{display}"', - font=ctk.CTkFont(size=11), - text_color=("gray25", "gray85"), - ).pack(side="left", padx=(8, 4), pady=2) - ctk.CTkLabel( - chip, text=f"x{count}", - font=ctk.CTkFont(size=10, weight="bold"), - text_color=self.colors["primary_button"], - ).pack(side="left", padx=(0, 8), pady=2) - - return container - - # =========================================================================== - # Lógica interna - # =========================================================================== - - def _effective_inferred_type(self, ui: _ColumnUI) -> str: - """Tipo mostrado para la columna: anulación del usuario o inferido.""" - t = self._type_overrides.get(ui.summary.name) or ui.summary.inferred_type - return t if t in _TYPE_LABELS else "text" - - def _on_user_column_type_change( - self, col_name: str, type_menu: ctk.CTkOptionMenu, spanish_label: str - ) -> None: - type_id = next( - (k for k, v in _TYPE_LABELS.items() if v == spanish_label), "text" - ) - ui = self._column_uis.get(col_name) - if ui is None: - return - s = ui.summary - if type_id == s.inferred_type: - self._type_overrides.pop(col_name, None) - else: - self._type_overrides[col_name] = type_id - eff = self._effective_inferred_type(ui) - tcol = _TYPE_COLORS.get(eff, ("#64748B", "#64748B")) - type_menu.configure( - button_color=tcol[0] if tcol[0] else tcol[1], - button_hover_color=tcol[0], - fg_color=tcol[0] if tcol[0] else tcol[1], - ) - ui.stats_frame.destroy() - new_stats = self._build_stats_chips(ui.body, s, eff) - new_stats.grid(row=1, column=0, sticky="ew", padx=4, pady=(0, 4)) - ui.stats_frame = new_stats - for w in list(ui.rules_frame.winfo_children()): - w.destroy() - applic = set(applicable_rule_ids_for_type(eff)) - new_vars: list[tuple[str, ctk.CTkCheckBox, ctk.BooleanVar]] = [] - col_in_row = 0 - row_in_rules = 0 - rmap = {r["id"]: r for r in CLEANING_RULES} - for rid in (r["id"] for r in CLEANING_RULES if r["id"] in applic): - rule = rmap.get(rid) - if rule is None: - continue - var = ctk.BooleanVar(value=False) - var.trace_add( - "write", - lambda *_a, c=col_name, r=rid: self._on_rule_toggle(c, r), - ) - label = _RULE_SHORT_LABELS.get(rid, rule["name"]) - chk = ctk.CTkCheckBox( - ui.rules_frame, text=label, variable=var, - font=self.fonts["ui"], checkbox_width=18, checkbox_height=18, - ) - chk.grid(row=row_in_rules, column=col_in_row, sticky="w", padx=10, pady=4) - new_vars.append((rid, chk, var)) - col_in_row += 1 - if col_in_row >= 2: - col_in_row = 0 - row_in_rules += 1 - ui.rule_vars = new_vars - if ui.cop_mod_fecha_frame is not None: - try: - ui.cop_mod_fecha_frame.destroy() - except Exception: - pass - ui.cop_mod_fecha_frame = None - ui.cop_mod_fecha_menu = None - ui.cop_mod_fecha_var = None - ordered_after = [r["id"] for r in CLEANING_RULES if r["id"] in applic] - if "cop_mod_split" in ordered_after: - fec_frame = ctk.CTkFrame(ui.body, fg_color="transparent") - fec_frame.grid(row=4, column=0, sticky="ew", padx=4, pady=(4, 0)) - fec_frame.grid_columnconfigure(1, weight=1) - ctk.CTkLabel( - fec_frame, - text="Fecha Servicio (columna):", - font=self.fonts["italic_small"], - text_color=self.colors["gray_dark"], - ).grid(row=0, column=0, sticky="w", padx=(0, 8)) - fec_var = ctk.StringVar(value=self._suggest_fecha_column_option()) - fec_menu = ctk.CTkOptionMenu( - fec_frame, - values=self._date_columns_for_cop_mod_menu(), - variable=fec_var, - font=self.fonts["ui"], - width=260, - ) - fec_menu.grid(row=0, column=1, sticky="ew") - fec_menu.configure(state="disabled") - ui.cop_mod_fecha_frame = fec_frame - ui.cop_mod_fecha_menu = fec_menu - ui.cop_mod_fecha_var = fec_var - self._refresh_cop_mod_menus() - self.set_status(f"Columna «{col_name}»: tipo «{_TYPE_LABELS.get(eff, eff)}».") - self._on_rule_toggle(col_name) - - def _toggle_column(self, col_name: str): - ui = self._column_uis.get(col_name) - if ui is None: - return - if ui.expanded: - ui.body.grid_remove() - ui.toggle_btn.configure(text="▶") - else: - ui.body.grid() - ui.toggle_btn.configure(text="▼") - ui.expanded = not ui.expanded - - def _set_all_expanded(self, expanded: bool): - for col_name, ui in self._column_uis.items(): - if ui.expanded != expanded: - self._toggle_column(col_name) - - def _on_rule_toggle(self, col_name: str, changed_rule_id: Optional[str] = None): - ui = self._column_uis.get(col_name) - if ui is None: - return - if ( - changed_rule_id == "doc_id_split_tipo_numero" - and not self._syncing_rule_vars - ): - split_selected = any( - rid == "doc_id_split_tipo_numero" and bool(var.get()) - for rid, _chk, var in ui.rule_vars - ) - if split_selected: - for rid, _chk, var in ui.rule_vars: - if rid == "doc_type_homologate" and not bool(var.get()): - self._syncing_rule_vars = True - try: - var.set(True) - finally: - self._syncing_rule_vars = False - self.set_status( - f"Columna «{col_name}»: se activó homologación de tipo de documento." - ) - break - if ui.cop_mod_fecha_menu is not None: - cop_on = any( - rid == "cop_mod_split" and bool(var.get()) - for rid, _chk, var in ui.rule_vars - ) - ui.cop_mod_fecha_menu.configure(state="normal" if cop_on else "disabled") - selected = sum(1 for _rid, _chk, var in ui.rule_vars if bool(var.get())) - text = f"{selected} regla{'s' if selected != 1 else ''}" - color = self.colors["success"] if selected else self.colors["gray_light"] - ui.count_label.configure(text=text, text_color=color) - self._update_global_count() - - def _update_global_count(self): - cols_with_rules = 0 - total_rules = 0 - for ui in self._column_uis.values(): - sel = sum(1 for _rid, _chk, var in ui.rule_vars if bool(var.get())) - if sel: - cols_with_rules += 1 - total_rules += sel - if total_rules == 0: - self.lbl_global_count.configure( - text="0 reglas seleccionadas", - text_color=self.colors["gray_light"], - ) - else: - self.lbl_global_count.configure( - text=f"{total_rules} reglas en {cols_with_rules} columnas", - text_color=self.colors["success"], - ) - - def _clear_all_rules(self): - for ui in self._column_uis.values(): - for _rid, _chk, var in ui.rule_vars: - if bool(var.get()): - var.set(False) - - def _on_bulk_apply(self): - """Aplica la regla seleccionada a todas las columnas del tipo elegido.""" - type_label = self._bulk_type_var.get() - rule_name = self._bulk_rule_var.get() - - type_id = next( - (k for k, v in _TYPE_LABELS.items() if v == type_label), None, - ) - rule_id = next( - (r["id"] for r in CLEANING_RULES if r["name"] == rule_name), None, - ) - if not type_id or not rule_id: - return - - matched = 0 - for ui in self._column_uis.values(): - if self._effective_inferred_type(ui) != type_id: - continue - for rid, _chk, var in ui.rule_vars: - if rid == rule_id: - var.set(True) - matched += 1 - break - self.set_status( - f"Regla aplicada a {matched} columna(s) de tipo {type_label}." - ) - - def _apply_search_filter(self): - if self._search_var is None: - return - query = (self._search_var.get() or "").strip().lower() - for col_name, ui in self._column_uis.items(): - if not query or query in col_name.lower(): - ui.card.grid() - else: - ui.card.grid_remove() - - def set_delimiter_chip(self, delimiter: str): - """Actualiza el chip del delimitador detectado.""" - label = {"\t": "TAB", " ": "SPACE"}.get(delimiter, delimiter) - self.chip_delim.configure(text=f"Delim: {label}") - - # =========================================================================== - # Renderizado de tablas - # =========================================================================== - - def _render_table_placeholder(self, holder: ctk.CTkFrame, text: str): - """Limpia el holder y muestra un mensaje centrado.""" - for child in list(holder.winfo_children()): - child.destroy() - placeholder = ctk.CTkLabel( - holder, text=text, font=self.fonts["italic_small"], - text_color=self.colors["gray_light"], - ) - placeholder.grid(row=0, column=0, padx=20, pady=20, sticky="nsew") - - def _render_dataframe_table( - self, holder: ctk.CTkFrame, df: pd.DataFrame, - ) -> ttk.Treeview: - """Renderiza un DataFrame como ttk.Treeview con scroll y estilo pulido.""" - for child in list(holder.winfo_children()): - child.destroy() - - palette = self._table_palette() - style = ttk.Style() - try: - style.theme_use("clam") - except Exception: - pass - style.configure( - "Cleaning.Treeview", - rowheight=30, - font=("Consolas", 10), - background=palette["row_even"], - fieldbackground=palette["row_even"], - foreground=palette["text"], - borderwidth=0, - relief="flat", - ) - style.configure( - "Cleaning.Treeview.Heading", - font=("Roboto", 10, "bold"), - background=palette["header_bg"], - foreground=palette["header_fg"], - relief="flat", - padding=(12, 10), - borderwidth=0, - ) - # Header con separadores verticales (el borde aparece al poner relief="solid") - style.map( - "Cleaning.Treeview.Heading", - background=[("active", palette["header_hover"])], - relief=[("active", "flat"), ("!active", "flat")], - ) - style.map( - "Cleaning.Treeview", - background=[("selected", palette["selected_bg"])], - foreground=[("selected", palette["selected_fg"])], - ) - # Truco: el layout de la celda le añade un separador visible ajustando - # el padding lateral para que la zebra se "vea" entre columnas. - try: - style.layout( - "Cleaning.Treeview.Heading", - [ - ("Treeheading.cell", {"sticky": "nswe"}), - ("Treeheading.border", { - "sticky": "nswe", "children": [ - ("Treeheading.padding", { - "sticky": "nswe", "children": [ - ("Treeheading.image", {"side": "right", "sticky": ""}), - ("Treeheading.text", {"sticky": "we"}), - ], - }), - ], - }), - ], - ) - except Exception: - pass - - # Wrapper con borde sutil para resaltar la tabla - outer = ctk.CTkFrame( - holder, fg_color=palette["outer_bg"], - border_width=1, border_color=palette["border"], - corner_radius=6, - ) - outer.grid(row=0, column=0, sticky="nsew", padx=4, pady=4) - outer.grid_columnconfigure(0, weight=1) - outer.grid_rowconfigure(0, weight=1) - - cols = [str(c) for c in df.columns] - tree = ttk.Treeview( - outer, columns=cols, show="headings", style="Cleaning.Treeview", - selectmode="browse", - ) - - for i, col in enumerate(cols): - tree.heading(col, text=f" {col} ", anchor="w") - width = self._estimate_column_width(df[col], col) - anchor = "w" - tree.column(col, width=width, minwidth=80, anchor=anchor, stretch=False) - - # Zebra striping + primera columna con acento sutil - tree.tag_configure( - "odd", background=palette["row_odd"], foreground=palette["text"], - ) - tree.tag_configure( - "even", background=palette["row_even"], foreground=palette["text"], - ) - - visible = df.head(_PREVIEW_MAX_ROWS) - for idx, (_, row) in enumerate(visible.iterrows()): - # Espacios laterales en cada celda para separar visualmente las columnas - values = [f" {self._truncate_cell(v)}" for v in row.tolist()] - tag = "odd" if idx % 2 else "even" - tree.insert("", "end", values=values, tags=(tag,)) - - sb_y = ttk.Scrollbar(outer, orient="vertical", command=tree.yview) - sb_x = ttk.Scrollbar(outer, orient="horizontal", command=tree.xview) - tree.configure(yscrollcommand=sb_y.set, xscrollcommand=sb_x.set) - - tree.grid(row=0, column=0, sticky="nsew") - sb_y.grid(row=0, column=1, sticky="ns") - sb_x.grid(row=1, column=0, sticky="ew") - - holder.grid_columnconfigure(0, weight=1) - holder.grid_rowconfigure(0, weight=1) - return tree - - def _table_palette(self) -> dict[str, str]: - """Paleta de colores adaptada al tema claro/oscuro activo.""" - try: - mode = ctk.get_appearance_mode() - except Exception: - mode = "Light" - if mode == "Dark": - return { - "outer_bg": "#1F2937", - "border": "#374151", - "header_bg": "#111827", - "header_fg": "#F9FAFB", - "header_hover": "#1F2937", - "row_even": "#1F2937", - "row_odd": "#273142", - "text": "#E5E7EB", - "selected_bg": "#2563EB", - "selected_fg": "#FFFFFF", - } - return { - "outer_bg": "#FFFFFF", - "border": "#D1D5DB", - "header_bg": "#1F2937", - "header_fg": "#FFFFFF", - "header_hover": "#374151", - "row_even": "#FFFFFF", - "row_odd": "#F3F4F6", - "text": "#111827", - "selected_bg": "#2563EB", - "selected_fg": "#FFFFFF", - } - - @staticmethod - def _truncate_cell(value) -> str: - if value is None: - return "" - if isinstance(value, float) and pd.isna(value): - return "" - s = str(value) - if len(s) > _PREVIEW_MAX_CELL_LEN: - return s[: _PREVIEW_MAX_CELL_LEN - 3] + "..." - return s - - @staticmethod - def _estimate_column_width(series: pd.Series, col_name: str) -> int: - """Estima el ancho de una columna en pixeles a partir del contenido.""" - char_w = 8 - # Muestreo más amplio para evitar cortar valores largos. - sample = series.head(40).astype(str).tolist() - max_len = max([len(col_name)] + [len(s) for s in sample] + [6]) - max_len = min(max_len, _PREVIEW_MAX_CELL_LEN) - # Padding extra para los espacios laterales agregados en cada celda - # (formato " valor ") que mejoran la separación visual entre columnas. - return max(110, min(460, max_len * char_w + 40)) diff --git a/gui/tabs/main_tab.py b/gui/tabs/main_tab.py deleted file mode 100644 index 6e1730c..0000000 --- a/gui/tabs/main_tab.py +++ /dev/null @@ -1,574 +0,0 @@ -"""Pestaña Procesar Documentos de la aplicación GUI.""" -import customtkinter as ctk -from gui.styles import AppStyles -from application.app_prefs import get_allowed_document_types - - -class MainProcessingTab: - """Encapsula la UI de la pestaña Procesar Documentos.""" - - def __init__(self, parent_tab): - """Inicializa la pestaña de Procesar Documentos. - - Args: - parent_tab: El CTkFrame que representa la pestaña. - """ - self.tab = parent_tab - self.fonts = AppStyles.get_fonts() - self.colors = AppStyles.get_colors() - self.spacing = AppStyles.get_spacing() - - # Variables internas - self._doc_type_rows = [] - - self._setup_layout() - self._create_paths_section() - self._create_config_section() - self._create_process_button() - self._create_output_section() - # Conectar el trace después de que todos los widgets existen - self.mode_var.trace_add("write", lambda *_: self.update_ui_for_mode()) - - def _setup_layout(self): - """Configura el layout base de la pestaña.""" - self.tab.grid_columnconfigure(0, weight=1) - self.tab.grid_rowconfigure(0, weight=1) - - self.scroll_container = ctk.CTkScrollableFrame(self.tab, fg_color="transparent") - self.scroll_container.grid(row=0, column=0, sticky="nsew", padx=5, pady=5) - self.scroll_container.grid_columnconfigure(0, weight=1) - - def _create_paths_section(self): - """Crea la sección de rutas y archivos.""" - self.frame_paths = ctk.CTkFrame(self.scroll_container) - self.frame_paths.grid(row=0, column=0, sticky="ew", padx=10, pady=(5, 2)) - self.frame_paths.grid_columnconfigure(1, weight=1) - - # Título - lbl_paths = ctk.CTkLabel( - self.frame_paths, - text="Rutas y Archivos", - font=self.fonts["section"] - ) - lbl_paths.grid(row=0, column=0, columnspan=3, sticky="w", padx=15, pady=(5, 2)) - - # Carpeta Raíz - self.root_label = ctk.CTkLabel(self.frame_paths, text="Carpeta Raíz:", font=self.fonts["ui"]) - self.root_label.grid(row=1, column=0, padx=15, pady=2, sticky="w") - - self.root_entry = ctk.CTkEntry( - self.frame_paths, - placeholder_text="Selecciona la carpeta con los documentos...", - font=self.fonts["ui"] - ) - self.root_entry.grid(row=1, column=1, padx=5, pady=2, sticky="ew") - - self.root_button = ctk.CTkButton(self.frame_paths, text="Explorar...", font=self.fonts["ui"], width=100) - self.root_button.grid(row=1, column=2, padx=15, pady=2) - - # CSV Salida - self.csv_label = ctk.CTkLabel(self.frame_paths, text="Archivo CSV:", font=self.fonts["ui"]) - self.csv_label.grid(row=2, column=0, padx=15, pady=2, sticky="w") - - self.csv_entry = ctk.CTkEntry( - self.frame_paths, - placeholder_text="Ruta para guardar el reporte CSV...", - font=self.fonts["ui"] - ) - self.csv_entry.grid(row=2, column=1, padx=5, pady=2, sticky="ew") - - self.csv_button = ctk.CTkButton( - self.frame_paths, - text="Guardar en...", - font=self.fonts["ui"], - width=100, - fg_color="transparent", - border_width=1 - ) - self.csv_button.grid(row=2, column=2, padx=15, pady=2) - - # Selector de Modo (Radicados vs General) - self.mode_var = ctk.StringVar(value="radicados") - - lbl_mode = ctk.CTkLabel(self.frame_paths, text="Estructura:", font=self.fonts["ui"]) - lbl_mode.grid(row=3, column=0, padx=15, pady=2, sticky="w") - - self.frame_radios_main = ctk.CTkFrame(self.frame_paths, fg_color="transparent") - self.frame_radios_main.grid(row=3, column=1, columnspan=2, sticky="w", padx=5, pady=2) - - self.radio_radicados_main = ctk.CTkRadioButton( - self.frame_radios_main, - text="Por Radicados (Subcarpetas)", - variable=self.mode_var, - value="radicados", - font=self.fonts["ui"] - ) - self.radio_radicados_main.pack(side="left", padx=(0, 20)) - - self.radio_general_main = ctk.CTkRadioButton( - self.frame_radios_main, - text="Carpeta General (Archivos Planos)", - variable=self.mode_var, - value="general", - font=self.fonts["ui"] - ) - self.radio_general_main.pack(side="left") - - # Filtro por Radicados - self.filter_enabled_var = ctk.BooleanVar(value=False) - self.filter_check = ctk.CTkCheckBox( - self.frame_paths, - text="Procesar solo radicados seleccionados", - variable=self.filter_enabled_var, - font=self.fonts["ui"], - command=self._on_filter_toggle - ) - self.filter_check.grid(row=4, column=0, columnspan=3, padx=15, pady=(4, 0), sticky="w") - - self.filter_frame = ctk.CTkFrame(self.frame_paths, fg_color="transparent") - self.filter_frame.grid(row=5, column=0, columnspan=3, padx=15, pady=(2, 6), sticky="ew") - self.filter_frame.grid_columnconfigure(0, weight=1) - - filter_hint = ctk.CTkLabel( - self.filter_frame, - text="Ingresa los radicados a procesar (uno por línea o separados por coma):", - font=self.fonts["ui"], - text_color=self.colors["text_muted"] - ) - filter_hint.grid(row=0, column=0, sticky="w", pady=(0, 2)) - - self.filter_textbox = ctk.CTkTextbox( - self.filter_frame, - height=80, - font=self.fonts["mono"], - ) - self.filter_textbox.grid(row=1, column=0, sticky="ew") - - # Ocultar hasta que se active el checkbox - self.filter_frame.grid_remove() - - def _on_filter_toggle(self): - """Muestra u oculta el área de texto del filtro.""" - if self.filter_enabled_var.get(): - self.filter_frame.grid() - else: - self.filter_frame.grid_remove() - - def _create_config_section(self): - """Crea la sección de configuración de tipos de documento y TIFF.""" - self.frame_config = ctk.CTkFrame(self.scroll_container) - self.frame_config.grid(row=1, column=0, sticky="ew", padx=10, pady=2) - self.frame_config.grid_columnconfigure(0, weight=1) - - # Keywords oculto (legacy) - self.kw_label = ctk.CTkLabel(self.frame_config, text="Keywords:") - self.kw_entry = ctk.CTkEntry(self.frame_config) - self.kw_entry.insert(0, "Factura_O_Documento_Equivalente,factura") - - # Tipos de Documento - self.doc_types_container = ctk.CTkFrame(self.frame_config, fg_color="transparent") - self.doc_types_container.grid(row=0, column=0, sticky="nsew", padx=10, pady=(5, 2)) - self.doc_types_container.grid_columnconfigure(0, weight=1) - - self.doc_types_header = ctk.CTkFrame(self.doc_types_container, fg_color="transparent") - self.doc_types_header.grid(row=0, column=0, sticky="ew", pady=(0, 2)) - self.doc_types_header.grid_columnconfigure(0, weight=1) - - self.doc_types_label = ctk.CTkLabel( - self.doc_types_header, - text="Definición de Tipos de Documento", - font=self.fonts["section"] - ) - self.doc_types_label.grid(row=0, column=0, sticky="w") - - allowed_types_text = ", ".join(get_allowed_document_types()) - self.doc_types_hint = ctk.CTkLabel( - self.doc_types_header, - text=f"Tipos válidos para documento_forzate: {allowed_types_text}. NIT no aplica.", - font=self.fonts["ui"], - text_color=self.colors["text_muted"], - ) - self.doc_types_hint.grid(row=1, column=0, sticky="w", pady=(2, 0)) - - self.add_doc_type_btn = ctk.CTkButton( - self.doc_types_header, - text="+ Añadir Tipo", - font=self.fonts["ui"], - width=120, - height=28 - ) - self.add_doc_type_btn.grid(row=0, column=1, rowspan=2, sticky="e") - self.add_doc_type_btn.configure(command=self._add_doc_type_row) - - # Lista de Tipos - self.doc_types_list = ctk.CTkFrame(self.doc_types_container, fg_color="transparent") - self.doc_types_list.grid(row=1, column=0, sticky="ew", pady=(2, 0)) - self.doc_types_list.grid_columnconfigure(0, weight=1) - - # Fila por defecto - self._add_doc_type_row(name="Factura", keyword="Factura_O_Documento_Equivalente") - - # Configuración Avanzada (TIFF Range) - self.frame_advanced = ctk.CTkFrame(self.frame_config, fg_color="transparent") - self.frame_advanced.grid(row=1, column=0, sticky="ew", padx=10, pady=(0, 2)) - self.frame_advanced.grid_columnconfigure(0, weight=1) - - self.retry_empty_var = ctk.BooleanVar(value=False) - self.retry_empty_check = ctk.CTkCheckBox( - self.frame_advanced, - text="Reintentar si la fila sale vacía (máx. 2 intentos)", - variable=self.retry_empty_var, - font=self.fonts["ui"] - ) - self.retry_empty_check.grid(row=0, column=0, columnspan=4, padx=5, pady=(0, 6), sticky="w") - - self.tiff_start_label = ctk.CTkLabel( - self.frame_advanced, - text="Pág. Inicio TIFF:", - font=self.fonts["ui"] - ) - self.tiff_start_entry = ctk.CTkEntry(self.frame_advanced, width=60) - self.tiff_start_entry.insert(0, "0") - - self.tiff_end_label = ctk.CTkLabel( - self.frame_advanced, - text="Pág. Fin TIFF:", - font=self.fonts["ui"] - ) - self.tiff_end_entry = ctk.CTkEntry(self.frame_advanced, width=60) - self.tiff_end_entry.insert(0, "0") - - # Ocultar inicialmente - self.tiff_start_label.grid_forget() - self.tiff_start_entry.grid_forget() - self.tiff_end_label.grid_forget() - self.tiff_end_entry.grid_forget() - - # Campos legacy ocultos - self.fields_entry = ctk.CTkEntry(self.tab) - self.fields_label = ctk.CTkLabel(self.tab, text="") - - def _create_process_button(self): - """Crea el botón de acción principal.""" - self.process_button = ctk.CTkButton( - self.scroll_container, - text="INICIAR PROCESAMIENTO", - font=self.fonts["bold_large"], - height=35, - fg_color=self.colors["primary_button"], - hover_color=self.colors["primary_hover"] - ) - self.process_button.grid(row=2, column=0, padx=15, pady=(10, 15), sticky="ew") - - def _create_output_section(self): - """Crea la sección de logs/salida.""" - self.log_frame = ctk.CTkFrame(self.scroll_container) - self.log_frame.grid(row=3, column=0, sticky="ew", padx=10, pady=(0, 15)) - self.log_frame.grid_columnconfigure(0, weight=1) - - lbl_logs = ctk.CTkLabel( - self.log_frame, - text="Registro de Actividad:", - font=self.fonts["ui"], - text_color=self.colors["text_muted"] - ) - lbl_logs.grid(row=0, column=0, sticky="w", padx=10, pady=(5, 0)) - - self.output_text = ctk.CTkTextbox( - self.log_frame, - font=self.fonts["mono"], - activate_scrollbars=True, - height=150 - ) - self.output_text.grid(row=1, column=0, padx=5, pady=5, sticky="ew") - - def _add_doc_type_row(self, name: str = "", keyword: str = "") -> None: - """Añade una fila de tipo de documento.""" - row = ctk.CTkFrame(self.doc_types_list, fg_color="transparent") - row.pack(fill="x", pady=2) - - grid_inner = ctk.CTkFrame(row, fg_color="transparent") - grid_inner.pack(fill="x", padx=5, pady=2) - grid_inner.grid_columnconfigure(0, weight=1) - grid_inner.grid_columnconfigure(1, weight=1) - - name_entry = ctk.CTkEntry( - grid_inner, - placeholder_text="Nombre Lógico", - height=28, - font=self.fonts["ui"] - ) - if name: - name_entry.insert(0, name) - name_entry.grid(row=0, column=0, padx=(0, 5), sticky="ew") - - kw_entry = ctk.CTkEntry( - grid_inner, - placeholder_text="Patrón / Keyword", - height=28, - font=self.fonts["ui"] - ) - if keyword: - kw_entry.insert(0, keyword) - kw_entry.grid(row=0, column=1, padx=(0, 5), sticky="ew") - - def _remove(): - # Buscar y eliminar de la lista interna - for i, r_data in enumerate(self._doc_type_rows): - if r_data["frame"] == row: - self._doc_type_rows.pop(i) - break - row.destroy() - - remove_btn = ctk.CTkButton( - grid_inner, - text="×", - width=30, - height=28, - fg_color=self.colors["danger"], - hover_color=self.colors["danger_hover"], - command=_remove - ) - remove_btn.grid(row=0, column=2, sticky="e") - - self._doc_type_rows.append({ - "frame": row, - "name_entry": name_entry, - "kw_entry": kw_entry, - "remove_btn": remove_btn - }) - - # --- API Pública --- - - def set_on_select_root(self, callback): - """Vincula callback al botón de selección de carpeta raíz.""" - self.root_button.configure(command=callback) - - def set_on_select_csv(self, callback): - """Vincula callback al botón de selección de CSV.""" - self.csv_button.configure(command=callback) - - def set_on_process(self, callback): - """Vincula callback al botón de procesar.""" - self.process_button.configure(command=callback) - - def set_on_mode_change(self, callback): - """Vincula callback al cambio de modo.""" - self.mode_var.trace("w", lambda *args: callback()) - - def get_root_path(self) -> str: - """Devuelve la ruta raíz seleccionada.""" - return self.root_entry.get().strip() - - def set_root_path(self, path: str) -> None: - """Establece la ruta raíz.""" - self.root_entry.configure(state="normal") - self.root_entry.delete(0, "end") - self.root_entry.insert(0, path) - - def get_csv_path(self) -> str: - """Devuelve la ruta del CSV.""" - return self.csv_entry.get().strip() - - def set_csv_path(self, path: str) -> None: - """Establece la ruta del CSV.""" - self.csv_entry.configure(state="normal") - self.csv_entry.delete(0, "end") - self.csv_entry.insert(0, path) - - def get_process_mode(self) -> str: - """Devuelve el modo de procesamiento: 'radicados' o 'general'.""" - try: - v = (self.mode_var.get() or "").strip().lower() - return v if v in ("radicados", "general") else "radicados" - except Exception: - return "radicados" - - def set_process_mode(self, mode: str) -> None: - """Establece el modo de procesamiento y refresca la UI.""" - try: - normalized = (mode or "").strip().lower() - self.mode_var.set(normalized if normalized in ("radicados", "general") else "radicados") - self.update_ui_for_mode() - except Exception: - self.mode_var.set("radicados") - self.update_ui_for_mode() - - def get_keywords(self) -> str: - """Devuelve los keywords.""" - return self.kw_entry.get().strip() - - def get_fields(self) -> str: - """Devuelve los fields (legacy).""" - return self.fields_entry.get().strip() - - def get_tiff_start_page(self) -> int: - """Devuelve la página de inicio TIFF.""" - try: - value = self.tiff_start_entry.get().strip() - return int(value) if value else 0 - except Exception: - return 0 - - def get_tiff_end_page(self) -> int: - """Devuelve la página final TIFF.""" - try: - value = self.tiff_end_entry.get().strip() - return int(value) if value else 0 - except Exception: - return 0 - - def get_retry_empty_enabled(self) -> bool: - """Indica si se debe reintentar cuando la fila solo trae el radicado.""" - try: - return bool(self.retry_empty_var.get()) - except Exception: - return False - - def set_retry_empty_enabled(self, enabled: bool) -> None: - """Establece el reintento automático por fila vacía.""" - self.retry_empty_var.set(bool(enabled)) - - def get_radicado_filter_enabled(self) -> bool: - """Indica si el filtro de radicados está activo.""" - try: - return bool(self.filter_enabled_var.get()) - except Exception: - return False - - def get_radicado_filter_list(self) -> list[str]: - """Devuelve la lista de radicados del filtro (sin blancos ni duplicados).""" - try: - raw = self.filter_textbox.get("1.0", "end") - items = [] - for part in raw.replace(",", "\n").splitlines(): - s = part.strip() - if s: - items.append(s) - return list(dict.fromkeys(items)) # preserva orden, elimina duplicados - except Exception: - return [] - - def set_radicado_filter_enabled(self, enabled: bool) -> None: - """Activa o desactiva el filtro y refresca la UI.""" - self.filter_enabled_var.set(bool(enabled)) - self._on_filter_toggle() - - def set_radicado_filter_list(self, radicados: list[str]) -> None: - """Carga una lista de radicados en el área de texto del filtro.""" - try: - self.filter_textbox.delete("1.0", "end") - self.filter_textbox.insert("1.0", "\n".join(radicados)) - except Exception: - pass - - def show_tiff_range_controls(self, show: bool) -> None: - """Muestra u oculta los controles de rango TIFF.""" - if show: - self.tiff_start_label.grid(row=0, column=0, padx=5, pady=5) - self.tiff_start_entry.grid(row=0, column=1, padx=5, pady=5) - self.tiff_end_label.grid(row=0, column=2, padx=5, pady=5) - self.tiff_end_entry.grid(row=0, column=3, padx=5, pady=5) - else: - self.tiff_start_label.grid_forget() - self.tiff_start_entry.grid_forget() - self.tiff_end_label.grid_forget() - self.tiff_end_entry.grid_forget() - - def get_doc_types(self) -> list[dict]: - """Devuelve la lista de tipos de documento.""" - out = [] - for it in self._doc_type_rows: - try: - name = it["name_entry"].get().strip() - kw = it["kw_entry"].get().strip() - if name and kw: - out.append({"name": name, "keyword": kw}) - except: - pass - return out - - def set_doc_types(self, doc_types: list[dict]) -> None: - """Establece los tipos de documento.""" - for it in self._doc_type_rows: - it["frame"].destroy() - self._doc_type_rows = [] - - if not doc_types: - self._add_doc_type_row("Factura", "Factura_O_Documento_Equivalente") - return - - for dt in doc_types: - self._add_doc_type_row(str(dt.get("name", "")), str(dt.get("keyword", ""))) - - def validate_doc_types(self) -> tuple[bool, str | None]: - """Valida los tipos de documento.""" - seen = set() - for dt in self.get_doc_types(): - name = dt.get("name") - if name in seen: - return False, f"Nombre duplicado: {name}" - seen.add(name) - if not seen: - return False, "Debe haber al menos un tipo de documento." - return True, None - - def set_allowed_document_types_hint(self, allowed_types: list[str]) -> None: - """Actualiza el texto de ayuda de tipos válidos.""" - text = ", ".join(allowed_types) - self.doc_types_hint.configure(text=f"Tipos válidos para documento_forzate: {text}. NIT no aplica.") - - def update_ui_for_mode(self): - """Actualiza la visibilidad de componentes según el modo seleccionado.""" - mode = self.get_process_mode() - if mode == "general": - # Ocultar Tipos de Documento - self.doc_types_container.grid_forget() - # Mostrar Keywords (opcional, para filtrar) - self.kw_label.grid(row=1, column=0, padx=15, pady=2, sticky="w") - self.kw_entry.grid(row=1, column=1, padx=5, pady=2, sticky="ew") - - # Limpiar valor por defecto si existe - curr = self.kw_entry.get() - if "Factura_O_Documento_Equivalente" in curr: - self.kw_entry.delete(0, "end") - else: - # Mostrar Tipos de Documento - self.doc_types_container.grid(row=0, column=0, sticky="nsew", padx=10, pady=(5, 2)) - # Ocultar Keywords - self.kw_label.grid_forget() - self.kw_entry.grid_forget() - - def append_output(self, text: str) -> None: - """Añade texto al área de salida.""" - self.output_text.insert("end", text + "\n") - self.output_text.see("end") - - def append_output_safe(self, text: str) -> None: - """Añade texto al área de salida de forma thread-safe.""" - # Nota: se espera que sea llamado con self.after(0, ...) - self.append_output(text) - - def enable_controls(self, enabled: bool): - """Habilita o deshabilita los controles.""" - state = "normal" if enabled else "disabled" - - self.process_button.configure(state=state, text="PROCESANDO..." if not enabled else "INICIAR PROCESAMIENTO") - if not enabled: - self.process_button.configure(fg_color="gray") - else: - self.process_button.configure(fg_color=self.colors["primary_button"]) - - self.root_button.configure(state=state) - self.root_entry.configure(state=state) - self.csv_button.configure(state=state) - self.csv_entry.configure(state=state) - self.add_doc_type_btn.configure(state=state) - self.retry_empty_check.configure(state=state) - self.filter_check.configure(state=state) - self.filter_textbox.configure(state=state) - - for row in self._doc_type_rows: - row["name_entry"].configure(state=state) - row["kw_entry"].configure(state=state) - row["remove_btn"].configure(state=state) diff --git a/gui/tabs/metrics_tab.py b/gui/tabs/metrics_tab.py deleted file mode 100644 index 760dc6a..0000000 --- a/gui/tabs/metrics_tab.py +++ /dev/null @@ -1,351 +0,0 @@ -"""Pestaña Dashboard & Métricas de la aplicación GUI. - -Muestra KPIs, gráficos y detalle de costos para Extracción, Normalización -y Post-Procesamiento. -""" -import customtkinter as ctk -import tkinter.ttk as ttk -from gui.styles import AppStyles - -try: - import matplotlib - matplotlib.use("Agg") - import numpy as np - from matplotlib.figure import Figure - from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg - HAS_MPL = True -except Exception: - HAS_MPL = False - - -class MetricsTab: - """Encapsula la UI de la pestaña Dashboard & Métricas.""" - - def __init__(self, parent_tab): - self.tab = parent_tab - self.fonts = AppStyles.get_fonts() - self.colors = AppStyles.get_colors() - self.spacing = AppStyles.get_spacing() - - self._cost_line_points_x: list[int] = [] - self._cost_line_points_y: list[float] = [] - - self._pp_history: list[dict] = [] - - self._setup_layout() - self._create_kpi_section() - self._create_pp_kpi_section() - self._create_charts_section() - self._create_comparison_section() - self._create_detail_section() - - # -- Layout ---------------------------------------------------------------- - - def _setup_layout(self): - self.tab.grid_columnconfigure(0, weight=1) - self.tab.grid_rowconfigure(0, weight=0) - - self.scroll = ctk.CTkScrollableFrame(self.tab, fg_color="transparent") - self.scroll.grid(row=0, column=0, sticky="nsew", padx=5, pady=5) - self.scroll.grid_columnconfigure(0, weight=1) - self.tab.grid_rowconfigure(0, weight=1) - - # -- KPI cards helper ------------------------------------------------------ - - @staticmethod - def _make_kpi_card(parent, title, row, col, accent=None): - card = ctk.CTkFrame(parent) - card.grid(row=row, column=col, sticky="nsew", padx=4, pady=4) - card.grid_columnconfigure(0, weight=1) - - header = ctk.CTkFrame(card, height=6, fg_color=accent or ("#1f77b4", "#1f77b4"), corner_radius=0) - header.grid(row=0, column=0, sticky="ew") - header.grid_propagate(False) - - lbl_title = ctk.CTkLabel(card, text=title, font=ctk.CTkFont(size=12, weight="bold")) - lbl_title.grid(row=1, column=0, sticky="w", padx=12, pady=(8, 2)) - - lbl_value = ctk.CTkLabel(card, text="—", font=ctk.CTkFont(family="Consolas", size=11), justify="left") - lbl_value.grid(row=2, column=0, sticky="w", padx=12, pady=(0, 8)) - - return lbl_value - - # -- Extracción KPIs ------------------------------------------------------- - - def _create_kpi_section(self): - frame = ctk.CTkFrame(self.scroll) - frame.grid(row=0, column=0, sticky="ew", padx=2, pady=(2, 1)) - frame.grid_columnconfigure((0, 1, 2, 3), weight=1) - - lbl = ctk.CTkLabel(frame, text="Extracción de Documentos", font=ctk.CTkFont(size=15, weight="bold")) - lbl.grid(row=0, column=0, columnspan=4, sticky="w", padx=12, pady=(10, 4)) - - self.kpi_overall = self._make_kpi_card(frame, "Total Sesión", 1, 0, accent=("#2E86C1", "#2E86C1")) - self.kpi_extract = self._make_kpi_card(frame, "Fase: Extracción", 1, 1, accent=("#1f77b4", "#1f77b4")) - self.kpi_normalize = self._make_kpi_card(frame, "Fase: Normalización", 1, 2, accent=("#ff7f0e", "#ff7f0e")) - self.kpi_docs = self._make_kpi_card(frame, "Documentos", 1, 3, accent=("#27AE60", "#27AE60")) - - zero = "input: 0\ncached: 0\noutput: 0\nUSD: $0.0000" - self.kpi_overall.configure(text=zero) - self.kpi_extract.configure(text=zero) - self.kpi_normalize.configure(text=zero) - self.kpi_docs.configure(text="Procesados: 0\nCosto prom: $0.0000") - - # -- Post-Procesamiento KPIs ---------------------------------------------- - - def _create_pp_kpi_section(self): - frame = ctk.CTkFrame(self.scroll) - frame.grid(row=1, column=0, sticky="ew", padx=2, pady=1) - frame.grid_columnconfigure((0, 1, 2, 3), weight=1) - - lbl = ctk.CTkLabel(frame, text="Post-Procesamiento (Agente Codex)", font=ctk.CTkFont(size=15, weight="bold")) - lbl.grid(row=0, column=0, columnspan=4, sticky="w", padx=12, pady=(10, 4)) - - self.kpi_pp_total = self._make_kpi_card(frame, "Total Acumulado", 1, 0, accent=("#8E44AD", "#8E44AD")) - self.kpi_pp_calls = self._make_kpi_card(frame, "Llamadas API", 1, 1, accent=("#9B59B6", "#9B59B6")) - self.kpi_pp_tokens = self._make_kpi_card(frame, "Tokens", 1, 2, accent=("#AF7AC5", "#AF7AC5")) - self.kpi_pp_last = self._make_kpi_card(frame, "Última ejecución", 1, 3, accent=("#D2B4DE", "#D2B4DE")) - - self.kpi_pp_total.configure(text="USD: $0.0000") - self.kpi_pp_calls.configure(text="Total: 0\nReintentos: 0") - self.kpi_pp_tokens.configure(text="input: 0\noutput: 0") - self.kpi_pp_last.configure(text="—") - - # -- Charts ---------------------------------------------------------------- - - def _create_charts_section(self): - frame = ctk.CTkFrame(self.scroll) - frame.grid(row=2, column=0, sticky="ew", padx=2, pady=1) - frame.grid_columnconfigure((0, 1), weight=1) - - lbl = ctk.CTkLabel(frame, text="Gráficos", font=ctk.CTkFont(size=15, weight="bold")) - lbl.grid(row=0, column=0, columnspan=2, sticky="w", padx=12, pady=(10, 4)) - - if HAS_MPL: - bg = "#2b2b2b" - - self.fig_line = Figure(figsize=(4.5, 2.8), dpi=100, facecolor=bg) - self.ax_line = self.fig_line.add_subplot(111) - self._style_axis(self.ax_line, "Costo Acumulado por Documento (USD)") - self._line_plot, = self.ax_line.plot([], [], marker="o", markersize=4, linewidth=1.5, color="#2ecc71") - self.fig_line.tight_layout(pad=1.5) - - self.canvas_line = FigureCanvasTkAgg(self.fig_line, master=frame) - self.canvas_line.get_tk_widget().grid(row=1, column=0, padx=8, pady=(0, 8), sticky="nsew") - - self.fig_bars = Figure(figsize=(4.5, 2.8), dpi=100, facecolor=bg) - self.ax_bars = self.fig_bars.add_subplot(111) - self._style_axis(self.ax_bars, "Tokens por Fase") - self.fig_bars.tight_layout(pad=1.5) - - self.canvas_bars = FigureCanvasTkAgg(self.fig_bars, master=frame) - self.canvas_bars.get_tk_widget().grid(row=1, column=1, padx=8, pady=(0, 8), sticky="nsew") - else: - ctk.CTkLabel(frame, text="matplotlib no disponible — instálelo para gráficos").grid( - row=1, column=0, columnspan=2, pady=10, - ) - - @staticmethod - def _style_axis(ax, title): - bg = "#2b2b2b" - fg = "#cccccc" - ax.set_facecolor(bg) - ax.set_title(title, fontsize=10, color=fg, pad=8) - ax.tick_params(colors=fg, labelsize=8) - for spine in ax.spines.values(): - spine.set_color("#555555") - ax.xaxis.label.set_color(fg) - ax.yaxis.label.set_color(fg) - - # -- Comparison table ------------------------------------------------------ - - def _create_comparison_section(self): - frame = ctk.CTkFrame(self.scroll) - frame.grid(row=3, column=0, sticky="ew", padx=2, pady=1) - frame.grid_columnconfigure(0, weight=1) - - lbl = ctk.CTkLabel(frame, text="Tarifas de Modelos (USD / 1M tokens)", font=ctk.CTkFont(size=15, weight="bold")) - lbl.grid(row=0, column=0, sticky="w", padx=12, pady=(10, 4)) - - style = ttk.Style() - style.configure("Metrics.Treeview", rowheight=26, font=("Consolas", 10)) - style.configure("Metrics.Treeview.Heading", font=("Roboto", 11, "bold")) - - self.tree_comparison = ttk.Treeview( - frame, - columns=("model", "input", "cached", "output"), - show="headings", height=6, style="Metrics.Treeview", - ) - self.tree_comparison.heading("model", text="Modelo") - self.tree_comparison.heading("input", text="Input") - self.tree_comparison.heading("cached", text="Cached Input") - self.tree_comparison.heading("output", text="Output") - - self.tree_comparison.column("model", width=180, anchor="w") - self.tree_comparison.column("input", width=110, anchor="e") - self.tree_comparison.column("cached", width=110, anchor="e") - self.tree_comparison.column("output", width=110, anchor="e") - - self.tree_comparison.grid(row=1, column=0, sticky="ew", padx=12, pady=(0, 10)) - - # -- Detail log ------------------------------------------------------------ - - def _create_detail_section(self): - frame = ctk.CTkFrame(self.scroll) - frame.grid(row=4, column=0, sticky="ew", padx=2, pady=(1, 2)) - frame.grid_columnconfigure(0, weight=1) - - lbl = ctk.CTkLabel(frame, text="Detalle de uso (log)", font=ctk.CTkFont(size=15, weight="bold")) - lbl.grid(row=0, column=0, sticky="w", padx=12, pady=(10, 4)) - - self.usage_text = ctk.CTkTextbox(frame, height=160, font=self.fonts["mono_small"], wrap="none") - self.usage_text.grid(row=1, column=0, sticky="ew", padx=12, pady=(0, 10)) - - # ========================================================================= - # API PÚBLICA - # ========================================================================= - - # -- Extracción KPIs ------------------------------------------------------- - - def update_kpis(self, overall: dict, extract_phase: dict, normalize_phase: dict) -> None: - def fmt(d): - return ( - f"input: {d.get('input', 0):,}\n" - f"cached: {d.get('cached', 0):,}\n" - f"output: {d.get('output', 0):,}\n" - f"USD: ${d.get('usd', 0.0):.4f}" - ) - self.kpi_overall.configure(text=fmt(overall)) - self.kpi_extract.configure(text=fmt(extract_phase)) - self.kpi_normalize.configure(text=fmt(normalize_phase)) - - def update_doc_kpi(self, processed: int, avg_cost: float) -> None: - self.kpi_docs.configure(text=f"Procesados: {processed}\nCosto prom: ${avg_cost:.4f}") - - # -- Post-Procesamiento KPIs ---------------------------------------------- - - def update_pp_kpis( - self, - total_usd: float, - total_calls: int, - total_retries: int, - input_tokens: int, - output_tokens: int, - last_description: str, - ) -> None: - self.kpi_pp_total.configure(text=f"USD: ${total_usd:.4f}") - self.kpi_pp_calls.configure(text=f"Total: {total_calls}\nReintentos: {total_retries}") - self.kpi_pp_tokens.configure(text=f"input: {input_tokens:,}\noutput: {output_tokens:,}") - self.kpi_pp_last.configure(text=last_description or "—") - - # -- Charts ---------------------------------------------------------------- - - def update_cost_line(self, doc_index: int, total_cost_usd: float) -> None: - if not HAS_MPL: - return - try: - self._cost_line_points_x.append(doc_index) - self._cost_line_points_y.append(total_cost_usd) - self._line_plot.set_data(self._cost_line_points_x, self._cost_line_points_y) - self.ax_line.relim() - self.ax_line.autoscale_view() - self.canvas_line.draw_idle() - except Exception: - pass - - def update_token_bars( - self, - input_extract: int, cached_extract: int, output_extract: int, - input_norm: int, cached_norm: int, output_norm: int, - input_pp: int = 0, output_pp: int = 0, - ) -> None: - if not HAS_MPL: - return - try: - self.ax_bars.clear() - self._style_axis(self.ax_bars, "Tokens por Fase") - - labels = ["Input", "Cached", "Output"] - extract_vals = [input_extract, cached_extract, output_extract] - norm_vals = [input_norm, cached_norm, output_norm] - pp_vals = [input_pp, 0, output_pp] - - has_pp = input_pp > 0 or output_pp > 0 - - x = np.arange(len(labels)) - width = 0.25 if has_pp else 0.35 - offset = width - - self.ax_bars.bar(x - offset, extract_vals, width, label="Extracción", color="#3498db") - self.ax_bars.bar(x, norm_vals, width, label="Normalización", color="#e67e22") - if has_pp: - self.ax_bars.bar(x + offset, pp_vals, width, label="Post-Proc.", color="#9b59b6") - - self.ax_bars.set_xticks(x, labels) - leg = self.ax_bars.legend(fontsize=8, framealpha=0.6) - for text in leg.get_texts(): - text.set_color("#cccccc") - self.fig_bars.tight_layout(pad=1.5) - self.canvas_bars.draw_idle() - except Exception: - pass - - # -- Comparison table ------------------------------------------------------ - - def update_comparison_table(self, comparisons: list[dict]) -> None: - try: - for item in self.tree_comparison.get_children(): - self.tree_comparison.delete(item) - for c in comparisons: - self.tree_comparison.insert("", "end", values=( - c.get("model", "N/D"), - f"${c.get('input', 0.0):.3f}", - f"${c.get('cached_input', 0.0):.4f}", - f"${c.get('output', 0.0):.2f}", - )) - except Exception: - pass - - # -- Detail log ------------------------------------------------------------ - - def append_usage(self, text: str) -> None: - self.usage_text.insert("end", text + "\n") - self.usage_text.see("end") - - def append_usage_safe(self, text: str) -> None: - self.append_usage(text) - - def append_usage_row(self, phase: str, radicado: str, usage: dict | None, cost: dict | None) -> None: - try: - usd = (cost or {}).get("total_usd", 0.0) - prompt = int((usage or {}).get("prompt_tokens", 0) or 0) - comp = int((usage or {}).get("completion_tokens", 0) or 0) - self.append_usage( - f"[{phase:<12}] {radicado:<22} | in={prompt:>7,} out={comp:>6,} | ${usd:.4f}" - ) - except Exception: - pass - - def clear(self) -> None: - self.usage_text.delete("1.0", "end") - - zero = "input: 0\ncached: 0\noutput: 0\nUSD: $0.0000" - self.kpi_overall.configure(text=zero) - self.kpi_extract.configure(text=zero) - self.kpi_normalize.configure(text=zero) - self.kpi_docs.configure(text="Procesados: 0\nCosto prom: $0.0000") - - if HAS_MPL: - self._cost_line_points_x.clear() - self._cost_line_points_y.clear() - self._line_plot.set_data([], []) - self.ax_line.relim() - self.ax_line.autoscale_view() - self.canvas_line.draw_idle() - - self.ax_bars.clear() - self._style_axis(self.ax_bars, "Tokens por Fase") - self.canvas_bars.draw_idle() - - def set_usage_totals(self, **kwargs): - pass diff --git a/gui/tabs/postprocess_tab.py b/gui/tabs/postprocess_tab.py deleted file mode 100644 index 02adf52..0000000 --- a/gui/tabs/postprocess_tab.py +++ /dev/null @@ -1,227 +0,0 @@ -"""Pestaña de Post-Procesamiento con agente inteligente.""" -import customtkinter as ctk -from gui.styles import AppStyles - - -class PostProcessTab: - """UI de la pestaña de Post-Procesamiento de CSV.""" - - def __init__(self, parent_tab): - self.tab = parent_tab - self.fonts = AppStyles.get_fonts() - self.colors = AppStyles.get_colors() - self.spacing = AppStyles.get_spacing() - - self._setup_layout() - self._create_file_section() - self._create_preview_section() - self._create_instructions_section() - self._create_code_section() - self._create_log_section() - self._create_result_section() - - # -- Layout ------------------------------------------------------------------ - - def _setup_layout(self): - self.tab.grid_columnconfigure(0, weight=1) - self.tab.grid_rowconfigure(0, weight=1) - - self.scroll = ctk.CTkScrollableFrame(self.tab, fg_color="transparent") - self.scroll.grid(row=0, column=0, sticky="nsew", padx=10, pady=10) - self.scroll.grid_columnconfigure(0, weight=1) - - # -- Sección archivo --------------------------------------------------------- - - def _create_file_section(self): - frame = ctk.CTkFrame(self.scroll) - frame.grid(row=0, column=0, sticky="ew", padx=5, pady=(5, 2)) - frame.grid_columnconfigure(1, weight=1) - - lbl_title = ctk.CTkLabel(frame, text="Archivo CSV", font=self.fonts["section"]) - lbl_title.grid(row=0, column=0, columnspan=3, sticky="w", padx=15, pady=(10, 5)) - - self.btn_select_csv = ctk.CTkButton( - frame, text="Seleccionar CSV", font=self.fonts["ui"], width=160, - ) - self.btn_select_csv.grid(row=1, column=0, padx=15, pady=8, sticky="w") - - self.lbl_csv_path = ctk.CTkLabel( - frame, text="Ningún archivo seleccionado", - font=self.fonts["italic_small"], text_color=self.colors["gray_light"], - ) - self.lbl_csv_path.grid(row=1, column=1, padx=10, pady=8, sticky="w") - - # -- Preview datos ----------------------------------------------------------- - - def _create_preview_section(self): - frame = ctk.CTkFrame(self.scroll) - frame.grid(row=1, column=0, sticky="ew", padx=5, pady=2) - frame.grid_columnconfigure(0, weight=1) - - lbl = ctk.CTkLabel(frame, text="Vista previa (muestra)", font=self.fonts["section"]) - lbl.grid(row=0, column=0, sticky="w", padx=15, pady=(10, 5)) - - self.txt_preview = ctk.CTkTextbox( - frame, height=200, font=self.fonts["mono_small"], - state="disabled", wrap="none", - ) - self.txt_preview.grid(row=1, column=0, sticky="ew", padx=15, pady=(0, 10)) - - # -- Instrucciones ----------------------------------------------------------- - - def _create_instructions_section(self): - frame = ctk.CTkFrame(self.scroll) - frame.grid(row=2, column=0, sticky="ew", padx=5, pady=2) - frame.grid_columnconfigure(0, weight=1) - - lbl = ctk.CTkLabel( - frame, text="Instrucciones de transformación", font=self.fonts["section"], - ) - lbl.grid(row=0, column=0, sticky="w", padx=15, pady=(10, 2)) - - hint = ctk.CTkLabel( - frame, - text="Describe qué cambios quieres aplicar y en qué columnas. Ejemplo: " - "\"En la columna 'monto', multiplicar por 1.19 para agregar IVA\"", - font=self.fonts["italic_small"], text_color=self.colors["gray_light"], - wraplength=900, justify="left", - ) - hint.grid(row=1, column=0, sticky="w", padx=15, pady=(0, 5)) - - self.txt_instructions = ctk.CTkTextbox( - frame, height=90, font=self.fonts["ui"], wrap="word", - ) - self.txt_instructions.grid(row=2, column=0, sticky="ew", padx=15, pady=(0, 5)) - - btn_frame = ctk.CTkFrame(frame, fg_color="transparent") - btn_frame.grid(row=3, column=0, sticky="ew", padx=15, pady=(0, 10)) - - self.btn_run = ctk.CTkButton( - btn_frame, text="Generar y Aplicar", font=self.fonts["button_large"], - fg_color=self.colors["primary_button"], hover_color=self.colors["primary_hover"], - height=38, - ) - self.btn_run.pack(side="left") - - self.lbl_status = ctk.CTkLabel( - btn_frame, text="", font=self.fonts["italic_small"], - text_color=self.colors["gray_light"], - ) - self.lbl_status.pack(side="left", padx=15) - - # -- Código generado --------------------------------------------------------- - - def _create_code_section(self): - frame = ctk.CTkFrame(self.scroll) - frame.grid(row=3, column=0, sticky="ew", padx=5, pady=2) - frame.grid_columnconfigure(0, weight=1) - - lbl = ctk.CTkLabel(frame, text="Código generado", font=self.fonts["section"]) - lbl.grid(row=0, column=0, sticky="w", padx=15, pady=(10, 5)) - - self.txt_code = ctk.CTkTextbox( - frame, height=180, font=self.fonts["mono_small"], - state="disabled", wrap="none", - ) - self.txt_code.grid(row=1, column=0, sticky="ew", padx=15, pady=(0, 10)) - - # -- Log de ejecución -------------------------------------------------------- - - def _create_log_section(self): - frame = ctk.CTkFrame(self.scroll) - frame.grid(row=4, column=0, sticky="ew", padx=5, pady=2) - frame.grid_columnconfigure(0, weight=1) - - lbl = ctk.CTkLabel(frame, text="Log de ejecución", font=self.fonts["section"]) - lbl.grid(row=0, column=0, sticky="w", padx=15, pady=(10, 5)) - - self.txt_log = ctk.CTkTextbox( - frame, height=100, font=self.fonts["mono_small"], - state="disabled", wrap="word", - ) - self.txt_log.grid(row=1, column=0, sticky="ew", padx=15, pady=(0, 10)) - - # -- Resultado --------------------------------------------------------------- - - def _create_result_section(self): - frame = ctk.CTkFrame(self.scroll) - frame.grid(row=5, column=0, sticky="ew", padx=5, pady=(2, 5)) - frame.grid_columnconfigure(0, weight=1) - - top = ctk.CTkFrame(frame, fg_color="transparent") - top.grid(row=0, column=0, sticky="ew", padx=15, pady=(10, 5)) - top.grid_columnconfigure(0, weight=1) - - lbl = ctk.CTkLabel(top, text="Vista previa del resultado", font=self.fonts["section"]) - lbl.pack(side="left") - - self.btn_save = ctk.CTkButton( - top, text="Guardar CSV", font=self.fonts["ui"], width=140, - fg_color=self.colors["success"], hover_color=self.colors["success_hover"], - state="disabled", - ) - self.btn_save.pack(side="right") - - self.txt_result = ctk.CTkTextbox( - frame, height=200, font=self.fonts["mono_small"], - state="disabled", wrap="none", - ) - self.txt_result.grid(row=1, column=0, sticky="ew", padx=15, pady=(0, 10)) - - # -- API pública ------------------------------------------------------------- - - def set_on_select_csv(self, callback): - self.btn_select_csv.configure(command=callback) - - def set_on_run(self, callback): - self.btn_run.configure(command=callback) - - def set_on_save(self, callback): - self.btn_save.configure(command=callback) - - def set_csv_path(self, path: str): - self.lbl_csv_path.configure(text=path or "Ningún archivo seleccionado") - - def get_instructions(self) -> str: - return self.txt_instructions.get("1.0", "end").strip() - - def set_status(self, text: str): - self.lbl_status.configure(text=text) - - def set_preview(self, text: str): - self.txt_preview.configure(state="normal") - self.txt_preview.delete("1.0", "end") - self.txt_preview.insert("1.0", text) - self.txt_preview.configure(state="disabled") - - def set_code(self, text: str): - self.txt_code.configure(state="normal") - self.txt_code.delete("1.0", "end") - self.txt_code.insert("1.0", text) - self.txt_code.configure(state="disabled") - - def append_log(self, text: str): - self.txt_log.configure(state="normal") - self.txt_log.insert("end", text + "\n") - self.txt_log.see("end") - self.txt_log.configure(state="disabled") - - def clear_log(self): - self.txt_log.configure(state="normal") - self.txt_log.delete("1.0", "end") - self.txt_log.configure(state="disabled") - - def set_result(self, text: str): - self.txt_result.configure(state="normal") - self.txt_result.delete("1.0", "end") - self.txt_result.insert("1.0", text) - self.txt_result.configure(state="disabled") - - def enable_save(self, enabled: bool): - self.btn_save.configure(state="normal" if enabled else "disabled") - - def enable_controls(self, enabled: bool): - state = "normal" if enabled else "disabled" - self.btn_select_csv.configure(state=state) - self.btn_run.configure(state=state) - self.txt_instructions.configure(state=state) diff --git a/gui/tabs/preclass_tab.py b/gui/tabs/preclass_tab.py deleted file mode 100644 index 87cb093..0000000 --- a/gui/tabs/preclass_tab.py +++ /dev/null @@ -1,478 +0,0 @@ -"""Pestaña Preclasificación de la aplicación GUI.""" -import customtkinter as ctk -from gui.styles import AppStyles - - -class PreclassificationTab: - """Encapsula la UI de la pestaña Preclasificación.""" - - def __init__(self, parent_tab): - """Inicializa la pestaña de Preclasificación. - - Args: - parent_tab: El CTkFrame que representa la pestaña. - """ - self.tab = parent_tab - self.fonts = AppStyles.get_fonts() - self.colors = AppStyles.get_colors() - self.spacing = AppStyles.get_spacing() - - # Variables internas - self._preclass_pattern_rows = [] - - self._setup_layout() - self._create_root_section() - self._create_options_section() - self._create_target_section() - self._create_patterns_section() - self._create_action_section() - self._create_output_section() - - def _setup_layout(self): - """Configura el layout base de la pestaña.""" - self.tab.grid_columnconfigure(0, weight=1) - self.tab.grid_rowconfigure(0, weight=1) - - self.preclass_container = ctk.CTkScrollableFrame(self.tab, fg_color="transparent") - self.preclass_container.grid(row=0, column=0, sticky="nsew", padx=5, pady=5) - self.preclass_container.grid_columnconfigure(0, weight=1) - self.preclass_container.grid_rowconfigure(3, weight=1) - self.preclass_container.grid_rowconfigure(5, weight=1) - - def _create_root_section(self): - """Crea la sección de carpeta raíz.""" - self.preclass_frame_root = ctk.CTkFrame(self.preclass_container) - self.preclass_frame_root.grid(row=0, column=0, sticky="ew", padx=5, pady=(5, 10)) - self.preclass_frame_root.grid_columnconfigure(1, weight=1) - - lbl_pre_root_title = ctk.CTkLabel( - self.preclass_frame_root, - text="Carpeta Raíz (Preclasificación)", - font=self.fonts["section"] - ) - lbl_pre_root_title.grid(row=0, column=0, columnspan=3, sticky="w", padx=15, pady=(10, 5)) - - self.preclass_root_label = ctk.CTkLabel( - self.preclass_frame_root, - text="Carpeta Raíz:", - font=self.fonts["ui"] - ) - self.preclass_root_label.grid(row=1, column=0, padx=15, pady=(0, 10), sticky="w") - - self.preclass_root_entry = ctk.CTkEntry( - self.preclass_frame_root, - placeholder_text="Selecciona la carpeta raíz para preclasificar...", - font=self.fonts["ui"] - ) - self.preclass_root_entry.grid(row=1, column=1, padx=5, pady=(0, 10), sticky="ew") - - self.preclass_root_button = ctk.CTkButton( - self.preclass_frame_root, - text="Explorar...", - font=self.fonts["ui"], - width=100 - ) - self.preclass_root_button.grid(row=1, column=2, padx=15, pady=(0, 10)) - - # Selector de Modo - self.preclass_mode_var = ctk.StringVar(value="radicados") - - lbl_mode = ctk.CTkLabel(self.preclass_frame_root, text="Estructura:", font=self.fonts["ui"]) - lbl_mode.grid(row=2, column=0, padx=15, pady=(0, 10), sticky="w") - - self.frame_radios = ctk.CTkFrame(self.preclass_frame_root, fg_color="transparent") - self.frame_radios.grid(row=2, column=1, columnspan=2, sticky="w", padx=5, pady=(0, 10)) - - self.radio_radicados = ctk.CTkRadioButton( - self.frame_radios, - text="Por Radicados (Subcarpetas)", - variable=self.preclass_mode_var, - value="radicados", - font=self.fonts["ui"] - ) - self.radio_radicados.pack(side="left", padx=(0, 20)) - - self.radio_general = ctk.CTkRadioButton( - self.frame_radios, - text="Carpeta General (Archivos Planos)", - variable=self.preclass_mode_var, - value="general", - font=self.fonts["ui"] - ) - self.radio_general.pack(side="left") - - def _create_options_section(self): - """Crea la sección de opciones de rango.""" - self.preclass_frame_options = ctk.CTkFrame(self.preclass_container) - self.preclass_frame_options.grid(row=1, column=0, sticky="ew", padx=5, pady=(0, 10)) - self.preclass_frame_options.grid_columnconfigure(1, weight=1) - - self.preclass_range_var = ctk.BooleanVar(value=False) - self.preclass_range_cb = ctk.CTkCheckBox( - self.preclass_frame_options, - text="Habilitar rangos de página", - variable=self.preclass_range_var, - font=self.fonts["ui"], - command=self._on_toggle_preclass_range - ) - self.preclass_range_cb.grid(row=0, column=0, padx=15, pady=10, sticky="w") - - self.preclass_range_inputs_frame = ctk.CTkFrame(self.preclass_frame_options, fg_color="transparent") - - lbl_desde = ctk.CTkLabel(self.preclass_range_inputs_frame, text="Desde:", font=self.fonts["ui"]) - lbl_desde.grid(row=0, column=0, padx=(10, 5), pady=5) - self.preclass_range_start_entry = ctk.CTkEntry(self.preclass_range_inputs_frame, width=60, font=self.fonts["ui"]) - self.preclass_range_start_entry.insert(0, "1") - self.preclass_range_start_entry.grid(row=0, column=1, padx=5, pady=5) - - lbl_hasta = ctk.CTkLabel(self.preclass_range_inputs_frame, text="Hasta:", font=self.fonts["ui"]) - lbl_hasta.grid(row=0, column=2, padx=(10, 5), pady=5) - self.preclass_range_end_entry = ctk.CTkEntry(self.preclass_range_inputs_frame, width=60, font=self.fonts["ui"]) - self.preclass_range_end_entry.insert(0, "1") - self.preclass_range_end_entry.grid(row=0, column=3, padx=5, pady=5) - - def _create_target_section(self): - """Crea la sección de archivo objetivo.""" - self.preclass_frame_target = ctk.CTkFrame(self.preclass_container) - self.preclass_frame_target.grid(row=2, column=0, sticky="ew", padx=5, pady=(0, 10)) - self.preclass_frame_target.grid_columnconfigure(0, weight=1) - - self.preclass_target_var = ctk.BooleanVar(value=False) - self.preclass_target_cb = ctk.CTkCheckBox( - self.preclass_frame_target, - text="Habilitar archivo objetivo", - variable=self.preclass_target_var, - font=self.fonts["ui"], - command=self._on_toggle_preclass_target - ) - self.preclass_target_cb.grid(row=0, column=0, padx=15, pady=10, sticky="w") - - self.preclass_target_inputs_frame = ctk.CTkFrame(self.preclass_frame_target, fg_color="transparent") - - lbl_target_def = ctk.CTkLabel(self.preclass_target_inputs_frame, text="Def. Tipo Doc:", font=self.fonts["ui"]) - lbl_target_def.grid(row=0, column=0, padx=(15, 5), pady=5, sticky="w") - self.preclass_target_def_entry = ctk.CTkEntry( - self.preclass_target_inputs_frame, - width=250, - font=self.fonts["ui"], - placeholder_text="Ej: Factura de Venta" - ) - self.preclass_target_def_entry.grid(row=0, column=1, padx=5, pady=5, sticky="w") - - lbl_target_pattern = ctk.CTkLabel(self.preclass_target_inputs_frame, text="Patrón:", font=self.fonts["ui"]) - lbl_target_pattern.grid(row=0, column=2, padx=(20, 5), pady=5, sticky="w") - self.preclass_target_pattern_entry = ctk.CTkEntry( - self.preclass_target_inputs_frame, - width=200, - font=self.fonts["ui"], - placeholder_text="Ej: *_FAC_*.pdf" - ) - self.preclass_target_pattern_entry.grid(row=0, column=3, padx=5, pady=5, sticky="w") - - def _create_patterns_section(self): - """Crea la sección de patrones.""" - self.preclass_frame_patterns = ctk.CTkFrame(self.preclass_container) - self.preclass_frame_patterns.grid(row=3, column=0, sticky="nsew", padx=5, pady=5) - self.preclass_frame_patterns.grid_columnconfigure(0, weight=1) - - self.preclass_patterns_header = ctk.CTkFrame(self.preclass_frame_patterns, fg_color="transparent") - self.preclass_patterns_header.grid(row=0, column=0, sticky="ew", padx=10, pady=(10, 2)) - self.preclass_patterns_header.grid_columnconfigure(0, weight=1) - - self.preclass_patterns_label = ctk.CTkLabel( - self.preclass_patterns_header, - text="Patrones", - font=self.fonts["section"] - ) - self.preclass_patterns_label.grid(row=0, column=0, sticky="w") - - self.add_preclass_pattern_btn = ctk.CTkButton( - self.preclass_patterns_header, - text="+ Añadir Patrón", - font=self.fonts["ui"], - width=140, - height=28 - ) - self.add_preclass_pattern_btn.grid(row=0, column=1, sticky="e") - self.add_preclass_pattern_btn.configure(command=self._add_preclass_pattern_row) - - self.preclass_patterns_hint = ctk.CTkLabel( - self.preclass_frame_patterns, - text="Agrega patrones de texto. Cada patrón puede ser solo 'key' o 'key + value'.", - font=self.fonts["italic_small"], - text_color=self.colors["gray_light"], - justify="left" - ) - self.preclass_patterns_hint.grid(row=1, column=0, sticky="w", padx=15, pady=(0, 5)) - - self.preclass_patterns_list = ctk.CTkFrame(self.preclass_frame_patterns, fg_color="transparent") - self.preclass_patterns_list.grid(row=2, column=0, sticky="ew", padx=10, pady=(2, 10)) - self.preclass_patterns_list.grid_columnconfigure(0, weight=1) - - # Fila por defecto (vacía) - self._add_preclass_pattern_row() - - def _create_action_section(self): - """Crea la sección de acciones.""" - self.preclass_action_frame = ctk.CTkFrame(self.preclass_container, fg_color="transparent") - self.preclass_action_frame.grid(row=4, column=0, sticky="ew", padx=5, pady=10) - self.preclass_action_frame.grid_columnconfigure(0, weight=1) - - self.preclass_run_btn = ctk.CTkButton( - self.preclass_action_frame, - text="EJECUTAR PRECLASIFICACIÓN", - font=self.fonts["bold_large"], - height=40, - fg_color=self.colors["secondary_button"], - hover_color=self.colors["secondary_hover"] - ) - self.preclass_run_btn.pack(side="left", fill="x", expand=True, padx=10) - - self.preclass_export_btn = ctk.CTkButton( - self.preclass_action_frame, - text="Exportar Excel", - font=self.fonts["ui"], - height=40, - state="disabled", - fg_color=self.colors["success"], - hover_color=self.colors["success_hover"] - ) - self.preclass_export_btn.pack(side="left", padx=(0, 10)) - - self.preclass_organize_btn = ctk.CTkButton( - self.preclass_action_frame, - text="Organizar Carpetas", - font=self.fonts["ui"], - height=40, - state="disabled", - fg_color=self.colors["warning"], - hover_color=self.colors["warning_hover"] - ) - self.preclass_organize_btn.pack(side="left", padx=(0, 10)) - - def _create_output_section(self): - """Crea la sección de salida/logs.""" - self.preclass_log_frame = ctk.CTkFrame(self.preclass_container) - self.preclass_log_frame.grid(row=5, column=0, sticky="nsew", padx=5, pady=(0, 10)) - self.preclass_log_frame.grid_columnconfigure(0, weight=1) - self.preclass_log_frame.grid_rowconfigure(1, weight=1) - - lbl_pre_logs = ctk.CTkLabel( - self.preclass_log_frame, - text="Resultados:", - font=self.fonts["ui"], - text_color=self.colors["text_muted"] - ) - lbl_pre_logs.grid(row=0, column=0, sticky="w", padx=10, pady=(5, 0)) - - self.preclass_output = ctk.CTkTextbox( - self.preclass_log_frame, - font=self.fonts["mono"], - activate_scrollbars=True - ) - self.preclass_output.grid(row=1, column=0, padx=5, pady=5, sticky="nsew") - - def _add_preclass_pattern_row(self, key: str = "", mode: str = "key", value: str = "") -> None: - """Añade una fila editable para un patrón de preclasificación.""" - row = ctk.CTkFrame(self.preclass_patterns_list, fg_color="transparent") - row.pack(fill="x", pady=2) - - grid_inner = ctk.CTkFrame(row, fg_color="transparent") - grid_inner.pack(fill="x", padx=5, pady=2) - grid_inner.grid_columnconfigure(0, weight=1) - grid_inner.grid_columnconfigure(2, weight=1) - - key_entry = ctk.CTkEntry(grid_inner, placeholder_text="Patrón / Key", height=28, font=self.fonts["ui"]) - if key: - key_entry.insert(0, key) - key_entry.grid(row=0, column=0, padx=(0, 5), sticky="ew") - - mode_combo = ctk.CTkComboBox( - grid_inner, - values=["key", "key+value"], - width=110, - font=self.fonts["ui"] - ) - mode_combo.grid(row=0, column=1, padx=(0, 5), sticky="w") - - value_entry = ctk.CTkEntry( - grid_inner, - placeholder_text="Value (si aplica)", - height=28, - font=self.fonts["ui"] - ) - if value: - value_entry.insert(0, value) - value_entry.grid(row=0, column=2, padx=(0, 5), sticky="ew") - - def _apply_mode(m: str): - m_norm = (m or "").strip().lower() - if m_norm in ("key+value", "key + value", "key_value", "key-value"): - value_entry.configure(state="normal", placeholder_text="Value (requerido)") - else: - value_entry.delete(0, "end") - value_entry.configure(state="disabled", placeholder_text="Value (no aplica)") - - def _on_mode_change(choice: str): - _apply_mode(choice) - - try: - mode_combo.set(mode if mode in ("key", "key+value") else "key") - except Exception: - mode_combo.set("key") - - mode_combo.configure(command=_on_mode_change) - _apply_mode(mode_combo.get()) - - def _remove(): - try: - for i, r_data in enumerate(list(self._preclass_pattern_rows)): - if r_data.get("frame") == row: - self._preclass_pattern_rows.pop(i) - break - except Exception: - pass - row.destroy() - - remove_btn = ctk.CTkButton( - grid_inner, - text="×", - width=30, - height=28, - fg_color=self.colors["danger"], - hover_color=self.colors["danger_hover"], - command=_remove - ) - remove_btn.grid(row=0, column=3, sticky="e") - - self._preclass_pattern_rows.append({ - "frame": row, - "key_entry": key_entry, - "mode_combo": mode_combo, - "value_entry": value_entry, - "remove_btn": remove_btn, - }) - - def _on_toggle_preclass_range(self): - """Muestra u oculta los inputs de rango según el checkbox.""" - if self.preclass_range_var.get(): - self.preclass_range_inputs_frame.grid(row=0, column=1, padx=10, pady=0, sticky="w") - else: - self.preclass_range_inputs_frame.grid_forget() - - def _on_toggle_preclass_target(self): - """Muestra u oculta los inputs de archivo objetivo según el checkbox.""" - if self.preclass_target_var.get(): - self.preclass_target_inputs_frame.grid(row=1, column=0, padx=10, pady=(0, 10), sticky="w") - else: - self.preclass_target_inputs_frame.grid_forget() - - # --- API Pública --- - - def set_on_run(self, callback): - """Vincula callback al botón de ejecutar.""" - self.preclass_run_btn.configure(command=callback) - - def set_on_export_excel(self, callback): - """Vincula callback al botón de exportar Excel.""" - self.preclass_export_btn.configure(command=callback) - - def set_on_organize_files(self, callback): - """Vincula callback al botón de organizar carpetas.""" - self.preclass_organize_btn.configure(command=callback) - - def set_on_select_root(self, callback): - """Vincula callback al botón de seleccionar carpeta.""" - self.preclass_root_button.configure(command=callback) - - def get_root_path(self) -> str: - """Devuelve la ruta raíz de preclasificación.""" - return self.preclass_root_entry.get().strip() - - def set_root_path(self, path: str) -> None: - """Establece la ruta raíz de preclasificación.""" - self.preclass_root_entry.configure(state="normal") - self.preclass_root_entry.delete(0, "end") - self.preclass_root_entry.insert(0, path) - - def get_mode(self) -> str: - """Devuelve el modo de preclasificación: 'radicados' o 'general'.""" - try: - v = (self.preclass_mode_var.get() or "").strip().lower() - return v if v in ("radicados", "general") else "radicados" - except Exception: - return "radicados" - - def get_range(self) -> tuple[bool, int, int]: - """Devuelve (habilitado, desde, hasta) para preclasificación.""" - enabled = self.preclass_range_var.get() - try: - start = int(self.preclass_range_start_entry.get().strip() or "1") - end = int(self.preclass_range_end_entry.get().strip() or "1") - except Exception: - start, end = 1, 1 - return enabled, start, end - - def get_target_info(self) -> dict: - """Devuelve la configuración del archivo objetivo.""" - return { - "enabled": self.preclass_target_var.get(), - "definition": self.preclass_target_def_entry.get().strip(), - "pattern": self.preclass_target_pattern_entry.get().strip() - } - - def get_patterns(self) -> list[dict]: - """Devuelve la lista de patrones definida.""" - out = [] - for it in getattr(self, "_preclass_pattern_rows", []): - try: - key = it["key_entry"].get().strip() - mode = (it["mode_combo"].get() or "").strip() - value = it["value_entry"].get().strip() - if not key: - continue - if mode == "key+value": - out.append({"key": key, "mode": "key+value", "value": value}) - else: - out.append({"key": key, "mode": "key"}) - except Exception: - pass - return out - - def append_output(self, text: str) -> None: - """Añade texto al área de salida.""" - self.preclass_output.insert("end", text + "\n") - self.preclass_output.see("end") - - def append_output_safe(self, text: str) -> None: - """Añade texto al área de salida de forma thread-safe.""" - self.append_output(text) - - def enable_actions(self, enabled: bool): - """Habilita o deshabilita los botones de acción.""" - state = "normal" if enabled else "disabled" - self.preclass_export_btn.configure(state=state) - self.preclass_organize_btn.configure(state=state) - - def enable_controls(self, enabled: bool): - """Habilita o deshabilita los controles principales.""" - state = "normal" if enabled else "disabled" - - self.preclass_root_button.configure(state=state) - self.preclass_root_entry.configure(state=state) - self.preclass_range_cb.configure(state=state) - self.preclass_range_start_entry.configure(state=state) - self.preclass_range_end_entry.configure(state=state) - self.preclass_target_cb.configure(state=state) - self.preclass_target_def_entry.configure(state=state) - self.preclass_target_pattern_entry.configure(state=state) - self.radio_radicados.configure(state=state) - self.radio_general.configure(state=state) - self.add_preclass_pattern_btn.configure(state=state) - self.preclass_run_btn.configure(state=state) - - for row in self._preclass_pattern_rows: - row["key_entry"].configure(state=state) - row["mode_combo"].configure(state=state) - row["value_entry"].configure(state=state) - row["remove_btn"].configure(state=state) diff --git a/gui/tabs/results_tab.py b/gui/tabs/results_tab.py deleted file mode 100644 index f1b1206..0000000 --- a/gui/tabs/results_tab.py +++ /dev/null @@ -1,304 +0,0 @@ -"""Pestaña de Resultados de la aplicación GUI.""" -import os -import tkinter as tk -import customtkinter as ctk -import tkinter.ttk as ttk -from gui.styles import AppStyles - - -class ResultsTab: - """Encapsula la UI de la pestaña Resultados.""" - - def __init__(self, parent_tab): - """Inicializa la pestaña de Resultados. - - Args: - parent_tab: El CTkFrame que representa la pestaña. - """ - self.tab = parent_tab - self.fonts = AppStyles.get_fonts() - self.colors = AppStyles.get_colors() - self.spacing = AppStyles.get_spacing() - - self._setup_layout() - self._create_subtabs() - self._setup_treeview_style() - self._create_extractor_tab() - self._create_preclass_results_tab() - - def _setup_layout(self): - """Configura el layout base de la pestaña.""" - self.tab.grid_columnconfigure(0, weight=1) - self.tab.grid_rowconfigure(0, weight=1) - - self.tabs_res = ctk.CTkTabview(self.tab) - self.tabs_res.grid(row=0, column=0, sticky="nsew", padx=10, pady=10) - - def _create_subtabs(self): - """Crea las sub-pestañas de Resultados.""" - self.tab_res_extractor = self.tabs_res.add("Extractor (Próximamente)") - self.tab_res_preclass = self.tabs_res.add("Preclasificación") - - def _setup_treeview_style(self): - """Configura el estilo de los treeviews.""" - style = ttk.Style() - style.theme_use("clam") - style.configure("Treeview", rowheight=25) - - def _create_extractor_tab(self): - """Crea la sub-pestaña del Extractor (en construcción).""" - lbl_const = ctk.CTkLabel( - self.tab_res_extractor, - text="En construcción...", - font=ctk.CTkFont(size=20) - ) - lbl_const.place(relx=0.5, rely=0.5, anchor="center") - - def _create_preclass_results_tab(self): - """Crea la sub-pestaña de resultados de preclasificación.""" - self.tab_res_preclass.grid_columnconfigure(0, weight=1) - self.tab_res_preclass.grid_rowconfigure(1, weight=1) - - self._create_filters_section() - self._create_tables_section() - self._create_actions_section() - - def _create_filters_section(self): - """Crea la sección de filtros.""" - self.frame_res_filters = ctk.CTkFrame(self.tab_res_preclass) - self.frame_res_filters.grid(row=0, column=0, sticky="ew", padx=10, pady=5) - - ctk.CTkLabel(self.frame_res_filters, text="Run ID:").grid(row=0, column=0, padx=5, pady=5) - self.combo_res_run = ctk.CTkComboBox(self.frame_res_filters, values=["Todos"], width=150) - self.combo_res_run.grid(row=0, column=1, padx=5, pady=5) - - ctk.CTkLabel(self.frame_res_filters, text="Estado:").grid(row=0, column=2, padx=5, pady=5) - self.combo_res_status = ctk.CTkComboBox( - self.frame_res_filters, - values=["Todos", "MATCH", "NO_MATCH", "ERROR", "NO_FILES"], - width=120 - ) - self.combo_res_status.grid(row=0, column=3, padx=5, pady=5) - - self.entry_res_search = ctk.CTkEntry( - self.frame_res_filters, - placeholder_text="Buscar radicado/archivo...", - width=200 - ) - self.entry_res_search.grid(row=0, column=4, padx=5, pady=5) - - self.btn_res_search = ctk.CTkButton(self.frame_res_filters, text="Buscar", width=100) - self.btn_res_search.grid(row=0, column=5, padx=10, pady=5) - - self.lbl_res_priority = ctk.CTkLabel( - self.frame_res_filters, - text=( - "Filtros estrictos: se combinan (AND) con importancia: " - "Run > estado (MATCH, etc.) > criterio de búsqueda. " - "Puede ampliar la búsqueda (sin tocar el run) abajo." - ), - text_color=("#5C6370", "#9CA3AF"), - font=ctk.CTkFont(size=12), - anchor="w", - wraplength=920, - ) - self.lbl_res_priority.grid( - row=1, column=0, columnspan=6, padx=5, pady=(4, 0), sticky="w" - ) - self._res_relax_var = tk.BooleanVar(value=False) - self.chk_res_relax = ctk.CTkCheckBox( - self.frame_res_filters, - text="Si 0 resultados, ampliar: quitar criterio de búsqueda, luego estado (el run fijado no se descarta).", - font=ctk.CTkFont(size=12), - variable=self._res_relax_var, - ) - self.chk_res_relax.grid( - row=2, column=0, columnspan=6, padx=5, pady=4, sticky="w" - ) - self.lbl_res_filter_note = ctk.CTkLabel( - self.frame_res_filters, - text="", - text_color=("#1D4ED8", "#60A5FA"), - font=ctk.CTkFont(size=12), - anchor="w", - wraplength=920, - ) - self.lbl_res_filter_note.grid( - row=3, column=0, columnspan=6, padx=5, pady=(0, 4), sticky="w" - ) - - def _create_tables_section(self): - """Crea la sección con las tablas (principal y de detalle).""" - self.frame_res_table = ctk.CTkFrame(self.tab_res_preclass, fg_color="transparent") - self.frame_res_table.grid(row=1, column=0, sticky="nsew", padx=10, pady=5) - self.frame_res_table.grid_columnconfigure(0, weight=1) - self.frame_res_table.grid_rowconfigure(0, weight=3) - self.frame_res_table.grid_rowconfigure(2, weight=2) - - self._create_main_results_table() - self._create_detail_label() - self._create_matches_table() - - def _create_main_results_table(self): - """Crea la tabla principal de resultados.""" - cols = ("ID", "Run", "Fecha", "Radicado", "Estado", "Archivo") - self.tree_results = ttk.Treeview( - self.frame_res_table, - columns=cols, - show="headings", - selectmode="browse" - ) - - for c in cols: - self.tree_results.heading(c, text=c) - self.tree_results.column(c, width=100) - - self.tree_results.column("Archivo", width=300) - self.tree_results.column("ID", width=50) - self.tree_results.column("Run", width=50) - - # Scrollbars - sb_y = ttk.Scrollbar(self.frame_res_table, orient="vertical", command=self.tree_results.yview) - sb_x = ttk.Scrollbar(self.frame_res_table, orient="horizontal", command=self.tree_results.xview) - self.tree_results.configure(yscrollcommand=sb_y.set, xscrollcommand=sb_x.set) - - self.tree_results.grid(row=0, column=0, sticky="nsew") - sb_y.grid(row=0, column=1, sticky="ns") - sb_x.grid(row=1, column=0, sticky="ew") - - def _create_detail_label(self): - """Crea la etiqueta de la sección de detalle.""" - lbl_detail = ctk.CTkLabel( - self.frame_res_table, - text="Detalle de Matches (Seleccionar fila arriba)", - font=ctk.CTkFont(size=12, weight="bold") - ) - lbl_detail.grid(row=1, column=0, sticky="w", pady=(10, 5)) - - def _create_matches_table(self): - """Crea la tabla de detalle de matches.""" - cols_detail = ("Key", "Valor Encontrado", "Existe", "Valor Esperado") - self.tree_matches = ttk.Treeview( - self.frame_res_table, - columns=cols_detail, - show="headings", - selectmode="browse", - height=5 - ) - - for c in cols_detail: - self.tree_matches.heading(c, text=c) - self.tree_matches.column(c, width=150) - - sb_detail_y = ttk.Scrollbar(self.frame_res_table, orient="vertical", command=self.tree_matches.yview) - self.tree_matches.configure(yscrollcommand=sb_detail_y.set) - - self.tree_matches.grid(row=2, column=0, sticky="nsew") - sb_detail_y.grid(row=2, column=1, sticky="ns") - - def _create_actions_section(self): - """Crea la sección de acciones.""" - self.frame_res_actions = ctk.CTkFrame(self.tab_res_preclass) - self.frame_res_actions.grid(row=2, column=0, sticky="ew", padx=10, pady=10) - - self.btn_res_export = ctk.CTkButton( - self.frame_res_actions, - text="Exportar Vista a Excel", - fg_color=self.colors["success"] - ) - self.btn_res_export.pack(side="right", padx=10, pady=5) - - self.btn_res_organize = ctk.CTkButton( - self.frame_res_actions, - text="Organizar Carpetas (Run)", - fg_color=self.colors["warning"], - hover_color=self.colors["warning_hover"] - ) - self.btn_res_organize.pack(side="right", padx=10, pady=5) - - self.btn_res_refresh = ctk.CTkButton(self.frame_res_actions, text="Refrescar Runs") - self.btn_res_refresh.pack(side="left", padx=10, pady=5) - - # --- API Pública --- - - def set_on_search(self, callback): - """Vincula el callback al botón de búsqueda.""" - self.btn_res_search.configure(command=callback) - - def set_on_export(self, callback): - """Vincula el callback al botón de exportación.""" - self.btn_res_export.configure(command=callback) - - def set_on_refresh(self, callback): - """Vincula el callback al botón de refrescar.""" - self.btn_res_refresh.configure(command=callback) - - def set_on_organize(self, callback): - """Vincula el callback al botón de organizar.""" - self.btn_res_organize.configure(command=callback) - - def set_on_select_result(self, callback): - """Vincula el callback al evento de selección de fila.""" - self.tree_results.bind("<>", callback) - - def get_selected_result_values(self) -> tuple | None: - """Devuelve los valores de la fila seleccionada o None si no hay selección.""" - selection = self.tree_results.selection() - if not selection: - return None - return self.tree_results.item(selection[0], "values") or None - - def update_matches_table(self, rows): - """Actualiza la tabla de matches con los datos proporcionados.""" - for item in self.tree_matches.get_children(): - self.tree_matches.delete(item) - - for r in rows: - exists_str = "✅" if r.get("exists_bool") else "❌" - vals = (r.get("key"), r.get("found_value"), exists_str, r.get("expected_value") or "-") - self.tree_matches.insert("", "end", values=vals) - - def update_runs_combo(self, runs: list): - """Actualiza el combo box con los IDs de ejecuciones disponibles.""" - vals = ["Todos"] + [str(r) for r in runs] - self.combo_res_run.configure(values=vals) - self.combo_res_run.set("Todos") - - def set_run_id(self, run_id: int | str): - """Establece el ID de ejecución seleccionado.""" - try: - self.combo_res_run.set(str(run_id)) - except Exception: - pass - - def get_filters(self) -> dict: - """Devuelve los filtros actuales.""" - return { - "run_id": self.combo_res_run.get(), - "status": self.combo_res_status.get(), - "text": self.entry_res_search.get(), - "relax_on_empty": bool(self._res_relax_var.get()), - } - - def set_filter_note(self, text: str) -> None: - """Mensaje bajo ampliar (p. ej. qué criterio se relajó). Texto vacío la limpia.""" - self.lbl_res_filter_note.configure(text=(text or "").strip()) - - def clear_filter_note(self) -> None: - self.set_filter_note("") - - def update_results_table(self, rows): - """Actualiza la tabla de resultados con los datos proporcionados.""" - for item in self.tree_results.get_children(): - self.tree_results.delete(item) - - for r in rows: - vals = ( - r.get("id"), - r.get("run_id"), - r.get("timestamp"), - r.get("radicado"), - r.get("status"), - os.path.basename(r.get("file_path") or "") - ) - self.tree_results.insert("", "end", values=vals) diff --git a/gui/tabs/settings_tab.py b/gui/tabs/settings_tab.py deleted file mode 100644 index 4ff24e6..0000000 --- a/gui/tabs/settings_tab.py +++ /dev/null @@ -1,314 +0,0 @@ -"""Pestaña de Configuración de la aplicación GUI.""" -import customtkinter as ctk -from gui.styles import AppStyles -from application.app_prefs import ( - format_moderadora_text_from_prefs, - get_allowed_document_types, - load_cop_mod_preferences, - normalize_document_types, - parse_moderadora_text, - default_moderadora_by_year_payload, -) - - -class SettingsTab: - """Encapsula la UI de la pestaña Configuración.""" - - def __init__(self, parent_tab): - """Inicializa la pestaña de Configuración. - - Args: - parent_tab: El CTkFrame que representa la pestaña. - """ - self.tab = parent_tab - self.fonts = AppStyles.get_fonts() - self.colors = AppStyles.get_colors() - self.spacing = AppStyles.get_spacing() - - self._setup_layout() - self._create_api_section() - self._create_processing_section() - self._create_document_types_section() - self._create_cop_mod_section() - self._create_save_button() - - def _setup_layout(self): - """Configura el layout: área con scroll y barra inferior fija con Guardar.""" - self.tab.grid_columnconfigure(0, weight=1) - self.tab.grid_rowconfigure(0, weight=1) - self.tab.grid_rowconfigure(1, weight=0) - - self.settings_container = ctk.CTkScrollableFrame( - self.tab, - fg_color="transparent", - ) - self.settings_container.grid(row=0, column=0, sticky="nsew", padx=20, pady=(20, 8)) - self.settings_container.grid_columnconfigure(0, weight=1) - - self.save_footer = ctk.CTkFrame(self.tab, fg_color="transparent") - self.save_footer.grid(row=1, column=0, sticky="ew", padx=20, pady=(4, 16)) - self.save_footer.grid_columnconfigure(0, weight=1) - - def _create_api_section(self): - """Crea la sección de configuración del modelo LLM.""" - self.frame_api = ctk.CTkFrame(self.settings_container) - self.frame_api.grid(row=0, column=0, sticky="ew", padx=5, pady=5) - self.frame_api.grid_columnconfigure(1, weight=1) - - # Título - lbl_api_title = ctk.CTkLabel( - self.frame_api, - text="Configuración del Modelo LLM", - font=self.fonts["section"] - ) - lbl_api_title.grid(row=0, column=0, columnspan=2, sticky="w", padx=15, pady=10) - - # Selector de modelo - lbl_model = ctk.CTkLabel( - self.frame_api, - text="Modelo (extracción):", - font=self.fonts["ui"] - ) - lbl_model.grid(row=1, column=0, padx=15, pady=10, sticky="w") - - self.combo_model = ctk.CTkComboBox( - self.frame_api, - values=[ - "gpt-5.4", - "gpt-5.4-mini", - "gpt-5.4-nano", - "gpt-4.1-mini", - "grok-4.3", - ], - width=250, - font=self.fonts["ui"] - ) - self.combo_model.grid(row=1, column=1, padx=15, pady=10, sticky="w") - self.combo_model.set("gpt-5.4") - - # Texto de ayuda - help_text = ( - "ℹ gpt-5.4: Modelo predeterminado (OpenAI / compatible)\n" - "ℹ gpt-5.4-mini / gpt-4.1-mini: opciones más rápidas\n" - "ℹ gpt-5.4-nano: opción más económica\n" - "ℹ grok-4.3: API xAI (requiere GROK_API_KEY en .env)" - ) - lbl_help = ctk.CTkLabel( - self.frame_api, - text=help_text, - font=self.fonts["italic_small"], - text_color=self.colors["gray_light"], - justify="left" - ) - lbl_help.grid(row=2, column=1, padx=15, pady=(0, 10), sticky="w") - - def _create_processing_section(self): - """Crea la sección de procesamiento y rendimiento.""" - self.frame_proc = ctk.CTkFrame(self.settings_container) - self.frame_proc.grid(row=1, column=0, sticky="ew", padx=5, pady=5) - self.frame_proc.grid_columnconfigure(1, weight=1) - - # Título - lbl_proc_title = ctk.CTkLabel( - self.frame_proc, - text="Procesamiento y Rendimiento", - font=self.fonts["section"] - ) - lbl_proc_title.grid(row=0, column=0, columnspan=2, sticky="w", padx=15, pady=10) - - # Límite de concurrencia - lbl_conc = ctk.CTkLabel( - self.frame_proc, - text="Límite de Concurrencia:", - font=self.fonts["ui"] - ) - lbl_conc.grid(row=1, column=0, padx=15, pady=10, sticky="w") - - self.concurrency_var = ctk.StringVar(value="100") - self.concurrency_entry = ctk.CTkEntry( - self.frame_proc, - textvariable=self.concurrency_var, - width=100, - font=self.fonts["ui"] - ) - self.concurrency_entry.grid(row=1, column=1, padx=15, pady=10, sticky="w") - - # Advertencia - warning_text = ( - "⚠️ Advertencia: Entre mayor el número, mayor la probabilidad de Rate Limits (429)." - ) - lbl_conc_warn = ctk.CTkLabel( - self.frame_proc, - text=warning_text, - font=self.fonts["italic_small"], - text_color="orange" - ) - lbl_conc_warn.grid(row=2, column=0, columnspan=2, padx=15, pady=(0, 10), sticky="w") - - def _create_document_types_section(self): - """Crea la sección editable de tipos documentales permitidos.""" - self.frame_doc_types = ctk.CTkFrame(self.settings_container) - self.frame_doc_types.grid(row=2, column=0, sticky="ew", padx=5, pady=5) - self.frame_doc_types.grid_columnconfigure(0, weight=1) - - lbl_title = ctk.CTkLabel( - self.frame_doc_types, - text="Tipos de Documento Permitidos", - font=self.fonts["section"] - ) - lbl_title.grid(row=0, column=0, sticky="w", padx=15, pady=(10, 4)) - - lbl_help = ctk.CTkLabel( - self.frame_doc_types, - text="Escríbelos separados por comas. Se usarán en documento_forzate. NIT se ignorará automáticamente.", - font=self.fonts["italic_small"], - text_color=self.colors["gray_light"], - justify="left" - ) - lbl_help.grid(row=1, column=0, sticky="w", padx=15, pady=(0, 6)) - - self.allowed_doc_types_entry = ctk.CTkEntry( - self.frame_doc_types, - font=self.fonts["ui"], - placeholder_text="CC, TI, CE, RC, PA, PPT, PEP, EPT, NUIP" - ) - self.allowed_doc_types_entry.grid(row=2, column=0, sticky="ew", padx=15, pady=(0, 10)) - self.set_allowed_document_types(list(get_allowed_document_types())) - - def _create_cop_mod_section(self): - """Cuotas moderadora por año (regla Separa COP/MOD).""" - self.frame_cop_mod = ctk.CTkFrame(self.settings_container) - self.frame_cop_mod.grid(row=3, column=0, sticky="ew", padx=5, pady=5) - self.frame_cop_mod.grid_columnconfigure(0, weight=1) - - lbl_title = ctk.CTkLabel( - self.frame_cop_mod, - text="Cuota moderadora (COP/MOD)", - font=self.fonts["section"], - ) - lbl_title.grid(row=0, column=0, sticky="w", padx=15, pady=(10, 4)) - - lbl_help = ctk.CTkLabel( - self.frame_cop_mod, - text=( - "Una línea por año: 2024: 4500, 18200, 47700. " - "La columna «Fecha Servicio» se elige en Proceso de Limpieza al activar la regla." - ), - font=self.fonts["italic_small"], - text_color=self.colors["gray_light"], - justify="left", - ) - lbl_help.grid(row=1, column=0, sticky="w", padx=15, pady=(0, 6)) - - merged, fecha_def = load_cop_mod_preferences() - initial_txt = format_moderadora_text_from_prefs(merged) - - self.moderadora_text = ctk.CTkTextbox( - self.frame_cop_mod, - font=self.fonts["mono_small"], - height=140, - ) - self.moderadora_text.grid(row=2, column=0, sticky="ew", padx=15, pady=(0, 8)) - self.moderadora_text.insert("1.0", initial_txt) - - row_fe = ctk.CTkFrame(self.frame_cop_mod, fg_color="transparent") - row_fe.grid(row=3, column=0, sticky="ew", padx=15, pady=(0, 6)) - row_fe.grid_columnconfigure(1, weight=1) - - ctk.CTkLabel( - row_fe, - text="Sugerencia nombre columna fecha (opcional):", - font=self.fonts["ui"], - ).grid(row=0, column=0, padx=(0, 10), sticky="w") - - self.fecha_column_default_var = ctk.StringVar(value=fecha_def) - self.fecha_column_default_entry = ctk.CTkEntry( - row_fe, - textvariable=self.fecha_column_default_var, - font=self.fonts["ui"], - placeholder_text="Fecha Servicio", - ) - self.fecha_column_default_entry.grid(row=0, column=1, sticky="ew") - - self.btn_moderadora_defaults = ctk.CTkButton( - self.frame_cop_mod, - text="Restaurar valores por defecto", - font=self.fonts["ui"], - command=self._on_restore_moderadora_defaults, - ) - self.btn_moderadora_defaults.grid(row=4, column=0, sticky="w", padx=15, pady=(0, 12)) - - def _on_restore_moderadora_defaults(self) -> None: - txt = format_moderadora_text_from_prefs(default_moderadora_by_year_payload()) - self.moderadora_text.delete("1.0", "end") - self.moderadora_text.insert("1.0", txt) - - def _create_save_button(self): - """Barra inferior fija: siempre visible aunque el formulario sea largo.""" - self.btn_save_settings = ctk.CTkButton( - self.save_footer, - text="Guardar cambios", - font=self.fonts["ui"], - height=38, - ) - self.btn_save_settings.grid(row=0, column=0, sticky="ew") - - # --- API Pública --- - - def set_on_save(self, callback): - """Vincula el callback al botón de guardar.""" - self.btn_save_settings.configure(command=callback) - - def get_selected_model(self) -> str: - """Devuelve el modelo seleccionado.""" - return self.combo_model.get() - - def set_selected_model(self, model: str) -> None: - """Establece el modelo seleccionado.""" - self.combo_model.set(model) - - def get_concurrency_limit(self) -> int: - """Devuelve el límite de concurrencia.""" - try: - return int(self.concurrency_var.get()) - except Exception: - return 20 - - def set_concurrency_limit(self, limit: int) -> None: - """Establece el límite de concurrencia.""" - self.concurrency_var.set(str(limit)) - - def get_allowed_document_types(self) -> list[str]: - """Devuelve la lista editable de tipos documentales válidos.""" - raw = self.allowed_doc_types_entry.get().strip() - values = [part.strip() for part in raw.split(",")] - return normalize_document_types(values) - - def set_allowed_document_types(self, values: list[str]) -> None: - """Establece la lista de tipos documentales válidos.""" - self.allowed_doc_types_entry.delete(0, "end") - self.allowed_doc_types_entry.insert(0, ", ".join(normalize_document_types(values))) - - def get_moderadora_cuotas_text(self) -> str: - """Texto multilínea año: valores.""" - return self.moderadora_text.get("1.0", "end").strip() - - def set_moderadora_cuotas_ui(self, payload: dict[str, list[int]], fecha_hint: str) -> None: - """Establece cuotas y sugerencia de columna fecha desde prefs.""" - self.moderadora_text.delete("1.0", "end") - self.moderadora_text.insert("1.0", format_moderadora_text_from_prefs(payload)) - self.fecha_column_default_var.set(fecha_hint or "") - - def get_fecha_servicio_column_hint(self) -> str: - return str(self.fecha_column_default_var.get() or "").strip() - - def enable_controls(self, enabled: bool): - """Habilita o deshabilita los controles.""" - state = "normal" if enabled else "disabled" - self.combo_model.configure(state=state) - self.concurrency_entry.configure(state=state) - self.allowed_doc_types_entry.configure(state=state) - self.moderadora_text.configure(state=state) - self.fecha_column_default_entry.configure(state=state) - self.btn_moderadora_defaults.configure(state=state) - self.btn_save_settings.configure(state=state) diff --git a/gui/view.py b/gui/view.py deleted file mode 100644 index 36d4cd9..0000000 --- a/gui/view.py +++ /dev/null @@ -1,418 +0,0 @@ -"""Vista principal de la aplicación GUI con `customtkinter`. - -Expone métodos para enlazar callbacks, leer entradas y escribir salida en un -área de texto, además de habilitar/deshabilitar controles durante el proceso. -""" -import os -import customtkinter as ctk -from gui.styles import AppStyles -from gui.tabs import SettingsTab, ResultsTab, MainProcessingTab, PreclassificationTab, MetricsTab, PostProcessTab, CleaningTab - - -class InvoiceAppView(ctk.CTk): - """Ventana principal con pestañas de Procesar y Configuración.""" - def __init__(self): - """Construye la ventana principal y sus pestañas/controles.""" - super().__init__() - self.title("IPSExtractor - Procesador de Documentos") - self.geometry("1100x800") - - # Configurar tema - AppStyles.configure_theme() - - # Obtener estilos - self.fonts = AppStyles.get_fonts() - self.colors = AppStyles.get_colors() - - # Retrocompatibilidad: usar los nombres antiguos - self.font_title = self.fonts["title"] - self.font_section = self.fonts["section"] - self.font_ui = self.fonts["ui"] - self.font_mono = self.fonts["mono"] - - # Layout principal - self.grid_columnconfigure(0, weight=1) - self.grid_rowconfigure(0, weight=1) - - # TabView principal con estilo - self.tabs = ctk.CTkTabview(self, width=1000, height=700) - self.tabs.grid(row=0, column=0, sticky="nsew", padx=20, pady=20) - - self.tab_main = self.tabs.add("Procesar Documentos") - self.tab_cleaning = self.tabs.add("Proceso de Limpieza") - self.tab_preclass = self.tabs.add("Preclasificación") - self.tab_results = self.tabs.add("Resultados") - self.tab_settings = self.tabs.add("Configuración") - self.tab_usage = self.tabs.add("Dashboard & Métricas") - self.tab_postprocess = self.tabs.add("Post-Procesamiento") - - # ========================================================================= - # --- TAB: PROCESAR DOCUMENTOS --- - # ========================================================================= - self.main_tab = MainProcessingTab(self.tab_main) - - # ========================================================================= - # --- TAB: PROCESO DE LIMPIEZA --- - # ========================================================================= - self.cleaning_tab = CleaningTab(self.tab_cleaning) - - # ========================================================================= - # --- TAB: PRECLASIFICACIÓN --- - # ========================================================================= - self.preclass_tab = PreclassificationTab(self.tab_preclass) - - # ========================================================================= - # --- TAB: RESULTADOS --- - # ========================================================================= - self.results_tab = ResultsTab(self.tab_results) - - # ========================================================================= - # --- TAB: CONFIGURACIÓN --- - # ========================================================================= - self.settings_tab = SettingsTab(self.tab_settings) - - # ========================================================================= - # --- TAB: DASHBOARD & MÉTRICAS --- - # ========================================================================= - self.metrics_tab = MetricsTab(self.tab_usage) - - # ========================================================================= - # --- TAB: POST-PROCESAMIENTO --- - # ========================================================================= - self.postprocess_tab = PostProcessTab(self.tab_postprocess) - - - # --------------------------------------------------------------------------- - # MÉTODOS PÚBLICOS (API COMPATIBLE) - # --------------------------------------------------------------------------- - - def set_on_select_root(self, callback): - self.main_tab.set_on_select_root(callback) - - def set_on_select_preclass_root(self, callback): - self.preclass_tab.set_on_select_root(callback) - - def set_on_process(self, callback): - self.main_tab.set_on_process(callback) - - def set_on_select_csv(self, callback): - self.main_tab.set_on_select_csv(callback) - - def set_on_save_settings(self, callback): - self.settings_tab.set_on_save(callback) - - def get_selected_model(self) -> str: - return self.settings_tab.get_selected_model() - - def set_selected_model(self, model: str) -> None: - self.settings_tab.set_selected_model(model) - - def get_concurrency_limit(self) -> int: - return self.settings_tab.get_concurrency_limit() - - def set_concurrency_limit(self, limit: int) -> None: - self.settings_tab.set_concurrency_limit(limit) - - def get_allowed_document_types(self) -> list[str]: - return self.settings_tab.get_allowed_document_types() - - def set_allowed_document_types(self, values: list[str]) -> None: - self.settings_tab.set_allowed_document_types(values) - - def get_moderadora_cuotas_text(self) -> str: - return self.settings_tab.get_moderadora_cuotas_text() - - def set_moderadora_cuotas_ui(self, payload: dict, fecha_hint: str) -> None: - self.settings_tab.set_moderadora_cuotas_ui(payload, fecha_hint) - - def get_fecha_servicio_column_hint(self) -> str: - return self.settings_tab.get_fecha_servicio_column_hint() - - def set_on_run_preclassification(self, callback): - self.preclass_tab.set_on_run(callback) - - def set_on_export_excel(self, callback): - self.preclass_tab.set_on_export_excel(callback) - - def set_on_organize_files(self, callback): - self.preclass_tab.set_on_organize_files(callback) - - def enable_preclass_actions(self, enabled: bool): - self.preclass_tab.enable_actions(enabled) - - # --- Métodos Resultados --- - def set_on_res_search(self, callback): - self.results_tab.set_on_search(callback) - - def set_on_res_export(self, callback): - self.results_tab.set_on_export(callback) - - def set_on_res_refresh(self, callback): - self.results_tab.set_on_refresh(callback) - - def set_on_res_organize(self, callback): - self.results_tab.set_on_organize(callback) - - def set_on_res_select(self, callback): - self.results_tab.set_on_select_result(callback) - - def update_res_matches_table(self, rows): - self.results_tab.update_matches_table(rows) - - def update_res_runs_combo(self, runs: list): - self.results_tab.update_runs_combo(runs) - - def set_res_run(self, run_id: int | str): - self.results_tab.set_run_id(run_id) - - def get_res_filters(self): - return self.results_tab.get_filters() - - def set_res_filter_note(self, text: str) -> None: - self.results_tab.set_filter_note(text) - - def clear_res_filter_note(self) -> None: - self.results_tab.clear_filter_note() - - def update_res_table(self, rows): - self.results_tab.update_results_table(rows) - - def append_preclass_output(self, text: str) -> None: - self.preclass_tab.append_output(text) - - def append_preclass_output_safe(self, text: str) -> None: - self.after(0, lambda: self.preclass_tab.append_output_safe(text)) - - def get_root_path(self) -> str: - return self.main_tab.get_root_path() - - def set_root_path(self, path: str) -> None: - self.main_tab.set_root_path(path) - - def set_csv_path(self, path: str) -> None: - self.main_tab.set_csv_path(path) - - def set_preclass_root_path(self, path: str) -> None: - self.preclass_tab.set_root_path(path) - - def get_selected_result_values(self) -> tuple | None: - return self.results_tab.get_selected_result_values() - - def get_process_mode(self) -> str: - """Devuelve el modo de procesamiento: 'radicados' o 'general'.""" - return self.main_tab.get_process_mode() - - def set_process_mode(self, mode: str) -> None: - self.main_tab.set_process_mode(mode) - - def get_preclass_root_path(self) -> str: - return self.preclass_tab.get_root_path() - - def get_preclass_mode(self) -> str: - """Devuelve el modo de preclasificación: 'radicados' o 'general'.""" - return self.preclass_tab.get_mode() - - def get_preclass_range(self) -> tuple[bool, int, int]: - """Devuelve (habilitado, desde, hasta) para preclasificación.""" - return self.preclass_tab.get_range() - - def get_preclass_target_info(self) -> dict: - """Devuelve la configuración del archivo objetivo.""" - return self.preclass_tab.get_target_info() - - def get_keywords(self) -> str: - return self.main_tab.get_keywords() - - def get_fields(self) -> str: - return self.main_tab.get_fields() - - def get_csv_path(self) -> str: - return self.main_tab.get_csv_path() - - def get_retry_empty_enabled(self) -> bool: - return self.main_tab.get_retry_empty_enabled() - - def set_retry_empty_enabled(self, enabled: bool) -> None: - self.main_tab.set_retry_empty_enabled(enabled) - - def get_radicado_filter_enabled(self) -> bool: - return self.main_tab.get_radicado_filter_enabled() - - def get_radicado_filter_list(self) -> list[str]: - return self.main_tab.get_radicado_filter_list() - - def show_tiff_range_controls(self, show: bool) -> None: - self.main_tab.show_tiff_range_controls(show) - - def get_tiff_start_page(self) -> int: - return self.main_tab.get_tiff_start_page() - - def get_tiff_end_page(self) -> int: - return self.main_tab.get_tiff_end_page() - - def append_output(self, text: str) -> None: - self.main_tab.append_output(text) - - def append_output_safe(self, text: str) -> None: - self.after(0, lambda: self.main_tab.append_output_safe(text)) - - def set_processing(self, processing: bool) -> None: - self.main_tab.enable_controls(not processing) - - - # --- TAB PRECLASIFICACIÓN --- - try: - # Actualizar controles de la pestaña de preclasificación - self.preclass_tab.enable_controls(not processing) - - # Actualizar controles de la pestaña de configuración - self.settings_tab.enable_controls(not processing) - except Exception: - pass - - # --- TAB USO --- - - def append_usage(self, text: str) -> None: - self.metrics_tab.append_usage(text) - - def append_usage_safe(self, text: str) -> None: - self.after(0, lambda: self.metrics_tab.append_usage_safe(text)) - - def clear_usage(self) -> None: - self.metrics_tab.clear() - - def set_usage_totals(self, **kwargs): - self.metrics_tab.set_usage_totals(**kwargs) - - def update_kpis(self, overall: dict, extract_phase: dict, normalize_phase: dict) -> None: - self.metrics_tab.update_kpis(overall, extract_phase, normalize_phase) - - def update_doc_kpi(self, processed: int, avg_cost: float) -> None: - self.metrics_tab.update_doc_kpi(processed, avg_cost) - - def update_cost_line(self, doc_index: int, total_cost_usd: float) -> None: - self.metrics_tab.update_cost_line(doc_index, total_cost_usd) - - def update_token_bars(self, input_extract: int, cached_extract: int, output_extract: int, - input_norm: int, cached_norm: int, output_norm: int, - input_pp: int = 0, output_pp: int = 0) -> None: - self.metrics_tab.update_token_bars(input_extract, cached_extract, output_extract, - input_norm, cached_norm, output_norm, - input_pp, output_pp) - - def update_comparison_table(self, comparisons: list[dict]) -> None: - self.metrics_tab.update_comparison_table(comparisons) - - def append_usage_row(self, phase: str, radicado: str, usage: dict | None, cost: dict | None) -> None: - self.metrics_tab.append_usage_row(phase, radicado, usage, cost) - - def update_pp_kpis(self, total_usd: float, total_calls: int, total_retries: int, - input_tokens: int, output_tokens: int, last_description: str) -> None: - self.metrics_tab.update_pp_kpis(total_usd, total_calls, total_retries, - input_tokens, output_tokens, last_description) - - # --- HELPERS INTERNOS --- - - def get_preclass_patterns(self) -> list[dict]: - """Devuelve la lista de patrones de preclasificación definida en la pestaña.""" - return self.preclass_tab.get_patterns() - - # --- TAB POST-PROCESAMIENTO --- - - def set_on_postprocess_select_csv(self, callback): - self.postprocess_tab.set_on_select_csv(callback) - - def set_on_postprocess_run(self, callback): - self.postprocess_tab.set_on_run(callback) - - def set_on_postprocess_save(self, callback): - self.postprocess_tab.set_on_save(callback) - - def get_postprocess_instructions(self) -> str: - return self.postprocess_tab.get_instructions() - - def set_postprocess_csv_path(self, path: str): - self.postprocess_tab.set_csv_path(path) - - def set_postprocess_preview(self, text: str): - self.postprocess_tab.set_preview(text) - - def set_postprocess_code(self, text: str): - self.postprocess_tab.set_code(text) - - def append_postprocess_log(self, text: str): - self.postprocess_tab.append_log(text) - - def append_postprocess_log_safe(self, text: str): - self.after(0, lambda: self.postprocess_tab.append_log(text)) - - def clear_postprocess_log(self): - self.postprocess_tab.clear_log() - - def set_postprocess_result(self, text: str): - self.postprocess_tab.set_result(text) - - def set_postprocess_status(self, text: str): - self.postprocess_tab.set_status(text) - - def enable_postprocess_save(self, enabled: bool): - self.postprocess_tab.enable_save(enabled) - - def enable_postprocess_controls(self, enabled: bool): - self.postprocess_tab.enable_controls(enabled) - - def get_doc_types(self) -> list[dict]: - return self.main_tab.get_doc_types() - - def set_doc_types(self, doc_types: list[dict]) -> None: - self.main_tab.set_doc_types(doc_types) - - def set_allowed_document_types_hint(self, allowed_types: list[str]) -> None: - self.main_tab.set_allowed_document_types_hint(allowed_types) - - def validate_doc_types(self) -> tuple[bool, str | None]: - return self.main_tab.validate_doc_types() - - # --- TAB PROCESO DE LIMPIEZA --- - - def set_on_cleaning_select_csv(self, callback): - self.cleaning_tab.set_on_select_csv(callback) - - def set_on_cleaning_apply(self, callback): - self.cleaning_tab.set_on_apply(callback) - - def set_on_cleaning_save(self, callback): - self.cleaning_tab.set_on_save(callback) - - def set_cleaning_csv_path(self, path: str): - self.cleaning_tab.set_csv_path(path) - - def set_cleaning_preview(self, df): - self.cleaning_tab.set_preview(df) - - def populate_cleaning_columns(self, summaries): - self.cleaning_tab.populate_columns(summaries) - - def get_cleaning_selected_rules(self) -> dict[str, list[str]]: - return self.cleaning_tab.get_selected_rules() - - def get_cleaning_cop_mod_fecha_columns(self) -> dict[str, str]: - return self.cleaning_tab.get_cop_mod_fecha_columns() - - def set_cleaning_result(self, df, info_text: str = ""): - self.cleaning_tab.set_result(df, info_text) - - def enable_cleaning_save(self, enabled: bool): - self.cleaning_tab.enable_save(enabled) - - def enable_cleaning_controls(self, enabled: bool): - self.cleaning_tab.enable_controls(enabled) - - def set_cleaning_status(self, text: str): - self.cleaning_tab.set_status(text) - - def set_cleaning_loading(self, loading: bool, text: str = ""): - self.cleaning_tab.set_loading(loading, text) - - def set_cleaning_delimiter_chip(self, delimiter: str): - self.cleaning_tab.set_delimiter_chip(delimiter) diff --git a/gui_main.py b/gui_main.py deleted file mode 100644 index 9f35e62..0000000 --- a/gui_main.py +++ /dev/null @@ -1,59 +0,0 @@ -"""Entrada para la interfaz gráfica. - -Crea el servicio con el extractor por defecto y lanza la ventana principal. -""" -import logging - -import customtkinter as ctk - -from application.service import InvoiceProcessingService -from application.strategies import OllamaExtractor -from gui.view import InvoiceAppView -from gui.controller import AppController -from application.logging_config import configure_logging -from application.config import settings - - -def main(): - """Inicializa logging y arranca la aplicación GUI.""" - configure_logging() - - # Log effective settings *after* logging is configured - _cfg_logger = logging.getLogger("application.config") - try: - _cfg_logger.info( - "effective_api_settings", - extra={ - "component": "config", - "base_url": settings.api.base_url, - "model": settings.api.model, - "timeout_s": getattr(settings.api, "request_timeout_s", None), - }, - ) - except Exception: - pass - - extractor = OllamaExtractor( - api_key=settings.api.api_key, - base_url=settings.api.base_url, - model=settings.api.model - ) - - from application.batch_processor import BatchProcessor - from application.estimator import Estimator - - service = InvoiceProcessingService( - extractor, - batch_processor_factory=lambda s: BatchProcessor(s), - estimator_factory=lambda s: Estimator(s) - ) - - app = InvoiceAppView() - AppController(app, service) - app.mainloop() - - -if __name__ == "__main__": - main() - - diff --git a/installer.iss b/installer.iss deleted file mode 100644 index 6c6d0ff..0000000 --- a/installer.iss +++ /dev/null @@ -1,49 +0,0 @@ -; ============================================================================ -; Inno Setup Script para IPSExtractor -; -; Prerequisito: ejecutar build_installer.bat primero para generar dist\IPSExtractor -; Luego: build_installer.bat --inno o compilar este .iss directamente -; -; Descargar Inno Setup: https://jrsoftware.org/isdl.php -; ============================================================================ - -#define MyAppName "IPSExtractor" -#define MyAppVersion "1.0.0" -#define MyAppPublisher "IPSExtractor" -#define MyAppExeName "IPSExtractor.exe" - -[Setup] -AppId={{A1B2C3D4-E5F6-7890-ABCD-EF1234567890} -AppName={#MyAppName} -AppVersion={#MyAppVersion} -AppPublisher={#MyAppPublisher} -DefaultDirName={autopf}\{#MyAppName} -DefaultGroupName={#MyAppName} -DisableProgramGroupPage=yes -OutputDir=dist -OutputBaseFilename=IPSExtractor_Setup_{#MyAppVersion} -Compression=lzma2/ultra64 -SolidCompression=yes -WizardStyle=modern -PrivilegesRequired=lowest -ArchitecturesAllowed=x64compatible -ArchitecturesInstallIn64BitMode=x64compatible -SetupLogging=yes - -[Languages] -Name: "spanish"; MessagesFile: "compiler:Languages\Spanish.isl" - -[Tasks] -Name: "desktopicon"; Description: "Crear acceso directo en el Escritorio"; GroupDescription: "Accesos directos:" - -[Files] -; Todos los archivos de dist\IPSExtractor -Source: "dist\IPSExtractor\*"; DestDir: "{app}"; Flags: ignoreversion recursesubdirs createallsubdirs - -[Icons] -Name: "{group}\{#MyAppName}"; Filename: "{app}\{#MyAppExeName}" -Name: "{group}\Desinstalar {#MyAppName}"; Filename: "{uninstallexe}" -Name: "{autodesktop}\{#MyAppName}"; Filename: "{app}\{#MyAppExeName}"; Tasks: desktopicon - -[Run] -Filename: "{app}\{#MyAppExeName}"; Description: "Ejecutar {#MyAppName}"; Flags: nowait postinstall skipifsilent diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index 21bf4e5..0000000 --- a/requirements.txt +++ /dev/null @@ -1,51 +0,0 @@ -annotated-types==0.7.0 -anyio==4.11.0 -certifi==2025.8.3 -charset-normalizer==3.4.3 -colorama==0.4.6 -coloredlogs==15.0.1 -contourpy==1.3.3 -customtkinter==5.2.2 -cycler==0.12.1 -darkdetect==0.8.0 -distro==1.9.0 -flatbuffers==25.9.23 -fonttools==4.60.1 -h11==0.16.0 -httpcore==1.0.9 -httpx==0.28.1 -humanfriendly==10.0 -idna==3.10 -jiter==0.11.0 -kiwisolver==1.4.9 -matplotlib==3.10.7 -mpmath==1.3.0 -numpy==2.2.6 -ollama==0.6.0 -onnxruntime==1.23.2 -openai==2.1.0 -opencv-python==4.12.0.88 -packaging==25.0 -pdf2image==1.17.0 -pillow==11.3.0 -protobuf==6.33.0 -pyclipper==1.3.0.post6 -pydantic==2.11.9 -pydantic-settings==2.11.0 -pydantic_core==2.33.2 -pyparsing==3.2.5 -pyreadline3==3.5.4 -python-dateutil==2.9.0.post0 -python-dotenv==1.1.1 -PyYAML==6.0.3 -rapidocr-onnxruntime==1.4.4 -requests==2.32.5 -shapely==2.1.2 -six==1.17.0 -sniffio==1.3.1 -sympy==1.14.0 -tenacity==9.1.2 -tqdm==4.67.1 -typing-inspection==0.4.2 -typing_extensions==4.15.0 -urllib3==2.5.0 diff --git a/server/app/__init__.py b/server/app/__init__.py deleted file mode 100644 index 8b13789..0000000 --- a/server/app/__init__.py +++ /dev/null @@ -1 +0,0 @@ - diff --git a/server/app/database.py b/server/app/database.py deleted file mode 100644 index 2cc751a..0000000 --- a/server/app/database.py +++ /dev/null @@ -1,159 +0,0 @@ -"""Capa de persistencia SQLite para el servidor de tracking. - -Guarda el historial completo de lotes de extracción, incluyendo -métricas de progreso, costos y la ruta del CSV generado. -""" -import sqlite3 -import json -import os -from typing import Dict, Any, List, Optional -from datetime import datetime - -DB_PATH = os.path.join(os.path.dirname(__file__), "..", "tracking.db") - - -def get_connection() -> sqlite3.Connection: - conn = sqlite3.connect(DB_PATH) - conn.row_factory = sqlite3.Row - return conn - - -def init_db(): - """Crea las tablas si no existen.""" - with get_connection() as conn: - conn.execute(""" - CREATE TABLE IF NOT EXISTS batch_logs ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - batch_id TEXT NOT NULL, - timestamp TEXT NOT NULL, - level TEXT NOT NULL DEFAULT 'info', - radicado TEXT, - message TEXT NOT NULL, - FOREIGN KEY (batch_id) REFERENCES batches(id) - ) - """) - conn.execute("CREATE INDEX IF NOT EXISTS idx_batch_logs_batch_id ON batch_logs(batch_id)") - conn.execute(""" - CREATE TABLE IF NOT EXISTS batches ( - id TEXT PRIMARY KEY, - root_path TEXT NOT NULL, - csv_output_path TEXT, - hostname TEXT, - username TEXT, - total_radicados INTEGER NOT NULL DEFAULT 0, - config TEXT, - status TEXT NOT NULL DEFAULT 'processing', - current_radicado_index INTEGER NOT NULL DEFAULT 0, - success_count INTEGER NOT NULL DEFAULT 0, - error_count INTEGER NOT NULL DEFAULT 0, - total_tokens INTEGER NOT NULL DEFAULT 0, - total_cost_usd REAL NOT NULL DEFAULT 0.0, - last_error TEXT, - current_phase TEXT, - current_radicado TEXT, - created_at TEXT NOT NULL, - updated_at TEXT NOT NULL - ) - """) - conn.commit() - - -def insert_batch(batch: Dict[str, Any]): - """Inserta un nuevo lote en la base de datos.""" - with get_connection() as conn: - conn.execute(""" - INSERT INTO batches ( - id, root_path, csv_output_path, hostname, username, total_radicados, config, - status, current_radicado_index, success_count, error_count, - total_tokens, total_cost_usd, last_error, current_phase, current_radicado, - created_at, updated_at - ) VALUES ( - :id, :root_path, :csv_output_path, :hostname, :username, :total_radicados, :config, - :status, :current_radicado_index, :success_count, :error_count, - :total_tokens, :total_cost_usd, :last_error, :current_phase, :current_radicado, - :created_at, :updated_at - ) - """, { - **batch, - "config": json.dumps(batch.get("config") or {}), - "created_at": _to_iso(batch.get("created_at")), - "updated_at": _to_iso(batch.get("updated_at")), - "current_phase": batch.get("current_phase"), - "current_radicado": batch.get("current_radicado"), - }) - conn.commit() - - -def update_batch(batch_id: str, fields: Dict[str, Any]): - """Actualiza campos de un lote existente.""" - if not fields: - return - fields["updated_at"] = _to_iso(datetime.now()) - set_clause = ", ".join(f"{k} = :{k}" for k in fields) - fields["batch_id"] = batch_id - with get_connection() as conn: - conn.execute( - f"UPDATE batches SET {set_clause} WHERE id = :batch_id", - fields - ) - conn.commit() - - -def get_all_batches() -> List[Dict[str, Any]]: - """Retorna todos los lotes ordenados por fecha de creación descendente.""" - with get_connection() as conn: - rows = conn.execute( - "SELECT * FROM batches ORDER BY created_at DESC" - ).fetchall() - return [_row_to_dict(r) for r in rows] - - -def insert_log(batch_id: str, message: str, level: str = "info", radicado: str = None): - """Inserta una entrada de log para un lote.""" - with get_connection() as conn: - conn.execute( - "INSERT INTO batch_logs (batch_id, timestamp, level, radicado, message) VALUES (?, ?, ?, ?, ?)", - (batch_id, datetime.now().isoformat(), level, radicado, message) - ) - conn.commit() - - -def get_logs(batch_id: str, limit: int = 200) -> List[Dict[str, Any]]: - """Retorna los logs de un lote, del más reciente al más antiguo.""" - with get_connection() as conn: - rows = conn.execute( - "SELECT * FROM batch_logs WHERE batch_id = ? ORDER BY id DESC LIMIT ?", - (batch_id, limit) - ).fetchall() - return [dict(r) for r in rows] - - -def get_batch(batch_id: str) -> Optional[Dict[str, Any]]: - """Retorna un lote por su ID.""" - with get_connection() as conn: - row = conn.execute( - "SELECT * FROM batches WHERE id = ?", (batch_id,) - ).fetchone() - return _row_to_dict(row) if row else None - - -def _row_to_dict(row: sqlite3.Row) -> Dict[str, Any]: - d = dict(row) - try: - d["config"] = json.loads(d.get("config") or "{}") - except Exception: - d["config"] = {} - - # Asegurar que hostname y username estén presentes - if "hostname" not in d: d["hostname"] = None - if "username" not in d: d["username"] = None - - return d - - -def _to_iso(value) -> str: - if isinstance(value, datetime): - return value.isoformat() - if isinstance(value, str): - return value - return datetime.now().isoformat() diff --git a/server/app/main.py b/server/app/main.py deleted file mode 100644 index b24114a..0000000 --- a/server/app/main.py +++ /dev/null @@ -1,187 +0,0 @@ -"""Servidor de tracking de IPSExtractor. - -Persiste el historial de lotes en SQLite y distribuye actualizaciones -en tiempo real vía WebSockets al dashboard web. -""" -from fastapi import FastAPI, WebSocket, WebSocketDisconnect, HTTPException -from fastapi.middleware.cors import CORSMiddleware -from fastapi.encoders import jsonable_encoder -from fastapi.responses import FileResponse -from contextlib import asynccontextmanager -from typing import Dict, List, Any, Optional -from pydantic import BaseModel -import uuid -import os -from datetime import datetime - -from .schemas.batch import BatchCreate, BatchUpdate, BatchRead, BatchStatus -from .database import init_db, insert_batch, update_batch, get_all_batches, get_batch, insert_log, get_logs - - -@asynccontextmanager -async def lifespan(app: FastAPI): - init_db() - yield - -app = FastAPI(title="IPSExtractor Tracking API", lifespan=lifespan) - -app.add_middleware( - CORSMiddleware, - allow_origins=["*"], - allow_credentials=True, - allow_methods=["*"], - allow_headers=["*"], -) - -# Cache en memoria para lotes activos (para broadcast rápido) -active_batches: Dict[str, Dict[str, Any]] = {} -active_connections: List[WebSocket] = [] - - -async def broadcast(message: dict): - json_compatible_message = jsonable_encoder(message) - disconnected = [] - for connection in active_connections: - try: - await connection.send_json(json_compatible_message) - except Exception: - disconnected.append(connection) - for conn in disconnected: - active_connections.remove(conn) - - -@app.post("/batches", response_model=BatchRead) -async def create_batch(batch: BatchCreate): - batch_id = str(uuid.uuid4()) - now = datetime.now() - new_batch = { - "id": batch_id, - **batch.model_dump(), - "status": BatchStatus.PROCESSING, - "current_radicado_index": 0, - "success_count": 0, - "error_count": 0, - "total_tokens": 0, - "total_cost_usd": 0.0, - "last_error": None, - "created_at": now, - "updated_at": now, - } - active_batches[batch_id] = new_batch - insert_batch(new_batch) - await broadcast({"event": "batch_created", "data": new_batch}) - return new_batch - - -@app.get("/batches", response_model=List[BatchRead]) -async def list_batches(): - # Retorna historial completo desde SQLite - db_batches = get_all_batches() - # Sobreescribe con datos en memoria si el lote está activo (más fresco) - result = [] - for b in db_batches: - if b["id"] in active_batches: - result.append(active_batches[b["id"]]) - else: - result.append(b) - return result - - -@app.get("/batches/{batch_id}", response_model=BatchRead) -async def get_batch_endpoint(batch_id: str): - if batch_id in active_batches: - return active_batches[batch_id] - batch = get_batch(batch_id) - if not batch: - raise HTTPException(status_code=404, detail="Lote no encontrado") - return batch - - -@app.patch("/batches/{batch_id}", response_model=BatchRead) -async def update_batch_endpoint(batch_id: str, update: BatchUpdate): - if batch_id in active_batches: - batch = active_batches[batch_id] - update_data = update.model_dump(exclude_unset=True) - for key, value in update_data.items(): - batch[key] = value - batch["updated_at"] = datetime.now() - - # Si el lote se completó, sacarlo del cache activo - if batch.get("status") in (BatchStatus.COMPLETED, BatchStatus.FAILED): - active_batches.pop(batch_id, None) - - update_batch(batch_id, update_data) - await broadcast({"event": "batch_updated", "data": batch}) - return batch - - # Lote no está en memoria (completado), actualizar solo en DB - batch = get_batch(batch_id) - if not batch: - raise HTTPException(status_code=404, detail="Lote no encontrado") - update_data = update.model_dump(exclude_unset=True) - update_batch(batch_id, update_data) - batch.update(update_data) - return batch - - -class LogEntry(BaseModel): - message: str - level: str = "info" - radicado: Optional[str] = None - -class LogRead(BaseModel): - id: int - batch_id: str - timestamp: str - level: str - radicado: Optional[str] - message: str - -@app.post("/batches/{batch_id}/logs", status_code=201) -async def add_log(batch_id: str, entry: LogEntry): - """Añade una entrada de log para un lote.""" - insert_log(batch_id, entry.message, entry.level, entry.radicado) - # Broadcast log también por WebSocket para actualizaciones en vivo - await broadcast({"event": "log_added", "batch_id": batch_id, "data": { - "level": entry.level, "radicado": entry.radicado, "message": entry.message - }}) - return {"ok": True} - - -@app.get("/batches/{batch_id}/logs", response_model=List[LogRead]) -async def list_logs(batch_id: str, limit: int = 200): - """Retorna el historial de logs de un lote.""" - return get_logs(batch_id, limit) - - -@app.get("/download/{batch_id}") -async def download_csv(batch_id: str): - """Sirve el archivo CSV generado por el lote de extracción.""" - batch = get_batch(batch_id) - if not batch: - raise HTTPException(status_code=404, detail="Lote no encontrado") - - csv_path = batch.get("csv_output_path") - if not csv_path: - raise HTTPException(status_code=404, detail="Este lote no tiene un archivo CSV asociado") - if not os.path.isfile(csv_path): - raise HTTPException(status_code=404, detail=f"Archivo CSV no encontrado en el servidor: {csv_path}") - - filename = os.path.basename(csv_path) - return FileResponse( - path=csv_path, - media_type="text/csv", - filename=filename, - ) - - -@app.websocket("/ws/tracking") -async def websocket_endpoint(websocket: WebSocket): - await websocket.accept() - active_connections.append(websocket) - try: - while True: - await websocket.receive_text() - except WebSocketDisconnect: - if websocket in active_connections: - active_connections.remove(websocket) diff --git a/server/app/schemas/__init__.py b/server/app/schemas/__init__.py deleted file mode 100644 index 8b13789..0000000 --- a/server/app/schemas/__init__.py +++ /dev/null @@ -1 +0,0 @@ - diff --git a/server/app/schemas/batch.py b/server/app/schemas/batch.py deleted file mode 100644 index 0b846ff..0000000 --- a/server/app/schemas/batch.py +++ /dev/null @@ -1,54 +0,0 @@ -from pydantic import BaseModel -from typing import Optional, List, Dict -from datetime import datetime -from enum import Enum - - -class BatchStatus(str, Enum): - PENDING = "pending" - PROCESSING = "processing" - COMPLETED = "completed" - FAILED = "failed" - - -class BatchBase(BaseModel): - root_path: str - total_radicados: int - csv_output_path: Optional[str] = None - hostname: Optional[str] = None - username: Optional[str] = None - config: Optional[Dict] = None - - -class BatchCreate(BatchBase): - pass - - -class BatchUpdate(BaseModel): - status: Optional[BatchStatus] = None - current_radicado_index: Optional[int] = None - success_count: Optional[int] = None - error_count: Optional[int] = None - total_tokens: Optional[int] = None - total_cost_usd: Optional[float] = None - last_error: Optional[str] = None - current_phase: Optional[str] = None - current_radicado: Optional[str] = None - - -class BatchRead(BatchBase): - id: str - status: BatchStatus - current_radicado_index: int = 0 - success_count: int = 0 - error_count: int = 0 - total_tokens: int = 0 - total_cost_usd: float = 0.0 - last_error: Optional[str] = None - current_phase: Optional[str] = None - current_radicado: Optional[str] = None - created_at: datetime - updated_at: datetime - - class Config: - from_attributes = True diff --git a/services/api_gateway/Dockerfile b/services/api_gateway/Dockerfile new file mode 100644 index 0000000..c9bd7d7 --- /dev/null +++ b/services/api_gateway/Dockerfile @@ -0,0 +1,15 @@ +FROM python:3.11-slim + +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + PYTHONPATH=/app + +WORKDIR /app + +COPY requirements.txt /app/requirements.txt +RUN pip install --no-cache-dir -r requirements.txt + +COPY api /app/api + +EXPOSE 8000 +CMD ["uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8000"] diff --git a/services/api_gateway/README.md b/services/api_gateway/README.md new file mode 100644 index 0000000..de32f3b --- /dev/null +++ b/services/api_gateway/README.md @@ -0,0 +1,29 @@ +# IPSExtractor API Gateway + +Gateway HTTP opcional. Esta carpeta puede vivir como repo propio: solo depende +de HTTP para hablar con `extraction` y `cleaning`; no importa codigo interno de +esos servicios. + +## Estructura + +```text +api/ Endpoints FastAPI y proxy HTTP +requirements.txt +Dockerfile +``` + +## Desarrollo local + +```bash +pip install -r requirements.txt +uvicorn api.main:app --host 0.0.0.0 --port 8000 +``` + +## Endpoints principales + +- `GET /health` +- `GET /models` +- `GET /config` +- `ANY /extraction/{path}` +- `ANY /cleaning/{path}` + diff --git a/services/api_gateway/api/__init__.py b/services/api_gateway/api/__init__.py new file mode 100644 index 0000000..20c2ab1 --- /dev/null +++ b/services/api_gateway/api/__init__.py @@ -0,0 +1 @@ +"""API central opcional para enrutar a los microservicios.""" diff --git a/services/api_gateway/api/main.py b/services/api_gateway/api/main.py new file mode 100644 index 0000000..373ee7c --- /dev/null +++ b/services/api_gateway/api/main.py @@ -0,0 +1,110 @@ +"""Gateway HTTP sin compartir codigo interno de los microservicios.""" +from __future__ import annotations + +import os +from typing import Any + +import httpx +from fastapi import FastAPI, Request, Response + + +EXTRACTION_URL = os.getenv("EXTRACTION_URL", "http://extraction:8001").rstrip("/") +CLEANING_URL = os.getenv("CLEANING_URL", "http://cleaning:8002").rstrip("/") + +app = FastAPI(title="IPSExtractor API Gateway", version="1.0.0") + + +@app.get("/health") +async def health() -> dict[str, Any]: + async with httpx.AsyncClient(timeout=10.0) as client: + extraction = await _safe_health(client, EXTRACTION_URL) + cleaning = await _safe_health(client, CLEANING_URL) + return { + "status": "ok" if extraction["ok"] and cleaning["ok"] else "degraded", + "services": { + "extraction": extraction, + "cleaning": cleaning, + }, + } + + +@app.get("/models") +async def models() -> Response: + return await _proxy("GET", f"{EXTRACTION_URL}/models") + + +@app.get("/config") +async def config() -> dict[str, Any]: + async with httpx.AsyncClient(timeout=15.0) as client: + extraction = await _safe_json(client, f"{EXTRACTION_URL}/config") + cleaning_rules = await _safe_json(client, f"{CLEANING_URL}/rules") + return { + "extraction": extraction, + "cleaning": {"rules": cleaning_rules}, + } + + +@app.api_route("/extraction/{path:path}", methods=["GET", "POST", "PUT", "PATCH", "DELETE"]) +async def extraction_proxy(path: str, request: Request) -> Response: + return await _proxy_request(request, f"{EXTRACTION_URL}/{path}") + + +@app.api_route("/cleaning/{path:path}", methods=["GET", "POST", "PUT", "PATCH", "DELETE"]) +async def cleaning_proxy(path: str, request: Request) -> Response: + return await _proxy_request(request, f"{CLEANING_URL}/{path}") + + +async def _safe_health(client: httpx.AsyncClient, base_url: str) -> dict[str, Any]: + try: + response = await client.get(f"{base_url}/health") + return {"ok": response.is_success, "status_code": response.status_code} + except Exception as exc: + return {"ok": False, "error": str(exc)} + + +async def _safe_json(client: httpx.AsyncClient, url: str) -> Any: + try: + response = await client.get(url) + response.raise_for_status() + return response.json() + except Exception as exc: + return {"error": str(exc)} + + +async def _proxy(method: str, url: str) -> Response: + async with httpx.AsyncClient(timeout=60.0) as client: + upstream = await client.request(method, url) + return _to_response(upstream) + + +async def _proxy_request(request: Request, url: str) -> Response: + body = await request.body() + headers = { + key: value + for key, value in request.headers.items() + if key.lower() not in {"host", "content-length"} + } + async with httpx.AsyncClient(timeout=None) as client: + upstream = await client.request( + request.method, + url, + params=request.query_params, + content=body, + headers=headers, + ) + return _to_response(upstream) + + +def _to_response(upstream: httpx.Response) -> Response: + excluded = {"content-encoding", "transfer-encoding", "connection"} + headers = { + key: value + for key, value in upstream.headers.items() + if key.lower() not in excluded + } + return Response( + content=upstream.content, + status_code=upstream.status_code, + headers=headers, + media_type=upstream.headers.get("content-type"), + ) diff --git a/services/api_gateway/requirements.txt b/services/api_gateway/requirements.txt new file mode 100644 index 0000000..ab3dca4 --- /dev/null +++ b/services/api_gateway/requirements.txt @@ -0,0 +1,3 @@ +fastapi==0.115.6 +httpx==0.28.1 +uvicorn[standard]==0.32.1 diff --git a/services/cleaning/Dockerfile b/services/cleaning/Dockerfile new file mode 100644 index 0000000..61ba96a --- /dev/null +++ b/services/cleaning/Dockerfile @@ -0,0 +1,21 @@ +FROM python:3.11-slim + +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + PYTHONPATH=/app + +WORKDIR /app + +RUN apt-get update \ + && apt-get install -y --no-install-recommends libgl1 libglib2.0-0 \ + && rm -rf /var/lib/apt/lists/* + +COPY requirements.txt /app/requirements.txt +RUN pip install --no-cache-dir -r requirements.txt + +COPY domain /app/domain +COPY infrastructure /app/infrastructure +COPY api /app/api + +EXPOSE 8002 +CMD ["uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8002"] diff --git a/services/cleaning/README.md b/services/cleaning/README.md new file mode 100644 index 0000000..0ec8191 --- /dev/null +++ b/services/cleaning/README.md @@ -0,0 +1,32 @@ +# IPSExtractor Cleaning Service + +Microservicio independiente de limpieza. Esta carpeta puede vivir como repo +propio: contiene su API REST, reglas de limpieza, infraestructura, dependencias +y Dockerfile sin importar codigo de otros servicios. + +## Estructura + +```text +api/ Endpoints FastAPI +domain/ Reglas y analisis de limpieza +infrastructure/ Persistencia SQLAlchemy del servicio +requirements.txt +Dockerfile +``` + +## Desarrollo local + +```bash +pip install -r requirements.txt +uvicorn api.main:app --host 0.0.0.0 --port 8002 +``` + +## Endpoints principales + +- `GET /health` +- `GET /rules` +- `GET /batches/{batch_id}/analysis` +- `POST /jobs` +- `GET /jobs/{job_id}` +- `GET /jobs/{job_id}/rows` + diff --git a/services/cleaning/api/__init__.py b/services/cleaning/api/__init__.py new file mode 100644 index 0000000..9c5dd31 --- /dev/null +++ b/services/cleaning/api/__init__.py @@ -0,0 +1 @@ +"""Servicio REST de limpieza.""" diff --git a/services/cleaning/api/main.py b/services/cleaning/api/main.py new file mode 100644 index 0000000..505da45 --- /dev/null +++ b/services/cleaning/api/main.py @@ -0,0 +1,200 @@ +"""API REST del microservicio de limpieza.""" +from __future__ import annotations + +from contextlib import asynccontextmanager +import json +from typing import Any + +import pandas as pd +from fastapi import FastAPI, HTTPException +from pydantic import BaseModel, Field +from sqlalchemy import delete, select + +from domain.csv_cleaning import CLEANING_RULES, analyze_dataframe, apply_rules +from infrastructure.database import ( + CleanedRow, + CleaningJob, + ExtractionBatch, + ExtractionRow, + init_db, + session_scope, + utcnow, +) + + +@asynccontextmanager +async def lifespan(app: FastAPI): + init_db() + yield + + +app = FastAPI( + title="IPSExtractor Cleaning Service", + version="1.0.0", + lifespan=lifespan, +) + + +class CleaningRequest(BaseModel): + extraction_batch_id: str + rules_per_column: dict[str, list[str]] = Field(default_factory=dict) + cop_mod_fecha_column: dict[str, str] = Field(default_factory=dict) + cop_mod_config: dict[str, Any] | None = None + + +def dataframe_to_records(df: pd.DataFrame) -> list[dict[str, Any]]: + return json.loads(df.to_json(orient="records", date_format="iso")) + + +def serialize_job(job: CleaningJob) -> dict[str, Any]: + return { + "id": job.id, + "extraction_batch_id": job.extraction_batch_id, + "status": job.status, + "request": job.request, + "log": job.log, + "error": job.error, + "created_at": job.created_at.isoformat(), + "updated_at": job.updated_at.isoformat(), + } + + +@app.get("/health") +def health() -> dict[str, str]: + return {"status": "ok"} + + +@app.get("/rules") +def list_rules() -> list[dict[str, Any]]: + return [ + { + "id": rule["id"], + "name": rule["name"], + "description": rule.get("description", ""), + } + for rule in CLEANING_RULES + ] + + +@app.get("/batches/{batch_id}/analysis") +def analyze_batch(batch_id: str) -> dict[str, Any]: + with session_scope() as session: + batch = session.get(ExtractionBatch, batch_id) + if batch is None: + raise HTTPException(status_code=404, detail="Lote de extraccion no encontrado") + rows = session.execute( + select(ExtractionRow.row_json) + .where(ExtractionRow.batch_id == batch_id) + .order_by(ExtractionRow.row_index, ExtractionRow.id) + ).all() + records = [row[0] for row in rows] + + df = pd.DataFrame(records) + return { + "batch_id": batch_id, + "columns": [summary.__dict__ for summary in analyze_dataframe(df)], + } + + +@app.post("/jobs") +def run_cleaning(request: CleaningRequest) -> dict[str, Any]: + with session_scope() as session: + batch = session.get(ExtractionBatch, request.extraction_batch_id) + if batch is None: + raise HTTPException(status_code=404, detail="Lote de extraccion no encontrado") + + extracted_rows = session.execute( + select(ExtractionRow.row_json) + .where(ExtractionRow.batch_id == request.extraction_batch_id) + .order_by(ExtractionRow.row_index, ExtractionRow.id) + ).all() + records = [row[0] for row in extracted_rows] + if not records: + raise HTTPException(status_code=400, detail="El lote no tiene filas extraidas") + + job = CleaningJob( + extraction_batch_id=request.extraction_batch_id, + status="running", + request=request.model_dump(), + updated_at=utcnow(), + ) + session.add(job) + session.flush() + job_id = job.id + + try: + df = pd.DataFrame(records) + result_df, log = apply_rules( + df, + request.rules_per_column, + cop_mod_fecha_column=request.cop_mod_fecha_column, + cop_mod_config=request.cop_mod_config, + ) + output_rows = dataframe_to_records(result_df) + + with session_scope() as session: + session.execute(delete(CleanedRow).where(CleanedRow.cleaning_job_id == job_id)) + for idx, row in enumerate(output_rows, start=1): + session.add( + CleanedRow( + cleaning_job_id=job_id, + extraction_batch_id=request.extraction_batch_id, + row_index=idx, + row_json=row, + ) + ) + job = session.get(CleaningJob, job_id) + if job is None: + raise RuntimeError("No se encontro el job de limpieza recien creado.") + job.status = "completed" + job.log = log + job.updated_at = utcnow() + session.flush() + return serialize_job(job) + except Exception as exc: + with session_scope() as session: + job = session.get(CleaningJob, job_id) + if job is not None: + job.status = "failed" + job.error = str(exc) + job.updated_at = utcnow() + raise HTTPException(status_code=500, detail=str(exc)) from exc + + +@app.get("/jobs") +def list_jobs() -> list[dict[str, Any]]: + with session_scope() as session: + jobs = session.scalars( + select(CleaningJob).order_by(CleaningJob.created_at.desc()) + ).all() + return [serialize_job(job) for job in jobs] + + +@app.get("/jobs/{job_id}") +def get_job(job_id: str) -> dict[str, Any]: + with session_scope() as session: + job = session.get(CleaningJob, job_id) + if job is None: + raise HTTPException(status_code=404, detail="Job de limpieza no encontrado") + return serialize_job(job) + + +@app.get("/jobs/{job_id}/rows") +def get_cleaned_rows(job_id: str, limit: int = 500, offset: int = 0) -> dict[str, Any]: + with session_scope() as session: + job = session.get(CleaningJob, job_id) + if job is None: + raise HTTPException(status_code=404, detail="Job de limpieza no encontrado") + rows = session.scalars( + select(CleanedRow) + .where(CleanedRow.cleaning_job_id == job_id) + .order_by(CleanedRow.row_index, CleanedRow.id) + .offset(offset) + .limit(limit) + ).all() + return { + "job_id": job_id, + "rows": [row.row_json for row in rows], + "limit": limit, + "offset": offset, + } diff --git a/services/cleaning/domain/__init__.py b/services/cleaning/domain/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/application/csv_cleaning.py b/services/cleaning/domain/csv_cleaning.py similarity index 100% rename from application/csv_cleaning.py rename to services/cleaning/domain/csv_cleaning.py diff --git a/services/cleaning/infrastructure/__init__.py b/services/cleaning/infrastructure/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/services/cleaning/infrastructure/database.py b/services/cleaning/infrastructure/database.py new file mode 100644 index 0000000..73ef768 --- /dev/null +++ b/services/cleaning/infrastructure/database.py @@ -0,0 +1,127 @@ +"""Persistencia del servicio sobre la base de datos de integracion.""" +from __future__ import annotations + +import os +import uuid +from contextlib import contextmanager +from datetime import datetime, timezone +from typing import Any, Iterator + +from sqlalchemy import ( + JSON, + DateTime, + ForeignKey, + Integer, + String, + Text, + create_engine, + select, + text, +) +from sqlalchemy.orm import DeclarativeBase, Mapped, Session, mapped_column, sessionmaker + + +DATABASE_URL = os.getenv( + "DATABASE_URL", + "postgresql+psycopg://ipsextractor:ipsextractor@database:5432/ipsextractor", +) + +engine = create_engine(DATABASE_URL, pool_pre_ping=True) +SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False) + + +class Base(DeclarativeBase): + pass + + +def utcnow() -> datetime: + return datetime.now(timezone.utc) + + +def new_id() -> str: + return str(uuid.uuid4()) + + +class ExtractionBatch(Base): + __tablename__ = "extraction_batches" + + id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_id) + status: Mapped[str] = mapped_column(String(32), default="pending", index=True) + root_path: Mapped[str | None] = mapped_column(Text, nullable=True) + request: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + error: Mapped[str | None] = mapped_column(Text, nullable=True) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + + +class ExtractionRow(Base): + __tablename__ = "extraction_rows" + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + batch_id: Mapped[str] = mapped_column(ForeignKey("extraction_batches.id"), index=True) + radicado: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True) + source_pdf: Mapped[str | None] = mapped_column(Text, nullable=True) + row_index: Mapped[int] = mapped_column(Integer, default=0) + row_json: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + raw_json: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + + +class CleaningJob(Base): + __tablename__ = "cleaning_jobs" + + id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_id) + extraction_batch_id: Mapped[str] = mapped_column(ForeignKey("extraction_batches.id"), index=True) + status: Mapped[str] = mapped_column(String(32), default="pending", index=True) + request: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + log: Mapped[list[str]] = mapped_column(JSON, default=list) + error: Mapped[str | None] = mapped_column(Text, nullable=True) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + + +class CleanedRow(Base): + __tablename__ = "cleaned_rows" + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + cleaning_job_id: Mapped[str] = mapped_column(ForeignKey("cleaning_jobs.id"), index=True) + extraction_batch_id: Mapped[str] = mapped_column(ForeignKey("extraction_batches.id"), index=True) + row_index: Mapped[int] = mapped_column(Integer, default=0) + row_json: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + + +def init_db() -> None: + if DATABASE_URL.startswith("postgresql"): + with engine.begin() as connection: + connection.execute(text("SELECT pg_advisory_lock(74512001)")) + try: + Base.metadata.create_all(bind=connection) + finally: + connection.execute(text("SELECT pg_advisory_unlock(74512001)")) + return + + Base.metadata.create_all(bind=engine) + + +@contextmanager +def session_scope() -> Iterator[Session]: + session = SessionLocal() + try: + yield session + session.commit() + except Exception: + session.rollback() + raise + finally: + session.close() + + +def get_extraction_rows(session: Session, batch_id: str) -> list[ExtractionRow]: + return list( + session.scalars( + select(ExtractionRow) + .where(ExtractionRow.batch_id == batch_id) + .order_by(ExtractionRow.row_index, ExtractionRow.id) + ) + ) diff --git a/services/cleaning/requirements.txt b/services/cleaning/requirements.txt new file mode 100644 index 0000000..4c07d5e --- /dev/null +++ b/services/cleaning/requirements.txt @@ -0,0 +1,7 @@ +fastapi==0.115.6 +numpy==2.2.6 +pandas==2.2.3 +psycopg[binary]==3.2.3 +pydantic==2.11.9 +SQLAlchemy==2.0.36 +uvicorn[standard]==0.32.1 diff --git a/services/extraction/Dockerfile b/services/extraction/Dockerfile new file mode 100644 index 0000000..a4e2781 --- /dev/null +++ b/services/extraction/Dockerfile @@ -0,0 +1,22 @@ +FROM python:3.11-slim + +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + PYTHONPATH=/app + +WORKDIR /app + +RUN apt-get update \ + && apt-get install -y --no-install-recommends poppler-utils libgl1 libglib2.0-0 \ + && rm -rf /var/lib/apt/lists/* + +COPY requirements.txt /app/requirements.txt +RUN pip install --no-cache-dir -r requirements.txt + +COPY domain /app/domain +COPY core /app/core +COPY infrastructure /app/infrastructure +COPY api /app/api + +EXPOSE 8001 +CMD ["uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8001"] diff --git a/services/extraction/README.md b/services/extraction/README.md new file mode 100644 index 0000000..6f6e5f5 --- /dev/null +++ b/services/extraction/README.md @@ -0,0 +1,35 @@ +# IPSExtractor Extraction Service + +Microservicio independiente de extraccion. Esta carpeta puede vivir como repo +propio: contiene su API REST, dominio, infraestructura, dependencias y +Dockerfile sin importar codigo de otros servicios. + +## Estructura + +```text +api/ Endpoints FastAPI +domain/ Logica de extraccion, prompts, estimacion y normalizacion +core/ Utilidades propias de PDF, texto y CSV +infrastructure/ Persistencia SQLAlchemy del servicio +requirements.txt +Dockerfile +``` + +## Desarrollo local + +```bash +pip install -r requirements.txt +uvicorn api.main:app --host 0.0.0.0 --port 8001 +``` + +## Endpoints principales + +- `GET /health` +- `GET /config` +- `GET /models` +- `POST /schemas` +- `POST /estimate` +- `POST /batches` +- `GET /batches/{batch_id}` +- `GET /batches/{batch_id}/rows` + diff --git a/services/extraction/api/__init__.py b/services/extraction/api/__init__.py new file mode 100644 index 0000000..492c7eb --- /dev/null +++ b/services/extraction/api/__init__.py @@ -0,0 +1 @@ +"""Servicio REST de extraccion.""" diff --git a/services/extraction/api/main.py b/services/extraction/api/main.py new file mode 100644 index 0000000..442341a --- /dev/null +++ b/services/extraction/api/main.py @@ -0,0 +1,327 @@ +"""API REST del microservicio de extraccion.""" +from __future__ import annotations + +from contextlib import asynccontextmanager +from typing import Any + +from fastapi import FastAPI, HTTPException +from pydantic import BaseModel, Field +from sqlalchemy import select + +from domain.batch_processor import BatchProcessor +from domain.config import extractor_llm_config, normalizer_llm_config, settings +from domain.config import get_allowed_document_types +from domain.estimator import Estimator +from domain.service import InvoiceProcessingService +from domain.strategies import OpenAICompatibleExtractor +from infrastructure.database import ( + ExtractionBatch, + ExtractionRow, + init_db, + session_scope, + utcnow, +) +from infrastructure.serialization import flatten_extraction_result + + +def build_processing_service() -> InvoiceProcessingService: + extractor_config = extractor_llm_config() + extractor = OpenAICompatibleExtractor( + api_key=str(extractor_config["api_key"]), + base_url=str(extractor_config["base_url"]), + model=str(extractor_config["model"]), + ) + return InvoiceProcessingService( + extractor, + batch_processor_factory=lambda service: BatchProcessor(service), + estimator_factory=lambda service: Estimator(service), + ) + + +processing_service = build_processing_service() + + +@asynccontextmanager +async def lifespan(app: FastAPI): + init_db() + yield + + +app = FastAPI( + title="IPSExtractor Extraction Service", + version="1.0.0", + lifespan=lifespan, +) + + +class DocumentType(BaseModel): + name: str + keyword: str = "" + + +class SchemaRequest(BaseModel): + document_path: str | None = None + root_path: str | None = None + doc_types: list[DocumentType] | None = None + instruction: str | None = None + tiff_page_range: tuple[int, int] | None = None + structure_mode: str = "radicados" + + +class ExtractionRequest(BaseModel): + root_path: str + keywords: list[str] | None = None + doc_types: list[DocumentType] | None = None + fields: list[str] = Field(default_factory=list) + alias_map: dict[str, str] = Field(default_factory=dict) + instruction: str | None = None + tiff_page_range: tuple[int, int] | None = None + structure_mode: str = "radicados" + retry_on_empty: bool = False + radicado_filter: list[str] | None = None + + +class EstimateRequest(BaseModel): + root_path: str + keywords: list[str] | None = None + doc_types: list[DocumentType] | None = None + schema_map: dict[str, str] | None = Field(default=None, alias="schema") + selected_fields: list[str] | None = None + schema_by_type: dict[str, dict[str, str]] | None = None + selected_paths: list[str] | None = None + structure_mode: str = "radicados" + + +class ExtractionBatchRead(BaseModel): + id: str + status: str + root_path: str | None + request: dict[str, Any] + error: str | None + created_at: str + updated_at: str + + +def serialize_batch(batch: ExtractionBatch) -> dict[str, Any]: + return { + "id": batch.id, + "status": batch.status, + "root_path": batch.root_path, + "request": batch.request, + "error": batch.error, + "created_at": batch.created_at.isoformat(), + "updated_at": batch.updated_at.isoformat(), + } + + +@app.get("/health") +def health() -> dict[str, str]: + return {"status": "ok"} + + +@app.get("/config") +def get_config() -> dict[str, Any]: + extractor_config = extractor_llm_config() + normalizer_config = normalizer_llm_config() + return { + "extractor": { + "provider": extractor_config["provider"], + "base_url": extractor_config["base_url"], + "model": extractor_config["model"], + "timeout_s": extractor_config["timeout_s"], + "api_key_configured": bool(str(extractor_config["api_key"]).strip()), + }, + "normalizer": { + "provider": normalizer_config["provider"], + "base_url": normalizer_config["base_url"], + "model": normalizer_config["model"], + "timeout_s": normalizer_config["timeout_s"], + "api_key_configured": bool(str(normalizer_config["api_key"]).strip()), + }, + "processing": { + "concurrency_limit": settings.processing.concurrency_limit, + "allowed_document_types": list(get_allowed_document_types()), + }, + "imaging": { + "pdf_dpi": settings.imaging.pdf_dpi, + "pdf_thread_count": settings.imaging.pdf_thread_count, + "tiff_pages": settings.imaging.tiff_pages, + "poppler_path": settings.imaging.poppler_path, + }, + } + + +@app.get("/models") +def list_models() -> dict[str, Any]: + extractor_config = extractor_llm_config() + normalizer_config = normalizer_llm_config() + try: + models = processing_service.extractor.llm.list_models() + return { + "available": models, + "configured_extractor_model": extractor_config["model"], + "configured_normalizer_model": normalizer_config["model"], + "source": extractor_config["base_url"], + } + except Exception as exc: + return { + "available": [], + "configured_extractor_model": extractor_config["model"], + "configured_normalizer_model": normalizer_config["model"], + "source": extractor_config["base_url"], + "warning": f"No se pudieron consultar modelos del backend: {exc}", + } + + +@app.post("/schemas") +def build_schema(request: SchemaRequest) -> dict[str, Any]: + """Descubre campos disponibles para que el cliente API elija `fields`.""" + if request.document_path: + schema, sample = processing_service.build_schema_from_first( + request.document_path, + instruction=request.instruction, + tiff_page_range=request.tiff_page_range, + ) + return {"schema": schema, "sample": sample} + + if request.root_path and request.doc_types: + schema_by_type, sample_by_type, first_radicado = ( + processing_service.build_schema_from_first_for_types( + request.root_path, + [dt.model_dump() for dt in request.doc_types], + instruction=request.instruction, + tiff_page_range=request.tiff_page_range, + structure_mode=request.structure_mode, + ) + ) + return { + "schema_by_type": schema_by_type, + "sample_by_type": sample_by_type, + "first_radicado": first_radicado, + } + + raise HTTPException( + status_code=400, + detail="Envia document_path o root_path con doc_types para descubrir campos.", + ) + + +@app.post("/estimate") +def estimate(request: EstimateRequest) -> dict[str, Any]: + return processing_service.estimate_tokens_and_cost( + request.root_path, + request.keywords, + request.schema_map, + request.selected_fields, + doc_types=[dt.model_dump() for dt in request.doc_types or []] or None, + schema_by_type=request.schema_by_type, + selected_paths=request.selected_paths, + structure_mode=request.structure_mode, + ) + + +@app.post("/batches", response_model=ExtractionBatchRead) +def run_extraction(request: ExtractionRequest) -> dict[str, Any]: + if not request.fields: + raise HTTPException( + status_code=400, + detail="El API no tiene selector visual: envia fields. Usa POST /schemas para descubrirlos.", + ) + + with session_scope() as session: + batch = ExtractionBatch( + status="running", + root_path=request.root_path, + request=request.model_dump(), + updated_at=utcnow(), + ) + session.add(batch) + session.flush() + batch_id = batch.id + + try: + results = processing_service.process_root_with_field_selection( + request.root_path, + request.keywords, + doc_types=[dt.model_dump() for dt in request.doc_types or []] or None, + fields=request.fields, + csv_output_path=None, + instruction_override=request.instruction, + alias_map=request.alias_map, + tiff_page_range=request.tiff_page_range, + structure_mode=request.structure_mode, + retry_on_empty=request.retry_on_empty, + radicado_filter=request.radicado_filter, + ) + + with session_scope() as session: + row_index = 0 + for item in results: + raw = item if isinstance(item, dict) else {"data": item} + rows = flatten_extraction_result(raw, request.fields, request.alias_map) + for row in rows: + row_index += 1 + session.add( + ExtractionRow( + batch_id=batch_id, + radicado=str(raw.get("radicado") or row.get("radicado") or ""), + source_pdf=raw.get("_source_pdf"), + row_index=row_index, + row_json=row, + raw_json=raw, + ) + ) + batch = session.get(ExtractionBatch, batch_id) + if batch is None: + raise RuntimeError("No se encontro el lote recien creado.") + batch.status = "completed" + batch.updated_at = utcnow() + session.flush() + return serialize_batch(batch) + except Exception as exc: + with session_scope() as session: + batch = session.get(ExtractionBatch, batch_id) + if batch is not None: + batch.status = "failed" + batch.error = str(exc) + batch.updated_at = utcnow() + raise HTTPException(status_code=500, detail=str(exc)) from exc + + +@app.get("/batches") +def list_batches() -> list[dict[str, Any]]: + with session_scope() as session: + batches = session.scalars( + select(ExtractionBatch).order_by(ExtractionBatch.created_at.desc()) + ).all() + return [serialize_batch(batch) for batch in batches] + + +@app.get("/batches/{batch_id}") +def get_batch(batch_id: str) -> dict[str, Any]: + with session_scope() as session: + batch = session.get(ExtractionBatch, batch_id) + if batch is None: + raise HTTPException(status_code=404, detail="Lote no encontrado") + return serialize_batch(batch) + + +@app.get("/batches/{batch_id}/rows") +def get_rows(batch_id: str, limit: int = 500, offset: int = 0) -> dict[str, Any]: + with session_scope() as session: + batch = session.get(ExtractionBatch, batch_id) + if batch is None: + raise HTTPException(status_code=404, detail="Lote no encontrado") + rows = session.scalars( + select(ExtractionRow) + .where(ExtractionRow.batch_id == batch_id) + .order_by(ExtractionRow.row_index, ExtractionRow.id) + .offset(offset) + .limit(limit) + ).all() + return { + "batch_id": batch_id, + "rows": [row.row_json for row in rows], + "limit": limit, + "offset": offset, + } diff --git a/core/__init__.py b/services/extraction/core/__init__.py similarity index 100% rename from core/__init__.py rename to services/extraction/core/__init__.py diff --git a/core/csv_utils.py b/services/extraction/core/csv_utils.py similarity index 100% rename from core/csv_utils.py rename to services/extraction/core/csv_utils.py diff --git a/core/pdf_utils.py b/services/extraction/core/pdf_utils.py similarity index 100% rename from core/pdf_utils.py rename to services/extraction/core/pdf_utils.py diff --git a/core/text_utils.py b/services/extraction/core/text_utils.py similarity index 100% rename from core/text_utils.py rename to services/extraction/core/text_utils.py diff --git a/core/utils.py b/services/extraction/core/utils.py similarity index 100% rename from core/utils.py rename to services/extraction/core/utils.py diff --git a/application/__init__.py b/services/extraction/domain/__init__.py similarity index 64% rename from application/__init__.py rename to services/extraction/domain/__init__.py index 46b84e0..ca1c0d6 100644 --- a/application/__init__.py +++ b/services/extraction/domain/__init__.py @@ -1,10 +1,10 @@ -"""Paquete `application`: lógica de alto nivel de extracción y orquestación. +"""Paquete `domain`: lógica de alto nivel de extracción y orquestación. Incluye: - `service`: fachada de procesamiento de facturas. -- `flows`: flujos y utilidades para encabezados/CSV y ejecución paralela. +- `batch_processor`: procesamiento por lotes y orquestación de radicados. - `normalizer`: normalización LLM a un esquema canónico. -- `strategies`: estrategias de extracción (cliente OpenAI/Ollama). +- `strategies`: estrategias de extracción con backends compatibles con Chat Completions. - `profiles`: perfiles por tipo de documento e instrucciones por defecto. - `prompts`: construcción de prompts de extracción/normalización. - `config`: configuración centralizada (Pydantic BaseSettings). diff --git a/application/batch_processor.py b/services/extraction/domain/batch_processor.py similarity index 93% rename from application/batch_processor.py rename to services/extraction/domain/batch_processor.py index bc59df0..9a9cfe1 100644 --- a/application/batch_processor.py +++ b/services/extraction/domain/batch_processor.py @@ -9,17 +9,15 @@ import openai from typing import List, Dict, Any, Optional, Tuple, Callable, Awaitable, TYPE_CHECKING if TYPE_CHECKING: - from application.service import InvoiceProcessingService + from domain.service import InvoiceProcessingService -from application.profiles import get_profile -from application.prompts import PromptBuilder -from application.normalizer import normalize_to_schema -from application.costs import compute_cost_from_usage, accumulate_usage -from application.file_discovery import get_sorted_radicados, get_files_as_radicados -from application.csv_manager import CSVManager -from application.config import settings -from application.app_prefs import get_allowed_document_types -from application.tracking_manager import TrackingManager +from domain.profiles import get_profile +from domain.prompts import PromptBuilder +from domain.normalizer import normalize_to_schema +from domain.costs import compute_cost_from_usage, accumulate_usage +from domain.file_discovery import get_sorted_radicados, get_files_as_radicados +from domain.csv_manager import CSVManager +from domain.config import settings, get_allowed_document_types from core.utils import collect_radicado_docs, find_first_matching_file, collect_documents_flat from core.csv_utils import merge_verbatim_long_strings, rows_from_data @@ -36,6 +34,27 @@ DOCUMENT_TYPE_HINT_ALIASES = { "PASS", } +class BatchEventSink: + """Recibe eventos del lote sin depender de un servidor externo.""" + + def start_batch(self, *args, **kwargs): + return None + + def update_progress(self, *args, **kwargs): + return None + + def log(self, *args, **kwargs): + return None + + def report_phase(self, *args, **kwargs): + return None + + def report_error(self, *args, **kwargs): + return None + + def complete_batch(self, *args, **kwargs): + return None + class BatchProcessor: def __init__(self, service: 'InvoiceProcessingService'): """ @@ -44,11 +63,11 @@ class BatchProcessor: """ self.service = service self.csv_manager = CSVManager() - self.tracking_manager = TrackingManager() + self.event_sink = BatchEventSink() self._pause_event = asyncio.Event() self._pause_event.set() # No pausado inicialmente self._stop_requested = False - # Contadores acumulados para tracking web + # Contadores acumulados para progreso de lote self._track_tokens: int = 0 self._track_cost_usd: float = 0.0 self._track_success: int = 0 @@ -77,7 +96,7 @@ class BatchProcessor: Delega en métodos específicos para multi-documento o single-document. """ - # Resetear estado de pausa/parada y contadores de tracking + # Resetear estado de pausa/parada y contadores de progreso self._pause_event.set() self._stop_requested = False self._track_tokens = 0 @@ -85,7 +104,7 @@ class BatchProcessor: self._track_success = 0 self._track_errors = 0 - # Notificar inicio a tracking si es procesamiento masivo + # Notificar inicio si es procesamiento masivo if fields: # Intentar estimar total de radicados try: @@ -98,7 +117,7 @@ class BatchProcessor: filter_set = {r.strip() for r in radicado_filter if r.strip()} temp_rads = [r for r in temp_rads if r in filter_set] - self.tracking_manager.start_batch( + self.event_sink.start_batch( root_path, len(temp_rads), config={ @@ -108,7 +127,7 @@ class BatchProcessor: csv_output_path=csv_output_path, ) except Exception as e: - logger.error(f"Error al iniciar tracking: {e}") + logger.error(f"Error al iniciar progreso del lote: {e}") # 1. Preparación de Schemas (Fase 1 Multi-doc) if doc_types and not fields: @@ -286,8 +305,8 @@ class BatchProcessor: prompt_t = PromptBuilder.build(selected_paths_for_type, {}, profile_name=prof.get('name', 'invoice')) # Llamada ASYNC - Fase: Extracción - self.tracking_manager.report_phase("extract", rad_id) - self.tracking_manager.log(f"Iniciando extracción IA", level="info", radicado=rad_id) + self.event_sink.report_phase("extract", rad_id) + self.event_sink.log(f"Iniciando extracción IA", level="info", radicado=rad_id) data, usage_ext = await self.service.extract_full_from_pdf_async(match, instruction=prompt_t, tiff_page_range=tiff_page_range) self._accumulate_usage(agg_extract_usage, usage_ext) @@ -299,8 +318,8 @@ class BatchProcessor: data = {"radicado": rad_id, "data": data} # Normalización en thread aparte - Fase: Normalización - self.tracking_manager.report_phase("normalize", rad_id) - self.tracking_manager.log(f"Extracción OK — normalizando campos", level="info", radicado=rad_id) + self.event_sink.report_phase("normalize", rad_id) + self.event_sink.log(f"Extracción OK — normalizando campos", level="info", radicado=rad_id) norm = await asyncio.to_thread( normalize_to_schema, data, @@ -478,8 +497,8 @@ class BatchProcessor: logger.info(f"Procesando radicado {entry['radicado']} [{idx}/{total}]", extra={"radicado": entry['radicado'], "pdf": entry['path']}) # Async Extract - Fase: Extracción - self.tracking_manager.report_phase("extract", entry['radicado']) - self.tracking_manager.log(f"Iniciando extracción IA", level="info", radicado=entry['radicado']) + self.event_sink.report_phase("extract", entry['radicado']) + self.event_sink.log(f"Iniciando extracción IA", level="info", radicado=entry['radicado']) data, usage_ext = await self.service.extract_full_from_pdf_async(entry["path"], instruction=prompt, tiff_page_range=tiff_page_range) self._report_usage(on_usage, entry['radicado'], "extract", usage_ext) @@ -491,8 +510,8 @@ class BatchProcessor: data["_source_pdf"] = entry["path"] # Normalizar (Thread) - Fase: Normalización - self.tracking_manager.report_phase("normalize", entry['radicado']) - self.tracking_manager.log(f"Extracción OK — normalizando campos", level="info", radicado=entry['radicado']) + self.event_sink.report_phase("normalize", entry['radicado']) + self.event_sink.log(f"Extracción OK — normalizando campos", level="info", radicado=entry['radicado']) norm = await asyncio.to_thread( normalize_to_schema, data, paths=selected_fields, profile_name=prof.get('name', 'invoice') @@ -714,25 +733,25 @@ class BatchProcessor: except Exception: pass - # Notificar a tracking web con contadores reales acumulados + # Notificar progreso con contadores reales acumulados try: self._track_success = idx - self._track_errors # Log de radicado completado exitosamente if rad and self._track_errors < idx: - self.tracking_manager.log( + self.event_sink.log( f"Radicado completado ({self._track_success}/{total} exitosos)", level="success", radicado=str(rad) ) if idx >= total: - self.tracking_manager.complete_batch( + self.event_sink.complete_batch( success_count=self._track_success, error_count=self._track_errors, total_tokens=self._track_tokens, total_cost_usd=self._track_cost_usd, ) else: - self.tracking_manager.update_progress( + self.event_sink.update_progress( current_index=idx, success_count=self._track_success, error_count=self._track_errors, @@ -741,7 +760,7 @@ class BatchProcessor: status="processing" ) except Exception as e: - logger.debug(f"Error al notificar progreso a tracking: {e}") + logger.debug(f"Error al notificar progreso del lote: {e}") def _accumulate_usage(self, agg, new_usage): accumulate_usage(agg, new_usage) @@ -754,11 +773,11 @@ class BatchProcessor: if phase == "extract": model = getattr(self.service.extractor, "model", None) elif phase == "normalize": - model = "gpt-4.1-mini" + model = settings.api.normalizer_model cost = compute_cost_from_usage(usage, phase, model=model) - # Acumular tokens y costos para el tracking web + # Acumular tokens y costos para el progreso del lote prompt_tokens = int(usage.get("prompt_tokens") or 0) completion_tokens = int(usage.get("completion_tokens") or 0) self._track_tokens += prompt_tokens + completion_tokens @@ -872,15 +891,15 @@ class BatchProcessor: if pdf_path: error_obj["_source_pdf"] = pdf_path logger.exception(f"Error procesando radicado {rad}", extra={"radicado": rad}) - # Acumular error en el contador de tracking + # Acumular error en el contador de lote self._track_errors += 1 error_message = str(e) try: - self.tracking_manager.report_error(f"[{rad}] {error_message}") + self.event_sink.report_error(f"[{rad}] {error_message}") except Exception: pass try: - self.tracking_manager.log( + self.event_sink.log( message=f"Error procesando radicado: {error_message}", level="error", radicado=str(rad) diff --git a/services/extraction/domain/config.py b/services/extraction/domain/config.py new file mode 100644 index 0000000..002e399 --- /dev/null +++ b/services/extraction/domain/config.py @@ -0,0 +1,170 @@ +"""Configuración centralizada de la aplicación. + +Usa Pydantic BaseSettings para cargar valores por defecto y variables de entorno. +Se lee automáticamente un archivo `.env` en la raíz del proyecto si existe. +""" +from __future__ import annotations + +from typing import Optional, Tuple +import logging +import os + +# Cargar variables desde .env en el entorno (para que otros módulos que usan os.environ las vean) +try: + from dotenv import load_dotenv # type: ignore + # En desarrollo, dar precedencia al archivo .env sobre variables previas del entorno + load_dotenv(override=True) +except Exception: + pass + +from pydantic import Field +from pydantic_settings import BaseSettings, SettingsConfigDict +from pydantic import AliasChoices + + +class ApiSettings(BaseSettings): + """Backends LLM compatibles con la API de OpenAI. + + Extraccion y normalizacion tienen credenciales independientes para poder + mezclar proveedores, por ejemplo Grok para vision y OpenAI para normalizar. + """ + extractor_provider: str = Field("custom", validation_alias=AliasChoices("EXTRACTOR_PROVIDER")) + extractor_base_url: str = Field("", validation_alias=AliasChoices("EXTRACTOR_BASE_URL")) + extractor_api_key: str = Field("", validation_alias=AliasChoices("EXTRACTOR_API_KEY")) + extractor_model: str = Field("", validation_alias=AliasChoices("EXTRACTOR_MODEL")) + + normalizer_provider: str = Field("custom", validation_alias=AliasChoices("NORMALIZER_PROVIDER")) + normalizer_base_url: str = Field("", validation_alias=AliasChoices("NORMALIZER_BASE_URL")) + normalizer_api_key: str = Field("", validation_alias=AliasChoices("NORMALIZER_API_KEY")) + normalizer_model: str = Field("", validation_alias=AliasChoices("NORMALIZER_MODEL")) + + request_timeout_s: int = Field(120, validation_alias=AliasChoices("EXTRACTOR_TIMEOUT_S", "LLM_TIMEOUT_S")) + normalizer_timeout_s: int = Field(0, validation_alias=AliasChoices("NORMALIZER_TIMEOUT_S")) + model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore") + + @property + def base_url(self) -> str: + """Alias de compatibilidad para codigo anterior: backend de extraccion.""" + return self.extractor_base_url + + @property + def api_key(self) -> str: + """Alias de compatibilidad para codigo anterior: API key de extraccion.""" + return self.extractor_api_key + + @property + def model(self) -> str: + """Alias de compatibilidad para codigo anterior: modelo de extraccion.""" + return self.extractor_model + + @property + def effective_normalizer_base_url(self) -> str: + return self.normalizer_base_url or self.extractor_base_url + + @property + def effective_normalizer_api_key(self) -> str: + return self.normalizer_api_key or self.extractor_api_key + + @property + def effective_normalizer_timeout_s(self) -> int: + return self.normalizer_timeout_s or self.request_timeout_s + + +class ImagingSettings(BaseSettings): + """Parámetros de conversión de documentos a imágenes.""" + poppler_path: Optional[str] = Field(None, validation_alias=AliasChoices("POPPLER_PATH")) + pdf_dpi: int = Field(600, validation_alias=AliasChoices("PDF_DPI")) + pdf_thread_count: int = Field(10, validation_alias=AliasChoices("PDF_THREAD_COUNT")) + tiff_pages: Optional[Tuple[int, int]] = Field((2, 3), validation_alias=AliasChoices("TIFF_PAGES")) + model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore") + + +class RetrySettings(BaseSettings): + """Estrategia de reintentos para llamadas al backend.""" + max_attempts: int = Field(3, validation_alias=AliasChoices("RETRY_MAX_ATTEMPTS")) + wait_min_s: float = Field(1.0, validation_alias=AliasChoices("RETRY_WAIT_MIN_S")) + wait_max_s: float = Field(8.0, validation_alias=AliasChoices("RETRY_WAIT_MAX_S")) + model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore") + + +class ProcessingSettings(BaseSettings): + """Configuración de procesamiento y concurrencia.""" + concurrency_limit: int = Field(100, validation_alias=AliasChoices("CONCURRENCY_LIMIT")) + model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore") + + +class AppSettings(BaseSettings): + """Configuración raíz agrupada.""" + api: ApiSettings = ApiSettings() + imaging: ImagingSettings = ImagingSettings() + retry: RetrySettings = RetrySettings() + processing: ProcessingSettings = ProcessingSettings() + csv_delimiter: str = Field("|", validation_alias=AliasChoices("CSV_DELIMITER")) + model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore") + + +settings = AppSettings() + + +def extractor_llm_config() -> dict[str, str | int]: + """Configuracion efectiva para llamadas de extraccion.""" + api = settings.api + return { + "provider": api.extractor_provider or "custom", + "api_key": api.extractor_api_key, + "base_url": api.extractor_base_url, + "model": api.extractor_model, + "timeout_s": api.request_timeout_s, + } + + +def normalizer_llm_config() -> dict[str, str | int]: + """Configuracion efectiva para llamadas de normalizacion.""" + api = settings.api + return { + "provider": api.normalizer_provider or "custom", + "api_key": api.effective_normalizer_api_key, + "base_url": api.effective_normalizer_base_url, + "model": api.normalizer_model, + "timeout_s": api.effective_normalizer_timeout_s, + } + + +# Tipos documentales válidos por defecto para el campo especial `documento_forzate`. +# NIT se excluye explícitamente porque no debe tratarse como tipo de documento personal. +DEFAULT_ALLOWED_DOCUMENT_TYPES: Tuple[str, ...] = ( + "CC", + "TI", + "CE", + "RC", + "PA", + "PPT", + "PEP", + "EPT", + "NUIP", +) + +# Alias de compatibilidad. +ALLOWED_DOCUMENT_TYPES: Tuple[str, ...] = DEFAULT_ALLOWED_DOCUMENT_TYPES + + +def normalize_document_types(values: object) -> tuple[str, ...]: + """Normaliza tipos documentales configurables para API/servicios.""" + raw_values = values if isinstance(values, (list, tuple, set)) else [] + normalized: list[str] = [] + seen: set[str] = set() + for item in raw_values: + text = str(item or "").strip().upper() + if not text or text == "NIT" or text in seen: + continue + seen.add(text) + normalized.append(text) + return tuple(normalized or DEFAULT_ALLOWED_DOCUMENT_TYPES) + + +def get_allowed_document_types() -> tuple[str, ...]: + """Devuelve tipos documentales válidos desde `ALLOWED_DOCUMENT_TYPES` o defaults.""" + raw = os.getenv("ALLOWED_DOCUMENT_TYPES", "") + if not raw.strip(): + return DEFAULT_ALLOWED_DOCUMENT_TYPES + return normalize_document_types([part.strip() for part in raw.split(",")]) diff --git a/application/costs.py b/services/extraction/domain/costs.py similarity index 94% rename from application/costs.py rename to services/extraction/domain/costs.py index 579847f..a1aa85a 100644 --- a/application/costs.py +++ b/services/extraction/domain/costs.py @@ -34,10 +34,16 @@ MODEL_PRICING = { }, } -# Legacy mapping o fase por defecto (si no se especifica modelo) +ZERO_PRICING = { + "input": 0.0, + "cached_input": 0.0, + "output": 0.0, +} + +# Fallback por fase cuando no hay tarifa conocida para el modelo configurado. PRICING = { - "extract": MODEL_PRICING["gpt-5.4"], - "normalize": MODEL_PRICING["gpt-4.1-mini"], + "extract": ZERO_PRICING, + "normalize": ZERO_PRICING, } diff --git a/application/csv_manager.py b/services/extraction/domain/csv_manager.py similarity index 100% rename from application/csv_manager.py rename to services/extraction/domain/csv_manager.py diff --git a/application/estimator.py b/services/extraction/domain/estimator.py similarity index 91% rename from application/estimator.py rename to services/extraction/domain/estimator.py index e8637de..a64e6ec 100644 --- a/application/estimator.py +++ b/services/extraction/domain/estimator.py @@ -6,13 +6,14 @@ import os from typing import List, Dict, Any, Optional, TYPE_CHECKING if TYPE_CHECKING: - from application.service import InvoiceProcessingService + from domain.service import InvoiceProcessingService -from application.profiles import get_profile -from application.prompts import PromptBuilder -from application.normalizer import normalize_to_schema -from application.costs import compute_cost_from_usage, sum_costs, accumulate_usage -from application.file_discovery import get_sorted_radicados, find_file_for_type_in_radicados, get_files_as_radicados +from domain.profiles import get_profile +from domain.prompts import PromptBuilder +from domain.normalizer import normalize_to_schema +from domain.costs import compute_cost_from_usage, sum_costs, accumulate_usage +from domain.config import settings +from domain.file_discovery import get_sorted_radicados, find_file_for_type_in_radicados, get_files_as_radicados from core.utils import collect_radicado_docs, collect_documents_flat class Estimator: @@ -112,8 +113,7 @@ class Estimator: def _build_result(self, usage_ext, usage_norm, total_docs, first_radicado): model_ext = getattr(self.service.extractor, "model", None) cost_ext = compute_cost_from_usage(usage_ext, "extract", model=model_ext) - # El normalizador está hardcodeado a gpt-4.1-mini en normalizer.py - cost_norm = compute_cost_from_usage(usage_norm, "normalize", model="gpt-4.1-mini") + cost_norm = compute_cost_from_usage(usage_norm, "normalize", model=settings.api.normalizer_model) per_doc_cost = sum_costs(cost_ext, cost_norm) estimated_total = {k: round(v * total_docs, 6) for k, v in per_doc_cost.items()} diff --git a/application/file_discovery.py b/services/extraction/domain/file_discovery.py similarity index 100% rename from application/file_discovery.py rename to services/extraction/domain/file_discovery.py diff --git a/application/llm_client.py b/services/extraction/domain/llm_client.py similarity index 87% rename from application/llm_client.py rename to services/extraction/domain/llm_client.py index 63b8692..185d372 100644 --- a/application/llm_client.py +++ b/services/extraction/domain/llm_client.py @@ -7,15 +7,15 @@ import logging from openai import OpenAI, AsyncOpenAI from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type -from .config import settings +from .config import extractor_llm_config, settings class LLMClient: """Cliente unificado para llamadas Chat Completions (SDK OpenAI). - - Centraliza api_key, base_url, modelo y timeout desde `settings`. + - Centraliza api_key, base_url, modelo y timeout desde configuracion. - Expone helpers para respuestas de texto y JSON. - - Registra métricas mínimas en `application.api_calls` (start/end/error + usage). + - Registra métricas mínimas en `domain.api_calls` (start/end/error + usage). - Aplica reintentos ante timeouts. """ @@ -26,13 +26,26 @@ class LLMClient: default_model: Optional[str] = None, request_timeout_s: Optional[float] = None, ) -> None: - self.api_key = api_key or settings.api.api_key - self.base_url = base_url or settings.api.base_url - self.default_model = default_model or settings.api.model - self.request_timeout_s = request_timeout_s or settings.api.request_timeout_s - self._client = OpenAI(api_key=self.api_key, base_url=self.base_url) - self._async_client = AsyncOpenAI(api_key=self.api_key, base_url=self.base_url) - self._api_logger = logging.getLogger("application.api_calls") + default_config = extractor_llm_config() + self.api_key = api_key if api_key is not None else str(default_config["api_key"]) + self.base_url = base_url if base_url is not None else str(default_config["base_url"]) + self.default_model = default_model if default_model is not None else str(default_config["model"]) + self.request_timeout_s = request_timeout_s or float(default_config["timeout_s"]) + self._client = self._build_sync_client() + self._async_client = self._build_async_client() + self._api_logger = logging.getLogger("domain.api_calls") + + def _build_sync_client(self) -> OpenAI: + kwargs: Dict[str, Any] = {"api_key": self.api_key} + if self.base_url: + kwargs["base_url"] = self.base_url + return OpenAI(**kwargs) + + def _build_async_client(self) -> AsyncOpenAI: + kwargs: Dict[str, Any] = {"api_key": self.api_key} + if self.base_url: + kwargs["base_url"] = self.base_url + return AsyncOpenAI(**kwargs) def update_config( self, @@ -50,8 +63,17 @@ class LLMClient: self.default_model = default_model if request_timeout_s is not None: self.request_timeout_s = request_timeout_s - self._client = OpenAI(api_key=self.api_key, base_url=self.base_url) - self._async_client = AsyncOpenAI(api_key=self.api_key, base_url=self.base_url) + self._client = self._build_sync_client() + self._async_client = self._build_async_client() + + def list_models(self) -> List[str]: + """Lista modelos publicados por el backend compatible con OpenAI.""" + models = self._client.models.list() + return sorted( + str(getattr(model, "id", "")) + for model in getattr(models, "data", []) + if getattr(model, "id", None) + ) @retry( stop=stop_after_attempt(settings.retry.max_attempts), diff --git a/application/logging_config.py b/services/extraction/domain/logging_config.py similarity index 98% rename from application/logging_config.py rename to services/extraction/domain/logging_config.py index 208427c..38e26f9 100644 --- a/application/logging_config.py +++ b/services/extraction/domain/logging_config.py @@ -147,7 +147,7 @@ def configure_logging(level: int = logging.INFO) -> None: root.addHandler(console) # Logger dedicado para normalizador (LLM) - llm_logger = logging.getLogger("application.normalizer") + llm_logger = logging.getLogger("domain.normalizer") # evitar duplicados si ya está configurado has_file = any(isinstance(h, logging.handlers.RotatingFileHandler) for h in llm_logger.handlers) if not has_file: @@ -164,7 +164,7 @@ def configure_logging(level: int = logging.INFO) -> None: llm_logger.propagate = False # No propagar al root para no duplicar en consola/main log si no se desea # Logger dedicado para llamadas a APIs (peticiones y tokens) - api_calls_logger = logging.getLogger("application.api_calls") + api_calls_logger = logging.getLogger("domain.api_calls") has_api_file = any(isinstance(h, logging.handlers.RotatingFileHandler) for h in api_calls_logger.handlers) if not has_api_file: api_handler = logging.handlers.RotatingFileHandler( diff --git a/application/normalizer.py b/services/extraction/domain/normalizer.py similarity index 85% rename from application/normalizer.py rename to services/extraction/domain/normalizer.py index 6f7ab8c..9e73029 100644 --- a/application/normalizer.py +++ b/services/extraction/domain/normalizer.py @@ -1,7 +1,7 @@ """Normalizador LLM: adapta un JSON fuente al esquema canónico de rutas. -Usa la API de OpenAI (Chat Completions) para transformar el JSON devuelto por -el extractor a un objeto que respete exactamente las rutas/campos canónicos +Usa un backend compatible con Chat Completions para transformar el JSON devuelto +por el extractor a un objeto que respete exactamente las rutas/campos canónicos seleccionados por el usuario (incluyendo listas items/*). """ from __future__ import annotations @@ -10,12 +10,22 @@ import logging import time from typing import Any, Dict, List, Optional -from .config import settings +from .config import normalizer_llm_config, settings from .prompts import PromptBuilder from .llm_client import LLMClient -logger = logging.getLogger('application.normalizer') -_llm_client = LLMClient() +logger = logging.getLogger('domain.normalizer') + + +def _build_normalizer_client() -> tuple[LLMClient, dict[str, Any]]: + config = normalizer_llm_config() + client = LLMClient( + api_key=str(config["api_key"]), + base_url=str(config["base_url"]), + default_model=str(config["model"]), + request_timeout_s=float(config["timeout_s"]), + ) + return client, config def _ensure_dict_at_path(root: Dict[str, Any], key: str) -> Dict[str, Any]: @@ -96,10 +106,10 @@ def normalize_to_schema( template = build_canonical_template(paths) # Construcción centralizada de mensajes messages = PromptBuilder.build_normalization_messages(profile_name, template, data) - # Forzar modelo del normalizador a gpt-5-mini - eff_model = "gpt-4.1-mini" - eff_base_url = settings.api.base_url or "https://api.openai.com/v1" - eff_timeout = getattr(settings.api, "request_timeout_s", 30) + llm_client, normalizer_config = _build_normalizer_client() + eff_model = str(normalizer_config["model"]) + eff_base_url = str(normalizer_config["base_url"]) + eff_timeout = float(normalizer_config["timeout_s"]) # log inicio try: logger.info( @@ -118,7 +128,7 @@ def normalize_to_schema( # Llamada centralizada vía LLMClient (JSON estricto) try: - normalized_obj, usage = _llm_client.chat_json( + normalized_obj, usage = llm_client.chat_json( messages=messages, model=eff_model, response_format="json_object", timeout_s=float(eff_timeout), @@ -178,13 +188,11 @@ def normalize_to_schema( "normalized": False, "error": str(ex), "duration_ms": dt_ms, - "model": (eff_model if 'eff_model' in locals() else settings.api.model or "gpt-4.1-nano"), - "base_url": (eff_base_url if 'eff_base_url' in locals() else (settings.api.base_url or "https://api.openai.com/v1")), + "model": (eff_model if 'eff_model' in locals() else settings.api.normalizer_model), + "base_url": (eff_base_url if 'eff_base_url' in locals() else settings.api.effective_normalizer_base_url), "radicado": (data.get("radicado") if isinstance(data, dict) else None), }, ) except Exception: pass return None - - diff --git a/application/profiles.py b/services/extraction/domain/profiles.py similarity index 100% rename from application/profiles.py rename to services/extraction/domain/profiles.py diff --git a/application/prompts.py b/services/extraction/domain/prompts.py similarity index 99% rename from application/prompts.py rename to services/extraction/domain/prompts.py index c196b41..abb30cd 100644 --- a/application/prompts.py +++ b/services/extraction/domain/prompts.py @@ -11,7 +11,7 @@ import json from typing import Dict, List, Any from .profiles import get_profile -from .app_prefs import get_allowed_document_types +from .config import get_allowed_document_types def _build_instructions() -> List[str]: @@ -114,4 +114,3 @@ class PromptBuilder: extra_meta={"tipo_documento": doc_type_name}, ) return json.dumps(payload, indent=2, ensure_ascii=False) - diff --git a/application/service.py b/services/extraction/domain/service.py similarity index 100% rename from application/service.py rename to services/extraction/domain/service.py diff --git a/application/strategies.py b/services/extraction/domain/strategies.py similarity index 79% rename from application/strategies.py rename to services/extraction/domain/strategies.py index 924d0ba..4ee3817 100644 --- a/application/strategies.py +++ b/services/extraction/domain/strategies.py @@ -1,15 +1,14 @@ """Estrategias de extracción de información desde imágenes de documentos. Define la interfaz `InvoiceExtractorStrategy` y una implementación basada en -un endpoint compatible con OpenAI (por ejemplo, Ollama) para modelos -multimodales como `qwen2.5vl`. +un endpoint compatible con la API de OpenAI para modelos multimodales. """ from __future__ import annotations from abc import ABC, abstractmethod from typing import List, Dict, Any, Optional -from .config import settings +from .config import extractor_llm_config from .llm_client import LLMClient import logging import time @@ -30,16 +29,23 @@ class InvoiceExtractorStrategy(ABC): _LOG_EXTRA_BASE = {"dpi": '-', "pdf": '-', "radicado": '-'} -class OllamaExtractor(InvoiceExtractorStrategy): - """Extractor contra un servidor Ollama/OpenAI-compatible. +class OpenAICompatibleExtractor(InvoiceExtractorStrategy): + """Extractor contra un backend compatible con Chat Completions. Parámetros configurables: `api_key`, `base_url` y `model`. """ def __init__(self, api_key: Optional[str] = None, base_url: Optional[str] = None, model: Optional[str] = None, llm_client: Optional[LLMClient] = None) -> None: - self.api_key = api_key or settings.api.api_key - self.base_url = base_url or settings.api.base_url - self.model = model or settings.api.model - self.llm = llm_client or LLMClient(api_key=self.api_key, base_url=self.base_url, default_model=self.model, request_timeout_s=settings.api.request_timeout_s) + config = extractor_llm_config() + self.api_key = api_key if api_key is not None else str(config["api_key"]) + self.base_url = base_url if base_url is not None else str(config["base_url"]) + self.model = model if model is not None else str(config["model"]) + self.request_timeout_s = float(config["timeout_s"]) + self.llm = llm_client or LLMClient( + api_key=self.api_key, + base_url=self.base_url, + default_model=self.model, + request_timeout_s=self.request_timeout_s, + ) self._logger = logging.getLogger(__name__) def _build_messages(self, images_b64: List[str], instruction: str, history: Optional[List[Dict[str, Any]]]) -> List[Dict[str, Any]]: @@ -57,7 +63,7 @@ class OllamaExtractor(InvoiceExtractorStrategy): self._logger.info( "backend_call_start", extra={**_LOG_EXTRA_BASE, "base_url": self.base_url, "model": self.model, - "timeout_s": settings.api.request_timeout_s, "images": len(images_b64), + "timeout_s": self.request_timeout_s, "images": len(images_b64), "prompt_len": prompt_len, "history_len": len(history) if history else 0}, ) self._logger.debug( @@ -69,7 +75,7 @@ class OllamaExtractor(InvoiceExtractorStrategy): self._logger.info( "Extracción completada", extra={**_LOG_EXTRA_BASE, "base_url": self.base_url, "model": self.model, - "timeout_s": settings.api.request_timeout_s, "elapsed_ms": int(elapsed * 1000)}, + "timeout_s": self.request_timeout_s, "elapsed_ms": int(elapsed * 1000)}, ) self._logger.debug(f"Backend OK (elapsed_ms={int(elapsed*1000)})", extra=_LOG_EXTRA_BASE) @@ -77,7 +83,7 @@ class OllamaExtractor(InvoiceExtractorStrategy): self._logger.exception( "Error en llamada al backend", extra={**_LOG_EXTRA_BASE, "base_url": self.base_url, "model": self.model, - "timeout_s": settings.api.request_timeout_s, "elapsed_ms": int(elapsed * 1000)}, + "timeout_s": self.request_timeout_s, "elapsed_ms": int(elapsed * 1000)}, ) self._logger.debug(f"Backend error tras {int(elapsed*1000)} ms", extra=_LOG_EXTRA_BASE) @@ -89,7 +95,7 @@ class OllamaExtractor(InvoiceExtractorStrategy): try: text, usage_dict = self.llm.chat_text( messages=messages, model=self.model, - timeout_s=settings.api.request_timeout_s, component="extractor", + timeout_s=self.request_timeout_s, component="extractor", ) self._log_call_ok(time.time() - start) return {"raw": text, "usage": usage_dict} @@ -105,7 +111,7 @@ class OllamaExtractor(InvoiceExtractorStrategy): try: text, usage_dict = await self.llm.chat_text_async( messages=messages, model=self.model, - timeout_s=settings.api.request_timeout_s, component="extractor", + timeout_s=self.request_timeout_s, component="extractor", ) self._log_call_ok(time.time() - start) return {"raw": text, "usage": usage_dict} @@ -122,4 +128,4 @@ class OllamaExtractor(InvoiceExtractorStrategy): if model is not None: self.model = model # actualizar cliente LLM con config actual - self.llm.update_config(api_key=self.api_key, base_url=self.base_url, default_model=self.model, request_timeout_s=settings.api.request_timeout_s) + self.llm.update_config(api_key=self.api_key, base_url=self.base_url, default_model=self.model, request_timeout_s=self.request_timeout_s) diff --git a/services/extraction/infrastructure/__init__.py b/services/extraction/infrastructure/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/services/extraction/infrastructure/database.py b/services/extraction/infrastructure/database.py new file mode 100644 index 0000000..73ef768 --- /dev/null +++ b/services/extraction/infrastructure/database.py @@ -0,0 +1,127 @@ +"""Persistencia del servicio sobre la base de datos de integracion.""" +from __future__ import annotations + +import os +import uuid +from contextlib import contextmanager +from datetime import datetime, timezone +from typing import Any, Iterator + +from sqlalchemy import ( + JSON, + DateTime, + ForeignKey, + Integer, + String, + Text, + create_engine, + select, + text, +) +from sqlalchemy.orm import DeclarativeBase, Mapped, Session, mapped_column, sessionmaker + + +DATABASE_URL = os.getenv( + "DATABASE_URL", + "postgresql+psycopg://ipsextractor:ipsextractor@database:5432/ipsextractor", +) + +engine = create_engine(DATABASE_URL, pool_pre_ping=True) +SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False) + + +class Base(DeclarativeBase): + pass + + +def utcnow() -> datetime: + return datetime.now(timezone.utc) + + +def new_id() -> str: + return str(uuid.uuid4()) + + +class ExtractionBatch(Base): + __tablename__ = "extraction_batches" + + id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_id) + status: Mapped[str] = mapped_column(String(32), default="pending", index=True) + root_path: Mapped[str | None] = mapped_column(Text, nullable=True) + request: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + error: Mapped[str | None] = mapped_column(Text, nullable=True) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + + +class ExtractionRow(Base): + __tablename__ = "extraction_rows" + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + batch_id: Mapped[str] = mapped_column(ForeignKey("extraction_batches.id"), index=True) + radicado: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True) + source_pdf: Mapped[str | None] = mapped_column(Text, nullable=True) + row_index: Mapped[int] = mapped_column(Integer, default=0) + row_json: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + raw_json: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + + +class CleaningJob(Base): + __tablename__ = "cleaning_jobs" + + id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_id) + extraction_batch_id: Mapped[str] = mapped_column(ForeignKey("extraction_batches.id"), index=True) + status: Mapped[str] = mapped_column(String(32), default="pending", index=True) + request: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + log: Mapped[list[str]] = mapped_column(JSON, default=list) + error: Mapped[str | None] = mapped_column(Text, nullable=True) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + + +class CleanedRow(Base): + __tablename__ = "cleaned_rows" + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + cleaning_job_id: Mapped[str] = mapped_column(ForeignKey("cleaning_jobs.id"), index=True) + extraction_batch_id: Mapped[str] = mapped_column(ForeignKey("extraction_batches.id"), index=True) + row_index: Mapped[int] = mapped_column(Integer, default=0) + row_json: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + + +def init_db() -> None: + if DATABASE_URL.startswith("postgresql"): + with engine.begin() as connection: + connection.execute(text("SELECT pg_advisory_lock(74512001)")) + try: + Base.metadata.create_all(bind=connection) + finally: + connection.execute(text("SELECT pg_advisory_unlock(74512001)")) + return + + Base.metadata.create_all(bind=engine) + + +@contextmanager +def session_scope() -> Iterator[Session]: + session = SessionLocal() + try: + yield session + session.commit() + except Exception: + session.rollback() + raise + finally: + session.close() + + +def get_extraction_rows(session: Session, batch_id: str) -> list[ExtractionRow]: + return list( + session.scalars( + select(ExtractionRow) + .where(ExtractionRow.batch_id == batch_id) + .order_by(ExtractionRow.row_index, ExtractionRow.id) + ) + ) diff --git a/services/extraction/infrastructure/serialization.py b/services/extraction/infrastructure/serialization.py new file mode 100644 index 0000000..275f368 --- /dev/null +++ b/services/extraction/infrastructure/serialization.py @@ -0,0 +1,40 @@ +"""Helpers para serializar resultados extraidos como filas de base de datos.""" +from __future__ import annotations + +from typing import Any + +from core.csv_utils import rows_from_data + + +def expand_alias_map(alias_map: dict[str, str]) -> dict[str, str]: + expanded = dict(alias_map or {}) + for original, alias in list(expanded.items()): + if "/*/" in original: + tail = original.split("/*/", 1)[1] + if tail: + expanded.setdefault(tail, alias) + return expanded + + +def flatten_extraction_result( + data: dict[str, Any], + fields: list[str] | None, + alias_map: dict[str, str] | None = None, +) -> list[dict[str, Any]]: + rows, _header = rows_from_data(data, allowed_paths=fields) + if not alias_map: + return rows + + aliases = expand_alias_map(alias_map) + aliased: list[dict[str, Any]] = [] + for row in rows: + new_row: dict[str, Any] = {} + for key, value in row.items(): + new_key = aliases.get(key, key) + if new_key in new_row and new_row[new_key] not in (None, ""): + if value not in (None, "") and str(value) != str(new_row[new_key]): + new_row[new_key] = f"{new_row[new_key]}, {value}" + else: + new_row[new_key] = value + aliased.append(new_row) + return aliased diff --git a/services/extraction/requirements.txt b/services/extraction/requirements.txt new file mode 100644 index 0000000..ce319a4 --- /dev/null +++ b/services/extraction/requirements.txt @@ -0,0 +1,12 @@ +coloredlogs==15.0.1 +fastapi==0.115.6 +openai==2.1.0 +pdf2image==1.17.0 +pillow==11.3.0 +psycopg[binary]==3.2.3 +pydantic==2.11.9 +pydantic-settings==2.11.0 +python-dotenv==1.1.1 +SQLAlchemy==2.0.36 +tenacity==9.1.2 +uvicorn[standard]==0.32.1 diff --git a/tests/conftest.py b/tests/conftest.py deleted file mode 100644 index 94e545f..0000000 --- a/tests/conftest.py +++ /dev/null @@ -1,13 +0,0 @@ -from __future__ import annotations - -import sys -from pathlib import Path - - -# Asegurar que el directorio raíz del proyecto esté en sys.path para imports como `from application...` -ROOT = Path(__file__).resolve().parents[1] -root_str = str(ROOT) -if root_str not in sys.path: - sys.path.insert(0, root_str) - - diff --git a/tests/test_csv_cleaning.py b/tests/test_csv_cleaning.py deleted file mode 100644 index b88afb5..0000000 --- a/tests/test_csv_cleaning.py +++ /dev/null @@ -1,347 +0,0 @@ -from __future__ import annotations - -import pandas as pd - -from application.csv_cleaning import ( - apply_rules, - homologate_doc_type, - split_doc_tipo_y_numero_texto, -) - - -def test_homologate_doc_type_aliases() -> None: - cases = { - "Cedula": "CC", - "C.C": "CC", - "C C": "CC", - "Registro Civil": "RC", - "Salvoconducto": "SC", - "TC": "RC", - "C\u00e9dula de Extranjeria": "CE", - } - - for raw, expected in cases.items(): - assert homologate_doc_type(raw) == expected - - -def test_split_doc_type_with_long_names() -> None: - cases = { - "Cedula 123.456": ("CC", "123456"), - "Registro Civil: 001-23": ("RC", "00123"), - "Permiso especial de permanencia 999": ("PE", "999"), - "Permiso por proteccion Temporal / 555": ("PT", "555"), - } - - for raw, expected in cases.items(): - assert split_doc_tipo_y_numero_texto(raw) == expected - - -def test_split_doc_type_with_abbreviations_and_noise() -> None: - cases = { - "C.C-123": ("CC", "123"), - "TL_987": ("TI", "987"), - "CE/1.234.567": ("CE", "1234567"), - "T1 # 44": ("TI", "44"), - } - - for raw, expected in cases.items(): - assert split_doc_tipo_y_numero_texto(raw) == expected - - -def test_split_unknown_type_preserves_clean_uppercase() -> None: - assert split_doc_tipo_y_numero_texto("ABC Especial: 123-45") == ( - "ABC ESPECIAL", - "12345", - ) - - -def test_split_without_homologation_preserves_clean_type() -> None: - assert split_doc_tipo_y_numero_texto( - "Cedula 123.456", - homologate=False, - ) == ("CEDULA", "123456") - - -def test_apply_rules_homologates_type_column_only() -> None: - df = pd.DataFrame( - { - "tipo": [ - "Cedula", - "C.C", - "Registro Civil", - "Salvoconducto", - "TC", - "Extranjera", - ] - } - ) - - result, log = apply_rules(df, {"tipo": ["doc_type_homologate"]}) - - assert result["tipo"].tolist() == ["CC", "CC", "RC", "SC", "RC", "CE"] - assert any("Homologar tipo de documento" in line for line in log) - - -def test_apply_rules_split_uses_homologation_when_selected() -> None: - df = pd.DataFrame( - { - "documento": [ - "Cedula 123.456", - "Registro Civil: 001-23", - "ABC Especial: 123-45", - ] - } - ) - - result, _log = apply_rules( - df, - {"documento": ["doc_id_split_tipo_numero", "doc_type_homologate"]}, - ) - - assert result["documento_doc_tipo"].tolist() == ["CC", "RC", "ABC ESPECIAL"] - assert result["documento_doc_numeros"].tolist() == ["123456", "00123", "12345"] - assert result["documento"].tolist() == df["documento"].tolist() - - -def test_apply_rules_split_can_skip_homologation() -> None: - df = pd.DataFrame({"documento": ["Cedula 123.456"]}) - - result, _log = apply_rules(df, {"documento": ["doc_id_split_tipo_numero"]}) - - assert result["documento_doc_tipo"].tolist() == ["CEDULA"] - assert result["documento_doc_numeros"].tolist() == ["123456"] - - -def test_apply_rules_unified_date_rule_numeric_and_separated() -> None: - df = pd.DataFrame( - { - "fecha": [ - "20260426", - "26042026", - "260426", - "26/04/2026", - "04/26/2026", - "2026-04-26 10:30", - ] - } - ) - - result, log = apply_rules(df, {"fecha": ["date_to_ddmmyyyy"]}) - - assert result["fecha"].tolist() == [ - "26/04/2026", - "26/04/2026", - "26/04/2026", - "26/04/2026", - "26/04/2026", - "26/04/2026", - ] - assert any("Normalizar fecha" in line for line in log) - - -def test_apply_rules_unified_date_rule_text_months() -> None: - df = pd.DataFrame( - { - "fecha": [ - "26-abril-2026", - "26 abril 2026", - "26/abr/2026", - "abril 26 2026", - "2026 abril 26", - "15-sept-25", - "07 setiembre 2024", - ] - } - ) - - result, _log = apply_rules(df, {"fecha": ["date_to_ddmmyyyy"]}) - - assert result["fecha"].tolist() == [ - "26/04/2026", - "26/04/2026", - "26/04/2026", - "26/04/2026", - "26/04/2026", - "15/09/2025", - "07/09/2024", - ] - - -def test_apply_rules_unified_date_spanish_abbr_month_with_time() -> None: - df = pd.DataFrame( - { - "fecha": [ - "15/may./2024 13:44", - "22/may./2024 11:09", - "30/04/2024", - ] - } - ) - - result, _log = apply_rules(df, {"fecha": ["date_to_ddmmyyyy"]}) - - assert result["fecha"].tolist() == [ - "15/05/2024", - "22/05/2024", - "30/04/2024", - ] - - -def test_apply_rules_date_after_strip_special_spaces_glued_year_time() -> None: - df = pd.DataFrame({"fecha": ["15/may./2024 13:44"]}) - - result, _log = apply_rules( - df, - {"fecha": ["strip_special_spaces", "date_to_ddmmyyyy"]}, - ) - - assert result["fecha"].tolist() == ["15/05/2024"] - - -def test_apply_rules_keep_date_seps_preserves_time_then_normalizes() -> None: - df = pd.DataFrame({"fecha": ["30/04/2024 13:44"]}) - - result, _log = apply_rules( - df, - {"fecha": ["strip_special_keep_date_seps", "date_to_ddmmyyyy"]}, - ) - - assert result["fecha"].tolist() == ["30/04/2024"] - - -def test_apply_rules_us_style_date_comma_then_am_pm_strips_time() -> None: - """Formato tipo Excel/locale US: mes/día/año, coma y hora — solo fecha DD/MM/AAAA.""" - df = pd.DataFrame({"fecha": ["12/27/24, 12:00 AM"]}) - - result, _log = apply_rules(df, {"fecha": ["date_to_ddmmyyyy"]}) - - assert result["fecha"].tolist() == ["27/12/2024"] - - -def test_apply_rules_unified_date_rule_noise_and_unparseable_values() -> None: - df = pd.DataFrame( - { - "fecha": [ - "**26_abril_2026++", - "Fecha: 26-abril-2026", - "xx26/04/2026yy", - "no es fecha", - "31-febrero-2026", - ] - } - ) - - result, _log = apply_rules(df, {"fecha": ["date_to_ddmmyyyy"]}) - - assert result["fecha"].tolist() == [ - "26/04/2026", - "26/04/2026", - "26/04/2026", - "no es fecha", - "31-febrero-2026", - ] - - -def test_apply_rules_cop_mod_moderadora_matches_year() -> None: - df = pd.DataFrame( - {"Fecha Servicio": ["01/01/2024"], "Importe": [4500]}, - ) - result, _log = apply_rules( - df, - {"Importe": ["cop_mod_split"]}, - cop_mod_fecha_column={"Importe": "Fecha Servicio"}, - cop_mod_config={"moderadora_by_year": {"2024": [4500, 18200, 47700]}}, - ) - assert result["Importe_moderadora"].tolist() == [4500] - assert result["Importe_copago"].tolist() == [0] - - -def test_apply_rules_cop_mod_non_moderadora_to_copago() -> None: - df = pd.DataFrame( - {"Fecha Servicio": ["01/01/2024"], "Importe": [4982]}, - ) - result, _log = apply_rules( - df, - {"Importe": ["cop_mod_split"]}, - cop_mod_fecha_column={"Importe": "Fecha Servicio"}, - cop_mod_config={"moderadora_by_year": {"2024": [4500, 18200, 47700]}}, - ) - assert result["Importe_copago"].tolist() == [4982] - assert result["Importe_moderadora"].tolist() == [0] - - -def test_apply_rules_cop_mod_two_phase_dirty_date() -> None: - df = pd.DataFrame( - {"Fecha Servicio": ["15/jun/2024"], "Importe": [4500]}, - ) - result, _log = apply_rules( - df, - { - "Fecha Servicio": ["date_to_ddmmyyyy"], - "Importe": ["cop_mod_split"], - }, - cop_mod_fecha_column={"Importe": "Fecha Servicio"}, - cop_mod_config={"moderadora_by_year": {"2024": [4500, 18200, 47700]}}, - ) - assert result["Fecha Servicio"].iloc[0] == "15/06/2024" - assert result["Importe_moderadora"].iloc[0] == 4500 - - -def test_apply_rules_cop_mod_year_not_in_table_goes_copago() -> None: - df = pd.DataFrame( - {"Fecha Servicio": ["01/01/2030"], "Importe": [4500]}, - ) - result, _log = apply_rules( - df, - {"Importe": ["cop_mod_split"]}, - cop_mod_fecha_column={"Importe": "Fecha Servicio"}, - cop_mod_config={"moderadora_by_year": {"2024": [4500]}}, - ) - assert result["Importe_copago"].tolist() == [4500] - assert result["Importe_moderadora"].tolist() == [0] - - -def test_apply_rules_cop_mod_null_amount_zeros() -> None: - df = pd.DataFrame({"Fecha Servicio": ["01/01/2024"], "Importe": [pd.NA]}) - result, _log = apply_rules( - df, - {"Importe": ["cop_mod_split"]}, - cop_mod_fecha_column={"Importe": "Fecha Servicio"}, - cop_mod_config={"moderadora_by_year": {"2024": [4500]}}, - ) - assert result["Importe_copago"].tolist() == [0] - assert result["Importe_moderadora"].tolist() == [0] - - -def test_apply_rules_nota_extract_one_line_sin_autorizacion() -> None: - texto = ( - "Las retenciones son a nivel informativo no afecta el total a pagar. " - "Paciente: 1063146603 - VILLA BARRIOS NORMA JUDITH Atención No: 63728; " - "Contrato No. URGENCIAS. Valor en Letras: CIENTO OCHENTA Y SIETE MIL " - "QUINIENTOS TREINTA Y SIETE PESOS Fecha ingreso 19/11/2023 fecha egreso 19/11/2023" - ) - df = pd.DataFrame({"notas": [texto]}) - result, log = apply_rules(df, {"notas": ["nota_paciente_autorizacion"]}) - assert result["notas_nota_paciente_doc"].iloc[0] == "1063146603" - assert pd.isna(result["notas_nota_autorizacion"].iloc[0]) - assert any("Extraer datos de notas" in line for line in log) - - -def test_apply_rules_nota_extract_multiline_con_autorizacion() -> None: - texto = ( - "Las retenciones son a nivel informativo no afecta el total a pagar.\n" - "Paciente: 30662603 - LLORENTE HERNANDEZ CARMEN LUCIA Atención No: 69426;\n" - "Autorización No. 227490546; Contrato No. 118. Valor en Letras: CIENTO NOVENTA Y OCHO\n" - "MIL SEISCIENTOS DIEZ PESOS Fecha ingreso 23/01/2024 fecha egreso 23/01/2024" - ) - df = pd.DataFrame({"notas": [texto]}) - result, _log = apply_rules(df, {"notas": ["nota_paciente_autorizacion"]}) - assert result["notas_nota_paciente_doc"].iloc[0] == "30662603" - assert result["notas_nota_autorizacion"].iloc[0] == "227490546" - - -def test_apply_rules_nota_extract_sin_patrones() -> None: - df = pd.DataFrame({"notas": ["sin datos relevantes aquí"]}) - result, _log = apply_rules(df, {"notas": ["nota_paciente_autorizacion"]}) - assert pd.isna(result["notas_nota_paciente_doc"].iloc[0]) - assert pd.isna(result["notas_nota_autorizacion"].iloc[0]) diff --git a/tests/test_csv_manager.py b/tests/test_csv_manager.py deleted file mode 100644 index c46dd7c..0000000 --- a/tests/test_csv_manager.py +++ /dev/null @@ -1,52 +0,0 @@ -from __future__ import annotations - -import tempfile -from pathlib import Path - -from application.csv_manager import CSVManager, sanitize_csv_cell_value - - -def test_sanitize_collapses_newlines() -> None: - assert sanitize_csv_cell_value("a\nb") == "a b" - assert sanitize_csv_cell_value("a\r\nb") == "a b" - assert sanitize_csv_cell_value(" x\v\fy ") == "x y" - - -def test_sanitize_unicode_line_separators() -> None: - assert sanitize_csv_cell_value("a\u2028b\u2029c") == "a b c" - assert sanitize_csv_cell_value("a\x85b") == "a b" - - -def test_sanitize_preserves_none_and_scalars() -> None: - assert sanitize_csv_cell_value(None) is None - assert sanitize_csv_cell_value(42) == 42 - assert sanitize_csv_cell_value(3.14) == 3.14 - assert sanitize_csv_cell_value(False) is False - assert sanitize_csv_cell_value(True) is True - - -def test_sanitize_plain_string_unchanged_identity() -> None: - s = "sin saltos" - assert sanitize_csv_cell_value(s) is s - - -def test_append_rows_one_physical_line_per_record() -> None: - with tempfile.TemporaryDirectory() as tmp: - path = Path(tmp) / "batch.csv" - mgr = CSVManager() - header = ["radicado", "descripcion"] - rows = [ - { - "radicado": "235662283", - "descripcion": ( - '"Las retenciones son a nivel informativo.\n' - "Paciente: 1010079549 - HERAZO\n" - 'Contrato No. URGENCIAS."' - ), - }, - {"radicado": "2", "descripcion": "ok"}, - ] - mgr.append_rows(str(path), rows, header) - text = path.read_text(encoding="utf-8-sig") - physical_lines = text.splitlines() - assert len(physical_lines) == 3 diff --git a/tests/test_csv_utils.py b/tests/test_csv_utils.py deleted file mode 100644 index 09de2a6..0000000 --- a/tests/test_csv_utils.py +++ /dev/null @@ -1,59 +0,0 @@ -from __future__ import annotations - -from core.csv_utils import merge_verbatim_long_strings, set_at_path - - -def test_merge_replaces_truncated_string_same_path() -> None: - raw = { - "Notas": ( - "Las retenciones son informativas. Paciente: 1 - NOMBRE " - "Contrato No. X. Valor en Letras: MIL PESOS Fecha ingreso 01/01/2024" - ), - } - norm = {"Notas": "Las retenciones son informativas. Paciente: 1 - NOMBRE "} - merge_verbatim_long_strings(raw, norm, ["Notas"]) - assert norm["Notas"] == raw["Notas"] - - -def test_merge_nested_path() -> None: - raw = {"cabecera": {"Notas": "A" * 200}} - norm = {"cabecera": {"Notas": "A" * 40}} - merge_verbatim_long_strings(raw, norm, ["cabecera/Notas"]) - assert norm["cabecera"]["Notas"] == "A" * 200 - - -def test_merge_resolves_by_unique_last_segment() -> None: - raw = {"Notas": "texto completo desde raíz cuando el normalizado usa cabecera/Notas"} - norm = {"cabecera": {"Notas": "texto completo"}} - merge_verbatim_long_strings(raw, norm, ["cabecera/Notas"]) - assert norm["cabecera"]["Notas"] == raw["Notas"] - - -def test_merge_skips_ambiguous_same_last_segment() -> None: - raw = { - "a": {"Notas": "primera nota " + "x" * 50}, - "b": {"Notas": "segunda nota " + "y" * 50}, - } - norm = {"cabecera": {"Notas": "primera nota"}} - merge_verbatim_long_strings(raw, norm, ["cabecera/Notas"]) - assert norm["cabecera"]["Notas"] == "primera nota" - - -def test_merge_skips_when_normalized_not_prefix_of_raw() -> None: - raw = {"Notas": "completamente distinto " + "z" * 80} - norm = {"Notas": "otro texto corto"} - merge_verbatim_long_strings(raw, norm, ["Notas"]) - assert norm["Notas"] == "otro texto corto" - - -def test_merge_skips_paths_with_wildcard() -> None: - raw = {"items": [{"x": "long " * 20}]} - norm = {"items": [{"x": "long "}]} - merge_verbatim_long_strings(raw, norm, ["items/*/x"]) - assert norm["items"][0]["x"] == "long " - - -def test_set_at_path_creates_intermediates() -> None: - root: dict = {} - set_at_path(root, "a/b/c", 1) - assert root == {"a": {"b": {"c": 1}}}