Aman Nalakath
7503a441c2
Some checks failed
tests / core (macos-latest, py3.11) (push) Has been cancelled
tests / core (macos-latest, py3.12) (push) Has been cancelled
tests / core (macos-latest, py3.13) (push) Has been cancelled
tests / core (ubuntu-latest, py3.11) (push) Has been cancelled
tests / core (ubuntu-latest, py3.12) (push) Has been cancelled
tests / core (ubuntu-latest, py3.13) (push) Has been cancelled
tests / doctor CLI smoke test (push) Has been cancelled
52 lines
3.8 KiB
Markdown
52 lines
3.8 KiB
Markdown
# Changelog
|
|
|
|
All notable changes to this project will be documented in this file.
|
|
|
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
|
## [Unreleased]
|
|
|
|
### Added
|
|
- Knowledge graph: `kg_ocr.graph` builds a NetworkX graph (documents, chunks, entities, citations, entity co-occurrence) from pipeline chunk markdown; `kg_ocr.graph.analyzer` provides summaries, top entities/citations, and anomaly detection (low OCR confidence, empty chunks/documents, entity hubs).
|
|
- Graph traversal: `chunks_for_entity`, `chunks_for_citation`, `related_entities`, `expand_context` (one-hop CO_OCCURS expansion for graph-RAG), exposed via `ocr-pipeline kg query`.
|
|
- `kg_ocr.embeddings.config_loader.KgConfig` for traversal tunables and Neo4j credentials from env.
|
|
- `kg_ocr.export`: node-link JSON (lossless round-trip), GraphML for Gephi/Cytoscape, and batched MERGE export to Neo4j via `Neo4jExporter`.
|
|
- CLI: `ocr-pipeline kg build|stats|query|export` subcommands with lazy kg_ocr imports.
|
|
- Optional extras: `kg` (networkx + txtai + litellm + python-dotenv), `neo4j` (driver).
|
|
- `init` command: writes a minimal `config.yaml` with platform-correct defaults and checks for Tesseract.
|
|
- `demo` command: runs the pipeline on a bundled sample screenshot, prints the output Markdown, and verifies the install in <30s.
|
|
- `setup [basic|full]` command: executes install steps with confirmation (`--yes`/`--dry-run`).
|
|
- `doctor` command: health check for Python, Tesseract, PaddleOCR, PyTorch backend, table/figure detectors, scispaCy model, config, database, and output directory.
|
|
- Lazy settings: `settings` is a proxy; importing the package no longer reads `config.yaml` from cwd.
|
|
- Config search paths: `$OCR_PIPELINE_CONFIG` → `./config.yaml` → per-user config dir.
|
|
- Optional dependency extras: `paddle`, `tables`, `figures`, `scientific`, `kg`, `full`.
|
|
- Engine-aware preprocessing: PaddleOCR reads the (size-capped) original image; Tesseract reads the binarized derivative. Fixes dark-mode OCR quality (confidence 0.83 → 0.95 on test screenshots).
|
|
- `data/test_run_full` sample run on 7 screenshots demonstrating full-stack output.
|
|
- Tests for citations (19), doctor (15), and startup (12).
|
|
|
|
### Changed
|
|
- Code defaults are now safe: Tesseract OCR, detectors/entities off. The repo's `config.yaml` keeps the full stack for development.
|
|
- Detectors default to `enabled: false`; enable per-detector in your config.
|
|
- OCR engine default is now `tesseract` (was `paddleocr`).
|
|
- `run` results table now shows Skipped count.
|
|
- Run directories are created lazily on first write (no empty `run_NNN`).
|
|
- `kg_ocr` is now a functional knowledge-graph package (graph/export need only networkx); txtai/litellm imports are lazy so the package imports in slim environments.
|
|
|
|
### Fixed
|
|
- ISBN regex no longer false-positives on years, URLs, job IDs, or dilutions. Now requires an explicit `ISBN` prefix and matches strict 13- or 10-digit structures.
|
|
- `opencv-python-headless` pinned to `<5.0.0`; the 5.0 wheel did not ship a working `cv2` module on Python 3.13.
|
|
- `import ocr_pipeline` no longer depends on the caller's cwd.
|
|
|
|
### Removed
|
|
- Dead `src/ocr_pipeline/watch.py` shim (shadowed by the `watch/` package).
|
|
- Unused core dependencies: `tqdm`, `scikit-image`, `sqlite-utils`, `xxhash`, `python-slugify`, `python-magic`, `legacy-cgi`.
|
|
- Core dependency on `spacy` (moved to `scientific` extra).
|
|
- Core dependency on `paddleocr`/`paddlepaddle` (moved to `paddle` extra).
|
|
- Core dependency on `torch`/`torchvision`/`transformers`/`timm` (moved to `tables` extra).
|
|
- Core dependency on `layoutparser` (moved to `figures` extra).
|
|
|
|
## [0.1.0] - Initial
|
|
|
|
Initial release with PaddleOCR + Tesseract fallback, scispaCy NER, layout-aware detectors, citation matching, and Markdown output with YAML frontmatter.
|