Aman Nalakath
c5bf5d2b0c
Some checks failed
tests / core (macos-latest, py3.11) (push) Has been cancelled
tests / core (macos-latest, py3.12) (push) Has been cancelled
tests / core (macos-latest, py3.13) (push) Has been cancelled
tests / core (ubuntu-latest, py3.11) (push) Has been cancelled
tests / core (ubuntu-latest, py3.12) (push) Has been cancelled
tests / core (ubuntu-latest, py3.13) (push) Has been cancelled
tests / doctor CLI smoke test (push) Has been cancelled
- Delete notebooks/ (personal scratch artifacts; gitignored) - Move nbformat to --extra migrate; lazy imports in migrate.py and cli.py so base CLI loads without it - Restore langchain-text-splitters in core (chunking dependency)
60 lines
4.5 KiB
Markdown
60 lines
4.5 KiB
Markdown
# Changelog
|
|
|
|
All notable changes to this project will be documented in this file.
|
|
|
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
|
## [0.2.0] - 2026-07-21
|
|
|
|
### Added
|
|
- Knowledge graph: `kg_ocr.graph` builds a NetworkX graph (documents, chunks, entities, citations, entity co-occurrence) from pipeline chunk markdown; `kg_ocr.graph.analyzer` provides summaries, top entities/citations, and anomaly detection (low OCR confidence, empty chunks/documents, entity hubs).
|
|
- Graph traversal: `chunks_for_entity`, `chunks_for_citation`, `related_entities`, `expand_context` (one-hop CO_OCCURS expansion for graph-RAG), exposed via `ocr-pipeline kg query`.
|
|
- `kg_ocr.embeddings.config_loader.KgConfig` for traversal tunables and Neo4j credentials from env.
|
|
- `kg_ocr.export`: node-link JSON (lossless round-trip), GraphML for Gephi/Cytoscape, and batched MERGE export to Neo4j via `Neo4jExporter`.
|
|
- CLI: `ocr-pipeline kg build|stats|query|export` subcommands with lazy kg_ocr imports.
|
|
- Optional extras: `kg` (networkx + txtai + litellm + python-dotenv), `neo4j` (driver), `migrate` (nbformat).
|
|
- `init` command: writes a minimal `config.yaml` with platform-correct defaults and checks for Tesseract.
|
|
- `demo` command: runs the pipeline on a bundled sample screenshot, prints the output Markdown, and verifies the install in <30s.
|
|
- `setup [basic|full]` command: executes install steps with confirmation (`--yes`/`--dry-run`).
|
|
- `doctor` command: health check for Python, Tesseract, PaddleOCR, PyTorch backend, table/figure detectors, scispaCy model, config, database, and output directory.
|
|
- Lazy settings: `settings` is a proxy; importing the package no longer reads `config.yaml` from cwd.
|
|
- Config search paths: `$OCR_PIPELINE_CONFIG` → `./config.yaml` → per-user config dir.
|
|
- Optional dependency extras: `paddle`, `tables`, `figures`, `scientific`, `kg`, `full`.
|
|
- Engine-aware preprocessing: PaddleOCR reads the (size-capped) original image; Tesseract reads the binarized derivative. Fixes dark-mode OCR quality (confidence 0.83 → 0.95 on test screenshots).
|
|
- `data/test_run_full` sample run on 7 screenshots demonstrating full-stack output.
|
|
- Tests for citations (19), doctor (15), and startup (12).
|
|
|
|
### Changed
|
|
- Code defaults are now safe: Tesseract OCR, detectors/entities off. The repo's `config.yaml` keeps the full stack for development.
|
|
- Detectors default to `enabled: false`; enable per-detector in your config.
|
|
- OCR engine default is now `tesseract` (was `paddleocr`).
|
|
- `run` results table now shows Skipped count.
|
|
- Run directories are created lazily on first write (no empty `run_NNN`).
|
|
- `kg_ocr` is now a functional knowledge-graph package (graph/export need only networkx); txtai/litellm imports are lazy so the package imports in slim environments.
|
|
- Ruff settings moved to `[tool.ruff.lint]` (silences per-run deprecation warning).
|
|
- Pre-commit: bumped ruff, mypy, pre-commit-hooks; dropped the redundant isort hook.
|
|
- mypy: added `ignore_missing_imports` for optional deps + `types-PyYAML` in dev extras; `python_version = "3.12"` to match numpy 2.5 stubs.
|
|
- README: minimal-style rewrite with collapsible `<details>` sections.
|
|
|
|
### Fixed
|
|
- ISBN regex no longer false-positives on years, URLs, job IDs, or dilutions. Now requires an explicit `ISBN` prefix and matches strict 13- or 10-digit structures.
|
|
- `opencv-python-headless` pinned to `<5.0.0`; the 5.0 wheel did not ship a working `cv2` module on Python 3.13.
|
|
- `import ocr_pipeline` no longer depends on the caller's cwd.
|
|
|
|
### Removed
|
|
- Dead `src/ocr_pipeline/watch.py` shim (shadowed by the `watch/` package).
|
|
- Stub `kg_ocr/cli/` directory (commands now in the main `ocr-pipeline` CLI).
|
|
- Untracked `src/ocr_pipeline.egg-info` from git (still gitignored).
|
|
- `notebooks/` (personal scratch artifacts; gitignored).
|
|
- `nbformat` from core dependencies (moved to `migrate` extra).
|
|
- Unused core dependencies: `tqdm`, `scikit-image`, `sqlite-utils`, `xxhash`, `python-slugify`, `python-magic`, `legacy-cgi`.
|
|
- Core dependency on `spacy` (moved to `scientific` extra).
|
|
- Core dependency on `paddleocr`/`paddlepaddle` (moved to `paddle` extra).
|
|
- Core dependency on `torch`/`torchvision`/`transformers`/`timm` (moved to `tables` extra).
|
|
- Core dependency on `layoutparser` (moved to `figures` extra).
|
|
|
|
## [0.1.0] - Initial
|
|
|
|
Initial release with PaddleOCR + Tesseract fallback, scispaCy NER, layout-aware detectors, citation matching, and Markdown output with YAML frontmatter.
|