refactor: solidify startup UX and engine-aware preprocessing

- Slim core deps: move ML stack to optional extras (paddle/tables/figures/scientific/full) - Lazy settings proxy with config search paths (env var, cwd, user dir) - New commands: init, demo, setup [basic|full], first-run guard on run/watch - Engine-aware preprocessing: Paddle gets original image (fixes dark mode 0.83->0.95) - Results table shows Skipped count; lazy run-dir creation - kg_ocr marked experimental with extra, Docker defaults with OCR_PIPELINE_CONFIG - 25/25 tests, ruff clean
2026-07-19 22:14:54 +02:00
parent deab02610b
commit a0211155fa
59 changed files with 2426 additions and 629 deletions
--- a/src/ocr_pipeline.egg-info/PKG-INFO
+++ b/src/ocr_pipeline.egg-info/PKG-INFO
@@ -4,7 +4,7 @@ Version: 0.1.0
 Summary: OCR pipeline for life science screenshots - RAG ready
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
-Requires-Dist: typer[all]>=0.9.0
+Requires-Dist: typer>=0.9.0
 Requires-Dist: pydantic>=2.6.0
 Requires-Dist: pydantic-settings>=2.2.0
 Requires-Dist: pyyaml>=6.0.1
@@ -13,20 +13,11 @@ Requires-Dist: rich>=13.7.0
 Requires-Dist: tqdm>=4.66.0
 Requires-Dist: opencv-python-headless>=4.9.0
 Requires-Dist: pillow>=10.2.0
-Requires-Dist: numpy>=1.26.0
-Requires-Dist: scikit-image>=0.22.0
-Requires-Dist: paddleocr<3.0.0,>=2.7.0
-Requires-Dist: paddlepaddle>=2.6.0
+Requires-Dist: numpy<2.0,>=1.26.0; python_version < "3.12"
+Requires-Dist: numpy>=1.26.0; python_version >= "3.12"
 Requires-Dist: pytesseract>=0.3.10
-Requires-Dist: torch>=2.2.0
-Requires-Dist: torchvision>=0.17.0
-Requires-Dist: transformers>=4.38.0
-Requires-Dist: timm>=1.0.0
-Requires-Dist: layoutparser>=0.3.0
 Requires-Dist: langchain-text-splitters>=0.0.2
-Requires-Dist: spacy>=3.7.0
 Requires-Dist: nbformat>=5.9.0
-Requires-Dist: legacy-cgi>=2.6.2
 Requires-Dist: sqlite-utils>=3.37.0
 Requires-Dist: watchdog>=3.0.0
 Requires-Dist: python-slugify>=8.0.0
@@ -41,32 +32,92 @@ Requires-Dist: ruff>=0.2.0; extra == "dev"
 Requires-Dist: mypy>=1.8.0; extra == "dev"
 Requires-Dist: pre-commit>=3.6.0; extra == "dev"
 Provides-Extra: grobid
+Provides-Extra: paddle
+Requires-Dist: paddleocr<3.0.0,>=2.7.0; extra == "paddle"
+Requires-Dist: paddlepaddle>=2.6.0; extra == "paddle"
+Provides-Extra: scientific
+Requires-Dist: spacy<3.8.0,>=3.7.0; python_version < "3.12" and extra == "scientific"
+Requires-Dist: spacy>=3.7.0; python_version >= "3.12" and extra == "scientific"
+Requires-Dist: scispacy==0.5.4; python_version < "3.12" and extra == "scientific"
+Provides-Extra: tables
+Requires-Dist: torch>=2.2.0; extra == "tables"
+Requires-Dist: torchvision>=0.17.0; extra == "tables"
+Requires-Dist: transformers>=4.38.0; extra == "tables"
+Requires-Dist: timm>=1.0.0; extra == "tables"
+Provides-Extra: figures
+Requires-Dist: layoutparser>=0.3.0; extra == "figures"
 Provides-Extra: full
-Requires-Dist: ocr-pipeline[dev]; extra == "full"
+Requires-Dist: ocr-pipeline[dev,figures,paddle,scientific,tables]; extra == "full"

 # OCR Pipeline for Life Science Screenshots

 Turns scientific screenshots into RAG-ready Markdown, with metadata attached.

- OCR via PaddleOCR, falls back to Tesseract
+- Reliable Tesseract OCR by default, with optional PaddleOCR
 - Preprocessing: deskew, denoise, CLAHE, line removal
- Figure/table detection
- Entity extraction with scispaCy
- Citation matching
- Chunking for embedding
+- Optional figure/table detection
+- Optional scientific entity extraction with scispaCy
+- Citation matching and embedding-ready chunking

 ## Quick start

 ```bash
-uv sync           # or: pip install -e .
-pre-commit install
+# The default basic profile is lightweight: Tesseract, cleanup, citations,
+# chunking, Markdown, and no model downloads.
+uv sync
+uv run ocr-pipeline doctor

-ocr-pipeline run                  # process screenshots
-ocr-pipeline watch                # watch a folder
-ocr-pipeline reprocess ocr_sc.ipynb   # pull screenshots out of a notebook
-ocr-pipeline status
+# Smoke-test a small directory.
+uv run ocr-pipeline run \
+  --input-dir ~/Pictures/test_screenshots \
+  --output-dir /tmp/ocr-output \
+  --workers 1 \
+  --exclude "*.photoslibrary/*"
 ```

+Use `ocr-pipeline doctor` before a first run or after changing environments. It checks the active profile without loading ML models or downloading model weights.
+
+Use `--force` to reprocess files already recorded in the processing database. The
+`--exclude` option is repeatable and prevents recursive scans from entering bundles
+such as macOS `.photoslibrary` directories.
+
+### Capability profiles
+
+| Profile | What it enables | Intended environment |
+| --- | --- | --- |
+| `basic` (default) | Tesseract OCR, cleanup, citations, chunks, Markdown | Python 3.11+ |
+| `scientific` | Basic + `en_core_sci_lg` entity extraction | **Python 3.11** |
+| `full` | Scientific + PaddleOCR + figure/table detection | Platform-specific ML environment |
+
+Select a profile per run with `--profile`, or set `profile: basic`, `scientific`, or
+`full` at the top of `config.yaml`. The profile is an explicit promise: unavailable
+optional features are reported by `doctor`; they are not silently initialized on a
+basic run.
+
+```bash
+uv run ocr-pipeline doctor --profile scientific
+uv run ocr-pipeline setup scientific  # prints the safe setup commands
+uv run ocr-pipeline run --profile scientific --input-dir ~/Pictures/test_screenshots
+```
+
+#### Scientific NER environment
+
+`en_core_sci_lg` is distributed separately from scispaCy. Create a dedicated Python
+3.11 environment—do not mix its compiled spaCy stack with a Python 3.13 environment:
+
+```bash
+uv venv --python 3.11 .venv-scientific
+uv pip install --python .venv-scientific/bin/python -e '.[scientific]'
+uv pip install --python .venv-scientific/bin/python \
+  https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_core_sci_lg-0.5.4.tar.gz
+
+.venv-scientific/bin/ocr-pipeline doctor --profile scientific
+.venv-scientific/bin/ocr-pipeline run --profile scientific --input-dir ~/Pictures/test_screenshots
+```
+
+Use the explicit `.venv-scientific/bin/ocr-pipeline` path for that environment;
+`uv run` always selects the project's default `.venv`.
+
 ## Docker

 ```bash
@@ -86,9 +137,10 @@ input:
  paths: ["~/Pictures", "/mnt/storage3/aman/screenshots"]
  patterns: ["SCR-*.png", "*.jpg", "*.jpeg", "*.tiff"]
  recursive: true
+  exclude_patterns: ["*.photoslibrary/*"]

 ocr:
-  engine: "paddleocr"  # paddleocr | tesseract | auto
+  engine: "tesseract"  # paddleocr | tesseract | auto
  languages: ["en", "latin"]
  use_gpu: false
  preprocess:
@@ -105,14 +157,14 @@ processing:

 detectors:
  figures:
-    enabled: true
+    enabled: false
    confidence_threshold: 0.7
  tables:
-    enabled: true
+    enabled: false
    confidence_threshold: 0.7

 entities:
-  enabled: true
+  enabled: false
  model: "en_core_sci_lg"

 citations:
@@ -126,6 +178,8 @@ chunking:
 output:
  base_directory: "./data/ocr_output"
  organize_by: "date_run"  # date_run | source_dir | flat
+  write_consolidated: true
+  consolidated_filename: "all_ocr.md"
  frontmatter:
    - source_path
    - source_hash
@@ -134,6 +188,7 @@ output:
    - ocr_confidence_mean
    - language
    - detected_entities
+    - entity_extraction_backend
    - has_figures
    - has_tables
    - citations_found
@@ -200,15 +255,16 @@ Lane 4: BRCA1 KO + pBRCA1-C61G mutant
 Anti-BRCA1 (1:1000), Anti-β-actin (1:5000)
 ```

-Files land in `data/ocr_output/<date>/run_NNN/`.
+Files land in `data/ocr_output/<date>/run_NNN/`. Individual chunk files are retained for RAG indexing. Set `output.write_consolidated: true` to also write `all_ocr.md` containing one frontmatter block and all source text grouped by image.

 ## Life science specifics

 Gene/protein names go through scispaCy's `en_core_sci_lg`. Chemical formulas and units (µM, ng/mL, kb/Mb/Gb, °C, ×g) get normalized, scientific notation gets cleaned up (`1.5×10⁻³` → `1.5×10^-3`), and gel/blot figures get their captions pulled out separately. Citations are matched by regex for DOI, PMID, arXiv, PMC, and ISBN.

-## Models
+## Optional models

-First run downloads what it needs, cached in `~/.cache/ocr_pipeline/`:
+The basic profile downloads no ML models. Advanced profiles download models only when
+the corresponding capability is enabled and used, cached in `~/.cache/ocr_pipeline/`:

 - PaddleOCR models (~200MB)
 - scispaCy `en_core_sci_lg` (~800MB)
@@ -224,7 +280,7 @@ The core pipeline can run with Tesseract alone. PaddleOCR, scientific NER, figur
 - **Figures:** `layoutparser`'s `Detectron2LayoutModel` requires a Detectron2 build matching your Torch/Python platform. It is intentionally not forced as a universal dependency because no single wheel supports every platform.
 - **Tables:** the Table Transformer model is downloaded by Transformers on first use; ensure the selected model's optional dependencies (including `timm`, when required by that model revision) are installed in the runtime image.

-Use `ocr-pipeline config` to verify effective settings. A missing optional model is logged and produces empty results for that detector instead of failing a complete batch.
+Use `ocr-pipeline doctor --profile <profile>` to verify dependencies before processing. Use `ocr-pipeline config` to inspect effective settings. A configured but unavailable optional model is reported as missing by `doctor`; basic runs never attempt to initialize it.

 ## TODO

--- a/src/ocr_pipeline.egg-info/SOURCES.txt
+++ b/src/ocr_pipeline.egg-info/SOURCES.txt
@@ -4,6 +4,7 @@ setup.py
 src/ocr_pipeline/__init__.py
 src/ocr_pipeline/cli.py
 src/ocr_pipeline/config.py
+src/ocr_pipeline/doctor.py
 src/ocr_pipeline/pipeline.py
 src/ocr_pipeline/watch.py
 src/ocr_pipeline.egg-info/PKG-INFO
@@ -38,4 +39,5 @@ tests/test_graph.py
 tests/test_indexer.py
 tests/test_ocr.py
 tests/test_output.py
-tests/test_postprocess.py
+tests/test_postprocess.py
+tests/test_profiles.py
--- a/src/ocr_pipeline.egg-info/requires.txt
+++ b/src/ocr_pipeline.egg-info/requires.txt
@@ -1,4 +1,4 @@
-typer[all]>=0.9.0
+typer>=0.9.0
 pydantic>=2.6.0
 pydantic-settings>=2.2.0
 pyyaml>=6.0.1
@@ -7,20 +7,9 @@ rich>=13.7.0
 tqdm>=4.66.0
 opencv-python-headless>=4.9.0
 pillow>=10.2.0
-numpy>=1.26.0
-scikit-image>=0.22.0
-paddleocr<3.0.0,>=2.7.0
-paddlepaddle>=2.6.0
 pytesseract>=0.3.10
-torch>=2.2.0
-torchvision>=0.17.0
-transformers>=4.38.0
-timm>=1.0.0
-layoutparser>=0.3.0
 langchain-text-splitters>=0.0.2
-spacy>=3.7.0
 nbformat>=5.9.0
-legacy-cgi>=2.6.2
 sqlite-utils>=3.37.0
 watchdog>=3.0.0
 python-slugify>=8.0.0
@@ -28,6 +17,12 @@ xxhash>=3.4.0
 python-magic>=0.4.27
 platformdirs>=4.2.0

+[:python_version < "3.12"]
+numpy<2.0,>=1.26.0
+
+[:python_version >= "3.12"]
+numpy>=1.26.0
+
 [dev]
 pytest>=8.0.0
 pytest-cov>=4.1.0
@@ -36,7 +31,29 @@ ruff>=0.2.0
 mypy>=1.8.0
 pre-commit>=3.6.0

+[figures]
+layoutparser>=0.3.0
+
 [full]
-ocr-pipeline[dev]
+ocr-pipeline[dev,figures,paddle,scientific,tables]

 [grobid]
+
+[paddle]
+paddleocr<3.0.0,>=2.7.0
+paddlepaddle>=2.6.0
+
+[scientific]
+
+[scientific:python_version < "3.12"]
+spacy<3.8.0,>=3.7.0
+scispacy==0.5.4
+
+[scientific:python_version >= "3.12"]
+spacy>=3.7.0
+
+[tables]
+torch>=2.2.0
+torchvision>=0.17.0
+transformers>=4.38.0
+timm>=1.0.0