kg-scr/README.md

# OCR Pipeline for Life Science Screenshots

Turns scientific screenshots into RAG-ready Markdown, with metadata attached.

- OCR via PaddleOCR, falls back to Tesseract
- Preprocessing: deskew, denoise, CLAHE, line removal
- Figure/table detection
- Entity extraction with scispaCy
- Citation matching (DOI, PMID, arXiv, ISBN)
- Chunking for embedding
- Knowledge graph over the output (`kg` commands)

## Quick start

```bash
uv sync
uv run ocr-pipeline setup     # installs Tesseract if missing
uv run ocr-pipeline init      # write a config
uv run ocr-pipeline demo      # check it works
uv run ocr-pipeline run       # process your screenshots
```

Default install is small: Tesseract only, no multi-GB ML downloads.

## Commands

- `run` / `watch` — process a directory, or watch for new screenshots
- `status` / `stats` — what's been processed
- `doctor` — check engines, models, config
- `kg build|stats|query|export` — knowledge graph over the output
- `setup [basic|full]` — install dependencies step by step

Use `--force` to reprocess. `--exclude "*.photoslibrary/*"` keeps recursive
scans out of macOS photo bundles.

<details>
<summary><b>Optional extras</b></summary>

Heavy ML features are opt-in and degrade gracefully when missing (logged, step
skipped, run continues):

| Extra | Provides |
|-------|----------|
| `paddle` | PaddleOCR engine |
| `scientific` | spaCy/scispaCy NER |
| `tables` | Table Transformer detection |
| `figures` | LayoutParser figure/caption detection |
| `full` | All of the above |
| `kg` | Knowledge graph (networkx, txtai, litellm) |
| `neo4j` | Neo4j export driver |

```bash
uv sync --extra full
# or guided:
uv run ocr-pipeline setup full
```

`en_core_sci_lg` is distributed separately from scispaCy and needs spaCy 3.7.x.
If the model install conflicts, use a dedicated Python 3.11 venv and enable
`entities` only there.

</details>

<details>
<summary><b>Knowledge graph</b></summary>

Build a graph from pipeline output: documents, chunks, entities, citations,
entity co-occurrence.

```bash
uv sync --extra kg

uv run ocr-pipeline kg build -d data/ocr_output
uv run ocr-pipeline kg stats data/ocr_output/kg_graph.json

# retrieval
uv run ocr-pipeline kg query data/ocr_output/kg_graph.json --entity BRCA1
uv run ocr-pipeline kg query data/ocr_output/kg_graph.json --entity BRCA1 --expand
uv run ocr-pipeline kg query data/ocr_output/kg_graph.json --citation 10.1038/nature12345

# export
uv run ocr-pipeline kg export data/ocr_output/kg_graph.json -f graphml
uv run ocr-pipeline kg export data/ocr_output/kg_graph.json -f neo4j
```

Anomaly detection flags low-confidence documents, empty chunks, and entity
hubs. Neo4j export needs `uv sync --extra neo4j` plus `NEO4J_URI`,
`NEO4J_USER`, `NEO4J_PASSWORD` env vars (or `--uri/--user/--password`).

</details>

<details>
<summary><b>Docker</b></summary>

The image ships the Tesseract-only core. It reads `/data/screenshots` and
writes `/data/output`:

```bash
docker build -t ocr-pipeline .
docker run -v ~/Pictures:/data/screenshots -v "$PWD/ocr-output:/data/output" ocr-pipeline run
docker run -v ~/Pictures:/data/screenshots -v "$PWD/ocr-output:/data/output" ocr-pipeline watch

# with ML extras:
docker build --build-arg EXTRAS="--extra full" -t ocr-pipeline:full .
```

</details>

<details>
<summary><b>Config</b></summary>

`ocr-pipeline init` writes a minimal config. Lookup order:
`$OCR_PIPELINE_CONFIG` → `./config.yaml` → per-user config dir.

Every knob, for reference:

```yaml
input:
  paths: ["~/Pictures", "/mnt/storage3/aman/screenshots"]
  patterns: ["SCR-*.png", "*.jpg", "*.jpeg", "*.tiff"]
  recursive: true
  exclude_patterns: ["*.photoslibrary/*"]

ocr:
  engine: "paddleocr"  # paddleocr | tesseract | auto
  languages: ["en", "latin"]
  use_gpu: false
  preprocess:
    deskew: true
    denoise: true
    clahe: true
    adaptive_threshold: true
    remove_lines: true

processing:
  workers: 4
  batch_size: 10
  retry_attempts: 2

detectors:
  figures:
    enabled: true
    confidence_threshold: 0.7
  tables:
    enabled: true
    confidence_threshold: 0.7

entities:
  enabled: true
  model: "en_core_sci_lg"

citations:
  regex_enabled: true
  grobid_enabled: false  # set true if you're running a GROBID server

chunking:
  chunk_size: 1000
  chunk_overlap: 200

output:
  base_directory: "./data/ocr_output"
  organize_by: "date_run"  # date_run | source_dir | flat
  write_consolidated: true
  consolidated_filename: "all_ocr.md"

watch:
  enabled: true
  debounce_seconds: 5
  db_path: "./data/processed_files.db"
```

</details>

<details>
<summary><b>Output example</b></summary>

Each chunk is a Markdown file with YAML frontmatter:

```markdown
---
source_path: "/Users/Aman/Pictures/SCR-20250115-gel.png"
source_hash: "a1b2c3d4e5f6..."
timestamp: "2025-01-15T10:30:00Z"
ocr_engine: "paddleocr"
ocr_confidence_mean: 0.91
language: "en"
detected_entities: ["GENE", "PROTEIN", "CHEMICAL"]
has_figures: true
has_tables: false
citations_found: ["doi:10.1038/nature12345", "pmid:PMID:12345678"]
chunk_index: 0
total_chunks: 2
---

# Screenshot: SCR-20250115-gel.png

## Extracted Text

**Western Blot Analysis of BRCA1 Expression**

Lane 1: WT control
Lane 2: BRCA1 KO (CRISPR)
...
```

Files land in `<output>/<date>/run_NNN/`. Chunk files are kept for RAG
indexing. `output.write_consolidated: true` also writes `all_ocr.md` with one
frontmatter block and all source text grouped by image.

</details>

<details>
<summary><b>Models and dependencies</b></summary>

Weights download on first use, cached in `~/.cache/ocr_pipeline/`:

- PaddleOCR models (~200MB)
- scispaCy `en_core_sci_lg` (~800MB)
- Table Transformer (~500MB)
- LayoutParser PubLayNet (~300MB)

Notes per capability:

- **PaddleOCR:** pinned to the legacy 2.x API. Falls back to Tesseract when it
  can't initialize; the fallback is logged and recorded in frontmatter.
- **Scientific NER:** without scispaCy + `en_core_sci_lg`, a conservative
  regex extractor runs instead and the backend is marked accordingly.
- **Figures:** needs a Detectron2 build matching your Torch/Python. No
  universal wheel exists, so it's never installed automatically.
- **Tables:** downloaded by Transformers on first use; needs `timm`.

`ocr-pipeline doctor` checks all of the above and prints remediation hints.

</details>

## Life science specifics

Gene/protein names go through scispaCy's `en_core_sci_lg`. Chemical formulas
and units (µM, ng/mL, kb/Mb/Gb, °C, ×g) get normalized, scientific notation
gets cleaned up (`1.5×10⁻³` → `1.5×10^-3`), and gel/blot figures get their
captions pulled out separately.

## TODO

- [x] Build a knowledge graph from extracted entities/citations (see `kg` commands)
- [ ] Entity linking to MeSH/UniProt identifiers
- [ ] GROBID-based structured citation parsing (server optional, regex today)