kg: graph build, traversal queries, neo4j export - kg_ocr.graph builds a networkx graph (docs, chunks, entities, citations, co-occurrence) from chunk markdown - analyzer for summaries, top entities/citations, anomaly checks - traversal: chunks_for_entity/citation, related_entities, expand_context - export: JSON round-trip, GraphML, batched MERGE into neo4j - new CLI: ocr-pipeline kg build|stats|query|export - lazy kg_ocr imports, networkx/neo4j behind extras - dropped dead watch.py shim, added KgConfig stub - trimmed README, updated TODO

README.md

@@ -1,90 +1,118 @@
 # OCR Pipeline for Life Science Screenshots
 
-[![Tests](https://img.shields.io/badge/tests-passing-brightgreen)](./.github/workflows/tests.yml)
-
 Turns scientific screenshots into RAG-ready Markdown, with metadata attached.
 
 - OCR via PaddleOCR, falls back to Tesseract
 - Preprocessing: deskew, denoise, CLAHE, line removal
 - Figure/table detection
 - Entity extraction with scispaCy
-- Citation matching
+- Citation matching (DOI, PMID, arXiv, ISBN)
 - Chunking for embedding
+- Knowledge graph over the output (`kg` commands)
 
 ## Quick start
 
-The default install is small: Tesseract OCR only, no multi-GB ML downloads.
-
 ```bash
-uv sync                          # install the core pipeline
-uv run ocr-pipeline setup        # installs Tesseract if missing (asks first)
-uv run ocr-pipeline init         # write a config for this machine
-uv run ocr-pipeline demo         # verify everything works on a sample screenshot
-uv run ocr-pipeline run          # process your screenshots
+uv sync
+uv run ocr-pipeline setup     # installs Tesseract if missing
+uv run ocr-pipeline init      # write a config
+uv run ocr-pipeline demo      # check it works
+uv run ocr-pipeline run       # process your screenshots
 ```
 
-Config is found in this order: `$OCR_PIPELINE_CONFIG`, `./config.yaml`, then the
-per-user config directory (`~/Library/Application Support/ocr-pipeline` on macOS,
-`~/.config/ocr-pipeline` on Linux). Running `run` with neither a config nor
-`--input-dir` stops with guidance instead of scanning a default directory.
+Default install is small: Tesseract only, no multi-GB ML downloads.
 
-Use `--force` to reprocess files already recorded in the processing database. The
-`--exclude` option is repeatable and prevents recursive scans from entering bundles
-such as macOS `.photoslibrary` directories.
+## Commands
 
-### Optional capabilities
+- `run` / `watch` — process a directory, or watch for new screenshots
+- `status` / `stats` — what's been processed
+- `doctor` — check engines, models, config
+- `kg build|stats|query|export` — knowledge graph over the output
+- `setup [basic|full]` — install dependencies step by step
 
-Heavy ML features are opt-in extras and degrade gracefully when absent (a warning
-is logged and that step is skipped):
+Use `--force` to reprocess. `--exclude "*.photoslibrary/*"` keeps recursive
+scans out of macOS photo bundles.
+
+<details>
+<summary><b>Optional extras</b></summary>
+
+Heavy ML features are opt-in and degrade gracefully when missing (logged, step
+skipped, run continues):
 
 | Extra | Provides |
 |-------|----------|
-| `paddle` | PaddleOCR engine (`ocr.engine: paddleocr` or `auto`) |
-| `scientific` | spaCy/scispaCy NER (`entities.enabled: true`) |
+| `paddle` | PaddleOCR engine |
+| `scientific` | spaCy/scispaCy NER |
 | `tables` | Table Transformer detection |
-| `figures` | LayoutParser figure/caption detection (needs a Detectron2 build) |
+| `figures` | LayoutParser figure/caption detection |
 | `full` | All of the above |
+| `kg` | Knowledge graph (networkx, txtai, litellm) |
+| `neo4j` | Neo4j export driver |
 
 ```bash
-uv run ocr-pipeline setup full   # runs: uv sync --extra full + downloads en_core_sci_lg
+uv sync --extra full
+# or guided:
+uv run ocr-pipeline setup full
 ```
 
-`en_core_sci_lg` is distributed separately from scispaCy. The compatible scispaCy
-0.5.4 release requires spaCy 3.7.x; if the model install conflicts with your spaCy
-version, use a dedicated Python 3.11 environment as described in
-`ocr-pipeline setup full --dry-run` output, and enable `entities` only there.
+`en_core_sci_lg` is distributed separately from scispaCy and needs spaCy 3.7.x.
+If the model install conflicts, use a dedicated Python 3.11 venv and enable
+`entities` only there.
 
-## kg_ocr (experimental)
+</details>
 
-The top-level `kg_ocr` package is a prototype txtai/litellm RAG interface over the
-pipeline's output. It is not the supported interface (that is the `ocr-pipeline`
-CLI) and its dependencies are not installed by default:
+<details>
+<summary><b>Knowledge graph</b></summary>
+
+Build a graph from pipeline output: documents, chunks, entities, citations,
+entity co-occurrence.
 
 ```bash
 uv sync --extra kg
+
+uv run ocr-pipeline kg build -d data/ocr_output
+uv run ocr-pipeline kg stats data/ocr_output/kg_graph.json
+
+# retrieval
+uv run ocr-pipeline kg query data/ocr_output/kg_graph.json --entity BRCA1
+uv run ocr-pipeline kg query data/ocr_output/kg_graph.json --entity BRCA1 --expand
+uv run ocr-pipeline kg query data/ocr_output/kg_graph.json --citation 10.1038/nature12345
+
+# export
+uv run ocr-pipeline kg export data/ocr_output/kg_graph.json -f graphml
+uv run ocr-pipeline kg export data/ocr_output/kg_graph.json -f neo4j
 ```
 
-## Docker
+Anomaly detection flags low-confidence documents, empty chunks, and entity
+hubs. Neo4j export needs `uv sync --extra neo4j` plus `NEO4J_URI`,
+`NEO4J_USER`, `NEO4J_PASSWORD` env vars (or `--uri/--user/--password`).
 
-The image ships the Tesseract-only core with a ready-made config that reads from
-`/data/screenshots` and writes to `/data/output`:
+</details>
+
+<details>
+<summary><b>Docker</b></summary>
+
+The image ships the Tesseract-only core. It reads `/data/screenshots` and
+writes `/data/output`:
 
 ```bash
 docker build -t ocr-pipeline .
 docker run -v ~/Pictures:/data/screenshots -v "$PWD/ocr-output:/data/output" ocr-pipeline run
 docker run -v ~/Pictures:/data/screenshots -v "$PWD/ocr-output:/data/output" ocr-pipeline watch
 
-# ML extras baked in:
+# with ML extras:
 docker build --build-arg EXTRAS="--extra full" -t ocr-pipeline:full .
 ```
 
-## Config
-
-`ocr-pipeline init` writes a minimal working config; the example below shows every
-knob for reference.
+</details>
 
 <details>
-<summary><code>config.yaml</code></summary>
+<summary><b>Config</b></summary>
+
+`ocr-pipeline init` writes a minimal config. Lookup order:
+`$OCR_PIPELINE_CONFIG` → `./config.yaml` → per-user config dir.
+
+Every knob, for reference:
 
 ```yaml
 input:
@@ -134,20 +162,6 @@ output:
   organize_by: "date_run"  # date_run | source_dir | flat
   write_consolidated: true
   consolidated_filename: "all_ocr.md"
-  frontmatter:
-    - source_path
-    - source_hash
-    - timestamp
-    - ocr_engine
-    - ocr_confidence_mean
-    - language
-    - detected_entities
-    - entity_extraction_backend
-    - has_figures
-    - has_tables
-    - citations_found
-    - chunk_index
-    - total_chunks
 
 watch:
   enabled: true
@@ -157,7 +171,8 @@ watch:
 
 </details>
 
-## Output
+<details>
+<summary><b>Output example</b></summary>
 
 Each chunk is a Markdown file with YAML frontmatter:
 
@@ -172,71 +187,61 @@ language: "en"
 detected_entities: ["GENE", "PROTEIN", "CHEMICAL"]
 has_figures: true
 has_tables: false
-citations_found: ["DOI:10.1038/nature12345", "PMID:12345678"]
+citations_found: ["doi:10.1038/nature12345", "pmid:PMID:12345678"]
 chunk_index: 0
 total_chunks: 2
 ---
 
 # Screenshot: SCR-20250115-gel.png
 
-## Figures
-
-### Figure 1
-- BBox: [100, 200, 800, 600]
-- Confidence: 0.92
-- Caption: "Western blot showing BRCA1 expression..."
-
-## Detected Entities
-
-- BRCA1
-- CRISPR
-- β-actin
-
-## Citations
-
-- DOI: 10.1038/nature12345
-- PMID: 12345678
-
 ## Extracted Text
 
 **Western Blot Analysis of BRCA1 Expression**
 
 Lane 1: WT control
 Lane 2: BRCA1 KO (CRISPR)
-Lane 3: BRCA1 KO + pBRCA1-WT rescue
-Lane 4: BRCA1 KO + pBRCA1-C61G mutant
-
-Anti-BRCA1 (1:1000), Anti-β-actin (1:5000)
+...
 ```
 
-Files land in `data/ocr_output/<date>/run_NNN/`. Individual chunk files are retained for RAG indexing. Set `output.write_consolidated: true` to also write `all_ocr.md` containing one frontmatter block and all source text grouped by image.
+Files land in `<output>/<date>/run_NNN/`. Chunk files are kept for RAG
+indexing. `output.write_consolidated: true` also writes `all_ocr.md` with one
+frontmatter block and all source text grouped by image.
 
-## Life science specifics
+</details>
 
-Gene/protein names go through scispaCy's `en_core_sci_lg`. Chemical formulas and units (µM, ng/mL, kb/Mb/Gb, °C, ×g) get normalized, scientific notation gets cleaned up (`1.5×10⁻³` → `1.5×10^-3`), and gel/blot figures get their captions pulled out separately. Citations are matched by regex for DOI, PMID, arXiv, PMC, and ISBN.
+<details>
+<summary><b>Models and dependencies</b></summary>
 
-## Models
-
-Model weights are only fetched when the matching extra is installed and enabled.
-First run downloads what it needs, cached in `~/.cache/ocr_pipeline/`:
+Weights download on first use, cached in `~/.cache/ocr_pipeline/`:
 
 - PaddleOCR models (~200MB)
 - scispaCy `en_core_sci_lg` (~800MB)
 - Table Transformer (~500MB)
 - LayoutParser PubLayNet (~300MB)
 
-## Runtime requirements and health
+Notes per capability:
 
-The core pipeline can run with Tesseract alone. PaddleOCR, scientific NER, figure detection, and table detection are optional capabilities with heavyweight, platform-specific dependencies. The pipeline records the OCR engine actually used and the entity-extraction backend in generated frontmatter; inspect them after each run rather than assuming configured models loaded.
+- **PaddleOCR:** pinned to the legacy 2.x API. Falls back to Tesseract when it
+  can't initialize; the fallback is logged and recorded in frontmatter.
+- **Scientific NER:** without scispaCy + `en_core_sci_lg`, a conservative
+  regex extractor runs instead and the backend is marked accordingly.
+- **Figures:** needs a Detectron2 build matching your Torch/Python. No
+  universal wheel exists, so it's never installed automatically.
+- **Tables:** downloaded by Transformers on first use; needs `timm`.
 
-- **PaddleOCR:** the project pins the legacy 2.x API used by the pipeline. Install the locked environment with `uv sync`; a startup fallback to Tesseract is logged when Paddle cannot initialize.
-- **Scientific NER:** install a compatible `scispacy` distribution and the separately distributed `en_core_sci_lg` model before enabling production scientific NER. Without it, the pipeline uses its conservative regex fallback and marks the backend accordingly.
-- **Figures:** `layoutparser`'s `Detectron2LayoutModel` requires a Detectron2 build matching your Torch/Python platform. It is intentionally not forced as a universal dependency because no single wheel supports every platform.
-- **Tables:** the Table Transformer model is downloaded by Transformers on first use; ensure the selected model's optional dependencies (including `timm`, when required by that model revision) are installed in the runtime image.
+`ocr-pipeline doctor` checks all of the above and prints remediation hints.
 
-Use `ocr-pipeline config` to verify effective settings. A missing optional model is logged and produces empty results for that detector instead of failing a complete batch.
+</details>
+
+## Life science specifics
+
+Gene/protein names go through scispaCy's `en_core_sci_lg`. Chemical formulas
+and units (µM, ng/mL, kb/Mb/Gb, °C, ×g) get normalized, scientific notation
+gets cleaned up (`1.5×10⁻³` → `1.5×10^-3`), and gel/blot figures get their
+captions pulled out separately.
 
 ## TODO
 
-- [ ] Build a knowledge graph from extracted entities/citations
-
+- [x] Build a knowledge graph from extracted entities/citations (see `kg` commands)
+- [ ] Entity linking to MeSH/UniProt identifiers
+- [ ] GROBID-based structured citation parsing (server optional, regex today)