# OCR Pipeline for Life Science Screenshots Turns scientific screenshots into RAG-ready Markdown, with metadata attached. - OCR via PaddleOCR, falls back to Tesseract - Preprocessing: deskew, denoise, CLAHE, line removal - Figure/table detection - Entity extraction with scispaCy - Citation matching (DOI, PMID, arXiv, ISBN) - Chunking for embedding - Knowledge graph over the output (`kg` commands) ## Quick start ```bash uv sync uv run ocr-pipeline setup # installs Tesseract if missing uv run ocr-pipeline init # write a config uv run ocr-pipeline demo # check it works uv run ocr-pipeline run # process your screenshots ``` Default install is small: Tesseract only, no multi-GB ML downloads. ## Commands - `run` / `watch` — process a directory, or watch for new screenshots - `status` / `stats` — what's been processed - `doctor` — check engines, models, config - `kg build|stats|query|export` — knowledge graph over the output - `setup [basic|full]` — install dependencies step by step Use `--force` to reprocess. `--exclude "*.photoslibrary/*"` keeps recursive scans out of macOS photo bundles.
Optional extras Heavy ML features are opt-in and degrade gracefully when missing (logged, step skipped, run continues): | Extra | Provides | |-------|----------| | `paddle` | PaddleOCR engine | | `scientific` | spaCy/scispaCy NER | | `tables` | Table Transformer detection | | `figures` | LayoutParser figure/caption detection | | `full` | All of the above | | `kg` | Knowledge graph (networkx, txtai, litellm) | | `neo4j` | Neo4j export driver | ```bash uv sync --extra full # or guided: uv run ocr-pipeline setup full ``` `en_core_sci_lg` is distributed separately from scispaCy and needs spaCy 3.7.x. If the model install conflicts, use a dedicated Python 3.11 venv and enable `entities` only there.
Knowledge graph Build a graph from pipeline output: documents, chunks, entities, citations, entity co-occurrence. ```bash uv sync --extra kg uv run ocr-pipeline kg build -d data/ocr_output uv run ocr-pipeline kg stats data/ocr_output/kg_graph.json # retrieval uv run ocr-pipeline kg query data/ocr_output/kg_graph.json --entity BRCA1 uv run ocr-pipeline kg query data/ocr_output/kg_graph.json --entity BRCA1 --expand uv run ocr-pipeline kg query data/ocr_output/kg_graph.json --citation 10.1038/nature12345 # export uv run ocr-pipeline kg export data/ocr_output/kg_graph.json -f graphml uv run ocr-pipeline kg export data/ocr_output/kg_graph.json -f neo4j ``` Anomaly detection flags low-confidence documents, empty chunks, and entity hubs. Neo4j export needs `uv sync --extra neo4j` plus `NEO4J_URI`, `NEO4J_USER`, `NEO4J_PASSWORD` env vars (or `--uri/--user/--password`).
Docker The image ships the Tesseract-only core. It reads `/data/screenshots` and writes `/data/output`: ```bash docker build -t ocr-pipeline . docker run -v ~/Pictures:/data/screenshots -v "$PWD/ocr-output:/data/output" ocr-pipeline run docker run -v ~/Pictures:/data/screenshots -v "$PWD/ocr-output:/data/output" ocr-pipeline watch # with ML extras: docker build --build-arg EXTRAS="--extra full" -t ocr-pipeline:full . ```
Config `ocr-pipeline init` writes a minimal config. Lookup order: `$OCR_PIPELINE_CONFIG` → `./config.yaml` → per-user config dir. Every knob, for reference: ```yaml input: paths: ["~/Pictures", "/mnt/storage3/aman/screenshots"] patterns: ["SCR-*.png", "*.jpg", "*.jpeg", "*.tiff"] recursive: true exclude_patterns: ["*.photoslibrary/*"] ocr: engine: "paddleocr" # paddleocr | tesseract | auto languages: ["en", "latin"] use_gpu: false preprocess: deskew: true denoise: true clahe: true adaptive_threshold: true remove_lines: true processing: workers: 4 batch_size: 10 retry_attempts: 2 detectors: figures: enabled: true confidence_threshold: 0.7 tables: enabled: true confidence_threshold: 0.7 entities: enabled: true model: "en_core_sci_lg" citations: regex_enabled: true grobid_enabled: false # set true if you're running a GROBID server chunking: chunk_size: 1000 chunk_overlap: 200 output: base_directory: "./data/ocr_output" organize_by: "date_run" # date_run | source_dir | flat write_consolidated: true consolidated_filename: "all_ocr.md" watch: enabled: true debounce_seconds: 5 db_path: "./data/processed_files.db" ```
Output example Each chunk is a Markdown file with YAML frontmatter: ```markdown --- source_path: "/Users/Aman/Pictures/SCR-20250115-gel.png" source_hash: "a1b2c3d4e5f6..." timestamp: "2025-01-15T10:30:00Z" ocr_engine: "paddleocr" ocr_confidence_mean: 0.91 language: "en" detected_entities: ["GENE", "PROTEIN", "CHEMICAL"] has_figures: true has_tables: false citations_found: ["doi:10.1038/nature12345", "pmid:PMID:12345678"] chunk_index: 0 total_chunks: 2 --- # Screenshot: SCR-20250115-gel.png ## Extracted Text **Western Blot Analysis of BRCA1 Expression** Lane 1: WT control Lane 2: BRCA1 KO (CRISPR) ... ``` Files land in `//run_NNN/`. Chunk files are kept for RAG indexing. `output.write_consolidated: true` also writes `all_ocr.md` with one frontmatter block and all source text grouped by image.
Models and dependencies Weights download on first use, cached in `~/.cache/ocr_pipeline/`: - PaddleOCR models (~200MB) - scispaCy `en_core_sci_lg` (~800MB) - Table Transformer (~500MB) - LayoutParser PubLayNet (~300MB) Notes per capability: - **PaddleOCR:** pinned to the legacy 2.x API. Falls back to Tesseract when it can't initialize; the fallback is logged and recorded in frontmatter. - **Scientific NER:** without scispaCy + `en_core_sci_lg`, a conservative regex extractor runs instead and the backend is marked accordingly. - **Figures:** needs a Detectron2 build matching your Torch/Python. No universal wheel exists, so it's never installed automatically. - **Tables:** downloaded by Transformers on first use; needs `timm`. `ocr-pipeline doctor` checks all of the above and prints remediation hints.
## Life science specifics Gene/protein names go through scispaCy's `en_core_sci_lg`. Chemical formulas and units (µM, ng/mL, kb/Mb/Gb, °C, ×g) get normalized, scientific notation gets cleaned up (`1.5×10⁻³` → `1.5×10^-3`), and gel/blot figures get their captions pulled out separately. - [ ] Entity linking to MeSH/UniProt identifiers - [ ] GROBID-based structured citation parsing (server optional, regex today)