Metadata-Version: 2.4
Name: ocr-pipeline
Version: 0.1.0
Summary: OCR pipeline for life science screenshots - RAG ready
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: typer[all]>=0.9.0
Requires-Dist: pydantic>=2.6.0
Requires-Dist: pydantic-settings>=2.2.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: structlog>=24.1.0
Requires-Dist: rich>=13.7.0
Requires-Dist: tqdm>=4.66.0
Requires-Dist: opencv-python-headless>=4.9.0
Requires-Dist: pillow>=10.2.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: scikit-image>=0.22.0
Requires-Dist: paddleocr>=2.7.0
Requires-Dist: pytesseract>=0.3.10
Requires-Dist: torch>=2.2.0
Requires-Dist: torchvision>=0.17.0
Requires-Dist: transformers>=4.38.0
Requires-Dist: layoutparser>=0.3.0
Requires-Dist: langchain-text-splitters>=0.0.2
Requires-Dist: spacy>=3.7.0
Requires-Dist: sqlite-utils>=3.37.0
Requires-Dist: watchdog>=3.0.0
Requires-Dist: python-slugify>=8.0.0
Requires-Dist: xxhash>=3.4.0
Requires-Dist: python-magic>=0.4.27
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: pytest-mock>=3.12.0; extra == "dev"
Requires-Dist: ruff>=0.2.0; extra == "dev"
Requires-Dist: mypy>=1.8.0; extra == "dev"
Requires-Dist: pre-commit>=3.6.0; extra == "dev"
Provides-Extra: grobid
Requires-Dist: grobid-client>=0.8.0; extra == "grobid"
Provides-Extra: full
Requires-Dist: ocr-pipeline[dev]; extra == "full"
Requires-Dist: ocr-pipeline[grobid]; extra == "full"

# OCR Pipeline for Life Science Screenshots

A production-ready OCR pipeline that extracts text from scientific screenshots and converts them into RAG-ready Markdown files with rich metadata.

## Features

- **Multi-engine OCR**: PaddleOCR (primary) with Tesseract fallback
- **Image preprocessing**: Deskewing, denoising, CLAHE contrast enhancement, line removal
- **Parallel processing**: Multi-process worker pool for throughput
- **Scientific entity recognition**: Genes, proteins, chemicals, species, diseases, cell lines via scispaCy
- **Figure & table detection**: LayoutParser + Table Transformer
- **Citation extraction**: DOI, PMID, arXiv, PMC, ISBN via regex + optional GROBID
- **Semantic chunking**: LangChain recursive splitter optimized for scientific text
- **RAG-ready output**: Markdown with YAML frontmatter (source, hash, timestamp, entities, confidence)
- **Watch mode**: File system monitoring with SQLite persistence for incremental processing
- **Notebook migration**: Reprocess screenshots from existing Jupyter notebooks
- **Life-science focused**: Handles scientific notation, units, gene symbols, chemical formulas

## Quick Start

```bash
# Install with uv (recommended)
uv sync

# Or with pip
pip install -e .

# Install pre-commit hooks
pre-commit install

# Run on your screenshots
ocr-pipeline run

# Watch for new screenshots
ocr-pipeline watch

# Reprocess from notebook
ocr-pipeline reprocess ocr_sc.ipynb

# Check status
ocr-pipeline status
```

## Configuration

Edit `config.yaml`:

```yaml
input:
  paths: ["~/Pictures", "/mnt/storage3/aman/screenshots"]
  patterns: ["SCR-*.png", "*.jpg", "*.jpeg", "*.tiff"]
  recursive: true

ocr:
  engine: "paddleocr"  # paddleocr | tesseract | auto
  languages: ["en", "latin"]
  use_gpu: false
  preprocess:
    deskew: true
    denoise: true
    clahe: true
    adaptive_threshold: true
    remove_lines: true

processing:
  workers: 4
  batch_size: 10
  retry_attempts: 2

detectors:
  figures:
    enabled: true
    confidence_threshold: 0.7
  tables:
    enabled: true
    confidence_threshold: 0.7

entities:
  enabled: true
  model: "en_core_sci_lg"  # scispaCy large model

citations:
  regex_enabled: true
  grobid_enabled: false  # Set true if running GROBID server

chunking:
  chunk_size: 1000
  chunk_overlap: 200

output:
  base_directory: "./data/ocr_output"
  organize_by: "date_run"  # date_run | source_dir | flat
  frontmatter:
    - source_path
    - source_hash
    - timestamp
    - ocr_engine
    - ocr_confidence_mean
    - language
    - detected_entities
    - has_figures
    - has_tables
    - citations_found
    - chunk_index
    - total_chunks

watch:
  enabled: true
  debounce_seconds: 5
  db_path: "./data/processed_files.db"
```

## Output Format

Each chunk produces a Markdown file with YAML frontmatter:

```markdown
---
source_path: "/Users/Aman/Pictures/SCR-20250115-gel.png"
source_hash: "a1b2c3d4e5f6..."
timestamp: "2025-01-15T10:30:00Z"
ocr_engine: "paddleocr"
ocr_confidence_mean: 0.91
language: "en"
detected_entities: ["GENE", "PROTEIN", "CHEMICAL"]
has_figures: true
has_tables: false
citations_found: ["DOI:10.1038/nature12345", "PMID:12345678"]
chunk_index: 0
total_chunks: 2
---

# Screenshot: SCR-20250115-gel.png

## Figures

### Figure 1
- **BBox**: [100, 200, 800, 600]
- **Confidence**: 0.92
- **Caption**: "Western blot showing BRCA1 expression..."

## Detected Entities

- BRCA1
- CRISPR
- β-actin

## Citations

- DOI: 10.1038/nature12345
- PMID: 12345678

## Extracted Text

**Western Blot Analysis of BRCA1 Expression**

Lane 1: WT control
Lane 2: BRCA1 KO (CRISPR)
Lane 3: BRCA1 KO + pBRCA1-WT rescue
Lane 4: BRCA1 KO + pBRCA1-C61G mutant

Anti-BRCA1 (1:1000), Anti-β-actin (1:5000)
```

## Directory Structure

```
data/ocr_output/
├── 2025-01-15/
│   ├── run_001/
│   │   ├── SCR-20250115-gel_chunk_000.md
│   │   └── SCR-20250115-gel_chunk_001.md
│   └── run_002/
│       └── ...
└── 2025-01-16/
    └── run_001/
        └── ...
```

## Docker

```bash
# Build
docker build -t ocr-pipeline .

# Run once
docker run -v ~/Pictures:/data/screenshots -v ./data:/app/data ocr-pipeline run

# Watch mode
docker run -v ~/Pictures:/data/screenshots -v ./data:/app/data ocr-pipeline watch
```

## Life Science Optimizations

| Feature | Implementation |
|---------|----------------|
| Gene/Protein names | scispaCy `en_core_sci_lg` NER |
| Chemical formulas | Regex + unit normalization (µM, ng/mL, kb, etc.) |
| Scientific notation | `1.5×10⁻³` → `1.5×10^-3` |
| Gel/blot lanes | Figure detection + caption extraction |
| Citations | DOI, PMID, arXiv, PMC, ISBN patterns |
| Units | µM, ng/mL, kb/Mb/Gb, °C, ×g, etc. |

## Development

```bash
# Install dev dependencies
uv sync --dev

# Run tests
pytest

# Lint
ruff check .
ruff format .

# Type check
mypy src/
```

## Model Downloads

First run downloads models automatically:
- PaddleOCR detection/recognition models (~200MB)
- scispaCy `en_core_sci_lg` (~800MB)
- Table Transformer (~500MB)
- LayoutParser PubLayNet (~300MB)

Cache location: `~/.cache/ocr_pipeline/`

## License

MIT
