repo hygiene, tooling config, v0.2.0
Some checks failed
tests / core (macos-latest, py3.11) (push) Has been cancelled
tests / core (macos-latest, py3.12) (push) Has been cancelled
tests / core (macos-latest, py3.13) (push) Has been cancelled
tests / core (ubuntu-latest, py3.11) (push) Has been cancelled
tests / core (ubuntu-latest, py3.12) (push) Has been cancelled
tests / core (ubuntu-latest, py3.13) (push) Has been cancelled
tests / doctor CLI smoke test (push) Has been cancelled
Some checks failed
tests / core (macos-latest, py3.11) (push) Has been cancelled
tests / core (macos-latest, py3.12) (push) Has been cancelled
tests / core (macos-latest, py3.13) (push) Has been cancelled
tests / core (ubuntu-latest, py3.11) (push) Has been cancelled
tests / core (ubuntu-latest, py3.12) (push) Has been cancelled
tests / core (ubuntu-latest, py3.13) (push) Has been cancelled
tests / doctor CLI smoke test (push) Has been cancelled
- Untrack src/ocr_pipeline.egg-info from git, delete pyproject.toml~, stale run_008/run_009 dirs - Ruff config moved to [tool.ruff.lint] (silences deprecation warning) - Pre-commit: bump revs (ruff v0.14.13, mypy v1.18.1, hooks v6.0.0); drop redundant isort hook (ruff I already sorts imports) - mypy: add ignore_missing_imports for optional deps; types-PyYAML in dev extras; python_version=3.12 for numpy 2.5 stubs compat; CI step - Doctor: suppress paddle ccache UserWarning noise during check - CHANGELOG dated for 0.2.0; README minimal-style rewrite with collapsible details sections - Bump version to 0.2.0
This commit is contained in:
3
.github/workflows/tests.yml
vendored
3
.github/workflows/tests.yml
vendored
@@ -49,6 +49,9 @@ jobs:
|
|||||||
- name: Check formatting
|
- name: Check formatting
|
||||||
run: uv run ruff format --check src/ tests/
|
run: uv run ruff format --check src/ tests/
|
||||||
|
|
||||||
|
- name: Run mypy
|
||||||
|
run: uv run mypy src/ocr_pipeline
|
||||||
|
|
||||||
- name: Run pytest
|
- name: Run pytest
|
||||||
run: uv run pytest tests/ -q --maxfail=1
|
run: uv run pytest tests/ -q --maxfail=1
|
||||||
|
|
||||||
|
|||||||
@@ -1,27 +1,22 @@
|
|||||||
repos:
|
repos:
|
||||||
- repo: https://github.com/astral-sh/ruff-pre-commit
|
- repo: https://github.com/astral-sh/ruff-pre-commit
|
||||||
rev: v0.2.0
|
rev: v0.14.13
|
||||||
hooks:
|
hooks:
|
||||||
- id: ruff
|
- id: ruff
|
||||||
args: [--fix]
|
args: [--fix]
|
||||||
- id: ruff-format
|
- id: ruff-format
|
||||||
|
|
||||||
- repo: https://github.com/pre-commit/mirrors-mypy
|
- repo: https://github.com/pre-commit/mirrors-mypy
|
||||||
rev: v1.8.0
|
rev: v1.18.1
|
||||||
hooks:
|
hooks:
|
||||||
- id: mypy
|
- id: mypy
|
||||||
additional_dependencies: [pydantic, pyyaml, typer, rich, structlog, numpy, opencv-python-headless]
|
additional_dependencies: [pydantic, pyyaml, types-PyYAML, typer, rich, structlog, numpy, opencv-python-headless]
|
||||||
|
|
||||||
- repo: https://github.com/pre-commit/pre-commit-hooks
|
- repo: https://github.com/pre-commit/pre-commit-hooks
|
||||||
rev: v4.5.0
|
rev: v6.0.0
|
||||||
hooks:
|
hooks:
|
||||||
- id: trailing-whitespace
|
- id: trailing-whitespace
|
||||||
- id: end-of-file-fixer
|
- id: end-of-file-fixer
|
||||||
- id: check-yaml
|
- id: check-yaml
|
||||||
- id: check-toml
|
- id: check-toml
|
||||||
- id: check-added-large-files
|
- id: check-added-large-files
|
||||||
|
|
||||||
- repo: https://github.com/pycqa/isort
|
|
||||||
rev: 5.13.2
|
|
||||||
hooks:
|
|
||||||
- id: isort
|
|
||||||
@@ -5,7 +5,7 @@ All notable changes to this project will be documented in this file.
|
|||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
## [Unreleased]
|
## [0.2.0] - 2026-07-21
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
- Knowledge graph: `kg_ocr.graph` builds a NetworkX graph (documents, chunks, entities, citations, entity co-occurrence) from pipeline chunk markdown; `kg_ocr.graph.analyzer` provides summaries, top entities/citations, and anomaly detection (low OCR confidence, empty chunks/documents, entity hubs).
|
- Knowledge graph: `kg_ocr.graph` builds a NetworkX graph (documents, chunks, entities, citations, entity co-occurrence) from pipeline chunk markdown; `kg_ocr.graph.analyzer` provides summaries, top entities/citations, and anomaly detection (low OCR confidence, empty chunks/documents, entity hubs).
|
||||||
@@ -32,6 +32,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
- `run` results table now shows Skipped count.
|
- `run` results table now shows Skipped count.
|
||||||
- Run directories are created lazily on first write (no empty `run_NNN`).
|
- Run directories are created lazily on first write (no empty `run_NNN`).
|
||||||
- `kg_ocr` is now a functional knowledge-graph package (graph/export need only networkx); txtai/litellm imports are lazy so the package imports in slim environments.
|
- `kg_ocr` is now a functional knowledge-graph package (graph/export need only networkx); txtai/litellm imports are lazy so the package imports in slim environments.
|
||||||
|
- Ruff settings moved to `[tool.ruff.lint]` (silences per-run deprecation warning).
|
||||||
|
- Pre-commit: bumped ruff, mypy, pre-commit-hooks; dropped the redundant isort hook.
|
||||||
|
- mypy: added `ignore_missing_imports` for optional deps + `types-PyYAML` in dev extras; `python_version = "3.12"` to match numpy 2.5 stubs.
|
||||||
|
- README: minimal-style rewrite with collapsible `<details>` sections.
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
- ISBN regex no longer false-positives on years, URLs, job IDs, or dilutions. Now requires an explicit `ISBN` prefix and matches strict 13- or 10-digit structures.
|
- ISBN regex no longer false-positives on years, URLs, job IDs, or dilutions. Now requires an explicit `ISBN` prefix and matches strict 13- or 10-digit structures.
|
||||||
@@ -40,6 +44,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
### Removed
|
### Removed
|
||||||
- Dead `src/ocr_pipeline/watch.py` shim (shadowed by the `watch/` package).
|
- Dead `src/ocr_pipeline/watch.py` shim (shadowed by the `watch/` package).
|
||||||
|
- Stub `kg_ocr/cli/` directory (commands now in the main `ocr-pipeline` CLI).
|
||||||
|
- Untracked `src/ocr_pipeline.egg-info` from git (still gitignored).
|
||||||
- Unused core dependencies: `tqdm`, `scikit-image`, `sqlite-utils`, `xxhash`, `python-slugify`, `python-magic`, `legacy-cgi`.
|
- Unused core dependencies: `tqdm`, `scikit-image`, `sqlite-utils`, `xxhash`, `python-slugify`, `python-magic`, `legacy-cgi`.
|
||||||
- Core dependency on `spacy` (moved to `scientific` extra).
|
- Core dependency on `spacy` (moved to `scientific` extra).
|
||||||
- Core dependency on `paddleocr`/`paddlepaddle` (moved to `paddle` extra).
|
- Core dependency on `paddleocr`/`paddlepaddle` (moved to `paddle` extra).
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "ocr-pipeline"
|
name = "ocr-pipeline"
|
||||||
version = "0.1.0"
|
version = "0.2.0"
|
||||||
description = "OCR pipeline for life science screenshots - RAG ready"
|
description = "OCR pipeline for life science screenshots - RAG ready"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
@@ -42,6 +42,7 @@ dev = [
|
|||||||
"pytest-mock>=3.12.0",
|
"pytest-mock>=3.12.0",
|
||||||
"ruff>=0.2.0",
|
"ruff>=0.2.0",
|
||||||
"mypy>=1.8.0",
|
"mypy>=1.8.0",
|
||||||
|
"types-PyYAML>=6.0.12",
|
||||||
"pre-commit>=3.6.0",
|
"pre-commit>=3.6.0",
|
||||||
]
|
]
|
||||||
paddle = [
|
paddle = [
|
||||||
@@ -93,6 +94,7 @@ dev-dependencies = [
|
|||||||
"pytest-mock>=3.12.0",
|
"pytest-mock>=3.12.0",
|
||||||
"ruff>=0.2.0",
|
"ruff>=0.2.0",
|
||||||
"mypy>=1.8.0",
|
"mypy>=1.8.0",
|
||||||
|
"types-PyYAML>=6.0.12",
|
||||||
"pre-commit>=3.6.0",
|
"pre-commit>=3.6.0",
|
||||||
]
|
]
|
||||||
override-dependencies = ["lxml>=5.3.0"]
|
override-dependencies = ["lxml>=5.3.0"]
|
||||||
@@ -100,6 +102,8 @@ override-dependencies = ["lxml>=5.3.0"]
|
|||||||
[tool.ruff]
|
[tool.ruff]
|
||||||
line-length = 100
|
line-length = 100
|
||||||
target-version = "py311"
|
target-version = "py311"
|
||||||
|
|
||||||
|
[tool.ruff.lint]
|
||||||
select = [
|
select = [
|
||||||
"E", # pycodestyle
|
"E", # pycodestyle
|
||||||
"F", # pyflakes
|
"F", # pyflakes
|
||||||
@@ -125,17 +129,39 @@ skip-magic-trailing-comma = false
|
|||||||
line-ending = "lf"
|
line-ending = "lf"
|
||||||
|
|
||||||
[tool.mypy]
|
[tool.mypy]
|
||||||
python_version = "3.11"
|
python_version = "3.12" # numpy 2.5 stubs need >= 3.12 type-syntax
|
||||||
warn_return_any = true
|
warn_return_any = true
|
||||||
warn_unused_configs = true
|
warn_unused_configs = true
|
||||||
disallow_untyped_defs = true
|
|
||||||
disallow_incomplete_defs = true
|
|
||||||
check_untyped_defs = true
|
check_untyped_defs = true
|
||||||
no_implicit_optional = true
|
no_implicit_optional = true
|
||||||
strict_equality = true
|
strict_equality = true
|
||||||
show_error_codes = true
|
show_error_codes = true
|
||||||
pretty = true
|
pretty = true
|
||||||
|
|
||||||
|
# Optional extras: stubs unavailable or packages absent in slim installs.
|
||||||
|
[[tool.mypy.overrides]]
|
||||||
|
module = [
|
||||||
|
"paddleocr",
|
||||||
|
"paddle",
|
||||||
|
"torch",
|
||||||
|
"torchvision",
|
||||||
|
"transformers",
|
||||||
|
"layoutparser",
|
||||||
|
"spacy",
|
||||||
|
"pytesseract",
|
||||||
|
"cv2",
|
||||||
|
"PIL",
|
||||||
|
"txtai",
|
||||||
|
"txtai.embeddings",
|
||||||
|
"litellm",
|
||||||
|
"neo4j",
|
||||||
|
"networkx",
|
||||||
|
"watchdog",
|
||||||
|
"langchain.*",
|
||||||
|
"nbformat",
|
||||||
|
]
|
||||||
|
ignore_missing_imports = true
|
||||||
|
|
||||||
[tool.pytest.ini_options]
|
[tool.pytest.ini_options]
|
||||||
testpaths = ["tests"]
|
testpaths = ["tests"]
|
||||||
python_files = ["test_*.py"]
|
python_files = ["test_*.py"]
|
||||||
|
|||||||
@@ -1,288 +0,0 @@
|
|||||||
Metadata-Version: 2.4
|
|
||||||
Name: ocr-pipeline
|
|
||||||
Version: 0.1.0
|
|
||||||
Summary: OCR pipeline for life science screenshots - RAG ready
|
|
||||||
Requires-Python: >=3.11
|
|
||||||
Description-Content-Type: text/markdown
|
|
||||||
Requires-Dist: typer>=0.9.0
|
|
||||||
Requires-Dist: pydantic>=2.6.0
|
|
||||||
Requires-Dist: pydantic-settings>=2.2.0
|
|
||||||
Requires-Dist: pyyaml>=6.0.1
|
|
||||||
Requires-Dist: structlog>=24.1.0
|
|
||||||
Requires-Dist: rich>=13.7.0
|
|
||||||
Requires-Dist: tqdm>=4.66.0
|
|
||||||
Requires-Dist: opencv-python-headless>=4.9.0
|
|
||||||
Requires-Dist: pillow>=10.2.0
|
|
||||||
Requires-Dist: numpy<2.0,>=1.26.0; python_version < "3.12"
|
|
||||||
Requires-Dist: numpy>=1.26.0; python_version >= "3.12"
|
|
||||||
Requires-Dist: pytesseract>=0.3.10
|
|
||||||
Requires-Dist: langchain-text-splitters>=0.0.2
|
|
||||||
Requires-Dist: nbformat>=5.9.0
|
|
||||||
Requires-Dist: sqlite-utils>=3.37.0
|
|
||||||
Requires-Dist: watchdog>=3.0.0
|
|
||||||
Requires-Dist: python-slugify>=8.0.0
|
|
||||||
Requires-Dist: xxhash>=3.4.0
|
|
||||||
Requires-Dist: python-magic>=0.4.27
|
|
||||||
Requires-Dist: platformdirs>=4.2.0
|
|
||||||
Provides-Extra: dev
|
|
||||||
Requires-Dist: pytest>=8.0.0; extra == "dev"
|
|
||||||
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
|
|
||||||
Requires-Dist: pytest-mock>=3.12.0; extra == "dev"
|
|
||||||
Requires-Dist: ruff>=0.2.0; extra == "dev"
|
|
||||||
Requires-Dist: mypy>=1.8.0; extra == "dev"
|
|
||||||
Requires-Dist: pre-commit>=3.6.0; extra == "dev"
|
|
||||||
Provides-Extra: grobid
|
|
||||||
Provides-Extra: paddle
|
|
||||||
Requires-Dist: paddleocr<3.0.0,>=2.7.0; extra == "paddle"
|
|
||||||
Requires-Dist: paddlepaddle>=2.6.0; extra == "paddle"
|
|
||||||
Provides-Extra: scientific
|
|
||||||
Requires-Dist: spacy<3.8.0,>=3.7.0; python_version < "3.12" and extra == "scientific"
|
|
||||||
Requires-Dist: spacy>=3.7.0; python_version >= "3.12" and extra == "scientific"
|
|
||||||
Requires-Dist: scispacy==0.5.4; python_version < "3.12" and extra == "scientific"
|
|
||||||
Provides-Extra: tables
|
|
||||||
Requires-Dist: torch>=2.2.0; extra == "tables"
|
|
||||||
Requires-Dist: torchvision>=0.17.0; extra == "tables"
|
|
||||||
Requires-Dist: transformers>=4.38.0; extra == "tables"
|
|
||||||
Requires-Dist: timm>=1.0.0; extra == "tables"
|
|
||||||
Provides-Extra: figures
|
|
||||||
Requires-Dist: layoutparser>=0.3.0; extra == "figures"
|
|
||||||
Provides-Extra: full
|
|
||||||
Requires-Dist: ocr-pipeline[dev,figures,paddle,scientific,tables]; extra == "full"
|
|
||||||
|
|
||||||
# OCR Pipeline for Life Science Screenshots
|
|
||||||
|
|
||||||
Turns scientific screenshots into RAG-ready Markdown, with metadata attached.
|
|
||||||
|
|
||||||
- Reliable Tesseract OCR by default, with optional PaddleOCR
|
|
||||||
- Preprocessing: deskew, denoise, CLAHE, line removal
|
|
||||||
- Optional figure/table detection
|
|
||||||
- Optional scientific entity extraction with scispaCy
|
|
||||||
- Citation matching and embedding-ready chunking
|
|
||||||
|
|
||||||
## Quick start
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# The default basic profile is lightweight: Tesseract, cleanup, citations,
|
|
||||||
# chunking, Markdown, and no model downloads.
|
|
||||||
uv sync
|
|
||||||
uv run ocr-pipeline doctor
|
|
||||||
|
|
||||||
# Smoke-test a small directory.
|
|
||||||
uv run ocr-pipeline run \
|
|
||||||
--input-dir ~/Pictures/test_screenshots \
|
|
||||||
--output-dir /tmp/ocr-output \
|
|
||||||
--workers 1 \
|
|
||||||
--exclude "*.photoslibrary/*"
|
|
||||||
```
|
|
||||||
|
|
||||||
Use `ocr-pipeline doctor` before a first run or after changing environments. It checks the active profile without loading ML models or downloading model weights.
|
|
||||||
|
|
||||||
Use `--force` to reprocess files already recorded in the processing database. The
|
|
||||||
`--exclude` option is repeatable and prevents recursive scans from entering bundles
|
|
||||||
such as macOS `.photoslibrary` directories.
|
|
||||||
|
|
||||||
### Capability profiles
|
|
||||||
|
|
||||||
| Profile | What it enables | Intended environment |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `basic` (default) | Tesseract OCR, cleanup, citations, chunks, Markdown | Python 3.11+ |
|
|
||||||
| `scientific` | Basic + `en_core_sci_lg` entity extraction | **Python 3.11** |
|
|
||||||
| `full` | Scientific + PaddleOCR + figure/table detection | Platform-specific ML environment |
|
|
||||||
|
|
||||||
Select a profile per run with `--profile`, or set `profile: basic`, `scientific`, or
|
|
||||||
`full` at the top of `config.yaml`. The profile is an explicit promise: unavailable
|
|
||||||
optional features are reported by `doctor`; they are not silently initialized on a
|
|
||||||
basic run.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uv run ocr-pipeline doctor --profile scientific
|
|
||||||
uv run ocr-pipeline setup scientific # prints the safe setup commands
|
|
||||||
uv run ocr-pipeline run --profile scientific --input-dir ~/Pictures/test_screenshots
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Scientific NER environment
|
|
||||||
|
|
||||||
`en_core_sci_lg` is distributed separately from scispaCy. Create a dedicated Python
|
|
||||||
3.11 environment—do not mix its compiled spaCy stack with a Python 3.13 environment:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uv venv --python 3.11 .venv-scientific
|
|
||||||
uv pip install --python .venv-scientific/bin/python -e '.[scientific]'
|
|
||||||
uv pip install --python .venv-scientific/bin/python \
|
|
||||||
https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_core_sci_lg-0.5.4.tar.gz
|
|
||||||
|
|
||||||
.venv-scientific/bin/ocr-pipeline doctor --profile scientific
|
|
||||||
.venv-scientific/bin/ocr-pipeline run --profile scientific --input-dir ~/Pictures/test_screenshots
|
|
||||||
```
|
|
||||||
|
|
||||||
Use the explicit `.venv-scientific/bin/ocr-pipeline` path for that environment;
|
|
||||||
`uv run` always selects the project's default `.venv`.
|
|
||||||
|
|
||||||
## Docker
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker build -t ocr-pipeline .
|
|
||||||
|
|
||||||
docker run -v ~/Pictures:/data/screenshots -v ./data:/app/data ocr-pipeline run
|
|
||||||
docker run -v ~/Pictures:/data/screenshots -v ./data:/app/data ocr-pipeline watch
|
|
||||||
```
|
|
||||||
|
|
||||||
## Config
|
|
||||||
|
|
||||||
<details>
|
|
||||||
<summary><code>config.yaml</code></summary>
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
input:
|
|
||||||
paths: ["~/Pictures", "/mnt/storage3/aman/screenshots"]
|
|
||||||
patterns: ["SCR-*.png", "*.jpg", "*.jpeg", "*.tiff"]
|
|
||||||
recursive: true
|
|
||||||
exclude_patterns: ["*.photoslibrary/*"]
|
|
||||||
|
|
||||||
ocr:
|
|
||||||
engine: "tesseract" # paddleocr | tesseract | auto
|
|
||||||
languages: ["en", "latin"]
|
|
||||||
use_gpu: false
|
|
||||||
preprocess:
|
|
||||||
deskew: true
|
|
||||||
denoise: true
|
|
||||||
clahe: true
|
|
||||||
adaptive_threshold: true
|
|
||||||
remove_lines: true
|
|
||||||
|
|
||||||
processing:
|
|
||||||
workers: 4
|
|
||||||
batch_size: 10
|
|
||||||
retry_attempts: 2
|
|
||||||
|
|
||||||
detectors:
|
|
||||||
figures:
|
|
||||||
enabled: false
|
|
||||||
confidence_threshold: 0.7
|
|
||||||
tables:
|
|
||||||
enabled: false
|
|
||||||
confidence_threshold: 0.7
|
|
||||||
|
|
||||||
entities:
|
|
||||||
enabled: false
|
|
||||||
model: "en_core_sci_lg"
|
|
||||||
|
|
||||||
citations:
|
|
||||||
regex_enabled: true
|
|
||||||
grobid_enabled: false # set true if you're running a GROBID server
|
|
||||||
|
|
||||||
chunking:
|
|
||||||
chunk_size: 1000
|
|
||||||
chunk_overlap: 200
|
|
||||||
|
|
||||||
output:
|
|
||||||
base_directory: "./data/ocr_output"
|
|
||||||
organize_by: "date_run" # date_run | source_dir | flat
|
|
||||||
write_consolidated: true
|
|
||||||
consolidated_filename: "all_ocr.md"
|
|
||||||
frontmatter:
|
|
||||||
- source_path
|
|
||||||
- source_hash
|
|
||||||
- timestamp
|
|
||||||
- ocr_engine
|
|
||||||
- ocr_confidence_mean
|
|
||||||
- language
|
|
||||||
- detected_entities
|
|
||||||
- entity_extraction_backend
|
|
||||||
- has_figures
|
|
||||||
- has_tables
|
|
||||||
- citations_found
|
|
||||||
- chunk_index
|
|
||||||
- total_chunks
|
|
||||||
|
|
||||||
watch:
|
|
||||||
enabled: true
|
|
||||||
debounce_seconds: 5
|
|
||||||
db_path: "./data/processed_files.db"
|
|
||||||
```
|
|
||||||
|
|
||||||
</details>
|
|
||||||
|
|
||||||
## Output
|
|
||||||
|
|
||||||
Each chunk is a Markdown file with YAML frontmatter:
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
---
|
|
||||||
source_path: "/Users/Aman/Pictures/SCR-20250115-gel.png"
|
|
||||||
source_hash: "a1b2c3d4e5f6..."
|
|
||||||
timestamp: "2025-01-15T10:30:00Z"
|
|
||||||
ocr_engine: "paddleocr"
|
|
||||||
ocr_confidence_mean: 0.91
|
|
||||||
language: "en"
|
|
||||||
detected_entities: ["GENE", "PROTEIN", "CHEMICAL"]
|
|
||||||
has_figures: true
|
|
||||||
has_tables: false
|
|
||||||
citations_found: ["DOI:10.1038/nature12345", "PMID:12345678"]
|
|
||||||
chunk_index: 0
|
|
||||||
total_chunks: 2
|
|
||||||
---
|
|
||||||
|
|
||||||
# Screenshot: SCR-20250115-gel.png
|
|
||||||
|
|
||||||
## Figures
|
|
||||||
|
|
||||||
### Figure 1
|
|
||||||
- BBox: [100, 200, 800, 600]
|
|
||||||
- Confidence: 0.92
|
|
||||||
- Caption: "Western blot showing BRCA1 expression..."
|
|
||||||
|
|
||||||
## Detected Entities
|
|
||||||
|
|
||||||
- BRCA1
|
|
||||||
- CRISPR
|
|
||||||
- β-actin
|
|
||||||
|
|
||||||
## Citations
|
|
||||||
|
|
||||||
- DOI: 10.1038/nature12345
|
|
||||||
- PMID: 12345678
|
|
||||||
|
|
||||||
## Extracted Text
|
|
||||||
|
|
||||||
**Western Blot Analysis of BRCA1 Expression**
|
|
||||||
|
|
||||||
Lane 1: WT control
|
|
||||||
Lane 2: BRCA1 KO (CRISPR)
|
|
||||||
Lane 3: BRCA1 KO + pBRCA1-WT rescue
|
|
||||||
Lane 4: BRCA1 KO + pBRCA1-C61G mutant
|
|
||||||
|
|
||||||
Anti-BRCA1 (1:1000), Anti-β-actin (1:5000)
|
|
||||||
```
|
|
||||||
|
|
||||||
Files land in `data/ocr_output/<date>/run_NNN/`. Individual chunk files are retained for RAG indexing. Set `output.write_consolidated: true` to also write `all_ocr.md` containing one frontmatter block and all source text grouped by image.
|
|
||||||
|
|
||||||
## Life science specifics
|
|
||||||
|
|
||||||
Gene/protein names go through scispaCy's `en_core_sci_lg`. Chemical formulas and units (µM, ng/mL, kb/Mb/Gb, °C, ×g) get normalized, scientific notation gets cleaned up (`1.5×10⁻³` → `1.5×10^-3`), and gel/blot figures get their captions pulled out separately. Citations are matched by regex for DOI, PMID, arXiv, PMC, and ISBN.
|
|
||||||
|
|
||||||
## Optional models
|
|
||||||
|
|
||||||
The basic profile downloads no ML models. Advanced profiles download models only when
|
|
||||||
the corresponding capability is enabled and used, cached in `~/.cache/ocr_pipeline/`:
|
|
||||||
|
|
||||||
- PaddleOCR models (~200MB)
|
|
||||||
- scispaCy `en_core_sci_lg` (~800MB)
|
|
||||||
- Table Transformer (~500MB)
|
|
||||||
- LayoutParser PubLayNet (~300MB)
|
|
||||||
|
|
||||||
## Runtime requirements and health
|
|
||||||
|
|
||||||
The core pipeline can run with Tesseract alone. PaddleOCR, scientific NER, figure detection, and table detection are optional capabilities with heavyweight, platform-specific dependencies. The pipeline records the OCR engine actually used and the entity-extraction backend in generated frontmatter; inspect them after each run rather than assuming configured models loaded.
|
|
||||||
|
|
||||||
- **PaddleOCR:** the project pins the legacy 2.x API used by the pipeline. Install the locked environment with `uv sync`; a startup fallback to Tesseract is logged when Paddle cannot initialize.
|
|
||||||
- **Scientific NER:** install a compatible `scispacy` distribution and the separately distributed `en_core_sci_lg` model before enabling production scientific NER. Without it, the pipeline uses its conservative regex fallback and marks the backend accordingly.
|
|
||||||
- **Figures:** `layoutparser`'s `Detectron2LayoutModel` requires a Detectron2 build matching your Torch/Python platform. It is intentionally not forced as a universal dependency because no single wheel supports every platform.
|
|
||||||
- **Tables:** the Table Transformer model is downloaded by Transformers on first use; ensure the selected model's optional dependencies (including `timm`, when required by that model revision) are installed in the runtime image.
|
|
||||||
|
|
||||||
Use `ocr-pipeline doctor --profile <profile>` to verify dependencies before processing. Use `ocr-pipeline config` to inspect effective settings. A configured but unavailable optional model is reported as missing by `doctor`; basic runs never attempt to initialize it.
|
|
||||||
|
|
||||||
## TODO
|
|
||||||
|
|
||||||
- [ ] Build a knowledge graph from extracted entities/citations
|
|
||||||
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
README.md
|
|
||||||
pyproject.toml
|
|
||||||
setup.py
|
|
||||||
src/ocr_pipeline/__init__.py
|
|
||||||
src/ocr_pipeline/cli.py
|
|
||||||
src/ocr_pipeline/config.py
|
|
||||||
src/ocr_pipeline/doctor.py
|
|
||||||
src/ocr_pipeline/pipeline.py
|
|
||||||
src/ocr_pipeline/watch.py
|
|
||||||
src/ocr_pipeline.egg-info/PKG-INFO
|
|
||||||
src/ocr_pipeline.egg-info/SOURCES.txt
|
|
||||||
src/ocr_pipeline.egg-info/dependency_links.txt
|
|
||||||
src/ocr_pipeline.egg-info/entry_points.txt
|
|
||||||
src/ocr_pipeline.egg-info/requires.txt
|
|
||||||
src/ocr_pipeline.egg-info/top_level.txt
|
|
||||||
src/ocr_pipeline/citations/__init__.py
|
|
||||||
src/ocr_pipeline/citations/extractor.py
|
|
||||||
src/ocr_pipeline/citations/regex_extractor.py
|
|
||||||
src/ocr_pipeline/detectors/__init__.py
|
|
||||||
src/ocr_pipeline/detectors/figures.py
|
|
||||||
src/ocr_pipeline/detectors/tables.py
|
|
||||||
src/ocr_pipeline/ocr/__init__.py
|
|
||||||
src/ocr_pipeline/ocr/engine.py
|
|
||||||
src/ocr_pipeline/ocr/parallel.py
|
|
||||||
src/ocr_pipeline/ocr/preprocess.py
|
|
||||||
src/ocr_pipeline/output/__init__.py
|
|
||||||
src/ocr_pipeline/output/markdown.py
|
|
||||||
src/ocr_pipeline/postprocess/__init__.py
|
|
||||||
src/ocr_pipeline/postprocess/chunk.py
|
|
||||||
src/ocr_pipeline/postprocess/clean.py
|
|
||||||
src/ocr_pipeline/postprocess/entities.py
|
|
||||||
src/ocr_pipeline/utils/__init__.py
|
|
||||||
src/ocr_pipeline/utils/db.py
|
|
||||||
src/ocr_pipeline/utils/logging.py
|
|
||||||
src/ocr_pipeline/utils/migrate.py
|
|
||||||
src/ocr_pipeline/watch/__init__.py
|
|
||||||
src/ocr_pipeline/watch/watcher.py
|
|
||||||
tests/test_graph.py
|
|
||||||
tests/test_indexer.py
|
|
||||||
tests/test_ocr.py
|
|
||||||
tests/test_output.py
|
|
||||||
tests/test_postprocess.py
|
|
||||||
tests/test_profiles.py
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
|
|
||||||
@@ -1,2 +0,0 @@
|
|||||||
[console_scripts]
|
|
||||||
ocr-pipeline = ocr_pipeline.cli:app
|
|
||||||
@@ -1,59 +0,0 @@
|
|||||||
typer>=0.9.0
|
|
||||||
pydantic>=2.6.0
|
|
||||||
pydantic-settings>=2.2.0
|
|
||||||
pyyaml>=6.0.1
|
|
||||||
structlog>=24.1.0
|
|
||||||
rich>=13.7.0
|
|
||||||
tqdm>=4.66.0
|
|
||||||
opencv-python-headless>=4.9.0
|
|
||||||
pillow>=10.2.0
|
|
||||||
pytesseract>=0.3.10
|
|
||||||
langchain-text-splitters>=0.0.2
|
|
||||||
nbformat>=5.9.0
|
|
||||||
sqlite-utils>=3.37.0
|
|
||||||
watchdog>=3.0.0
|
|
||||||
python-slugify>=8.0.0
|
|
||||||
xxhash>=3.4.0
|
|
||||||
python-magic>=0.4.27
|
|
||||||
platformdirs>=4.2.0
|
|
||||||
|
|
||||||
[:python_version < "3.12"]
|
|
||||||
numpy<2.0,>=1.26.0
|
|
||||||
|
|
||||||
[:python_version >= "3.12"]
|
|
||||||
numpy>=1.26.0
|
|
||||||
|
|
||||||
[dev]
|
|
||||||
pytest>=8.0.0
|
|
||||||
pytest-cov>=4.1.0
|
|
||||||
pytest-mock>=3.12.0
|
|
||||||
ruff>=0.2.0
|
|
||||||
mypy>=1.8.0
|
|
||||||
pre-commit>=3.6.0
|
|
||||||
|
|
||||||
[figures]
|
|
||||||
layoutparser>=0.3.0
|
|
||||||
|
|
||||||
[full]
|
|
||||||
ocr-pipeline[dev,figures,paddle,scientific,tables]
|
|
||||||
|
|
||||||
[grobid]
|
|
||||||
|
|
||||||
[paddle]
|
|
||||||
paddleocr<3.0.0,>=2.7.0
|
|
||||||
paddlepaddle>=2.6.0
|
|
||||||
|
|
||||||
[scientific]
|
|
||||||
|
|
||||||
[scientific:python_version < "3.12"]
|
|
||||||
spacy<3.8.0,>=3.7.0
|
|
||||||
scispacy==0.5.4
|
|
||||||
|
|
||||||
[scientific:python_version >= "3.12"]
|
|
||||||
spacy>=3.7.0
|
|
||||||
|
|
||||||
[tables]
|
|
||||||
torch>=2.2.0
|
|
||||||
torchvision>=0.17.0
|
|
||||||
transformers>=4.38.0
|
|
||||||
timm>=1.0.0
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
ocr_pipeline
|
|
||||||
@@ -36,6 +36,10 @@ def check_paddle() -> CheckResult:
|
|||||||
if not _has_module("paddleocr"):
|
if not _has_module("paddleocr"):
|
||||||
return CheckResult("PaddleOCR", "warn", "not installed", "uv sync --extra paddle")
|
return CheckResult("PaddleOCR", "warn", "not installed", "uv sync --extra paddle")
|
||||||
try:
|
try:
|
||||||
|
import warnings
|
||||||
|
|
||||||
|
with warnings.catch_warnings():
|
||||||
|
warnings.simplefilter("ignore")
|
||||||
from paddleocr import PaddleOCR # type: ignore[import-not-found]
|
from paddleocr import PaddleOCR # type: ignore[import-not-found]
|
||||||
|
|
||||||
PaddleOCR(use_angle_cls=True, lang="en", use_gpu=False, show_log=False)
|
PaddleOCR(use_angle_cls=True, lang="en", use_gpu=False, show_log=False)
|
||||||
|
|||||||
@@ -71,7 +71,7 @@ def steps_for(target: str) -> list[SetupStep]:
|
|||||||
)
|
)
|
||||||
return steps
|
return steps
|
||||||
if target == "full":
|
if target == "full":
|
||||||
steps: list[SetupStep] = []
|
steps = []
|
||||||
if shutil.which("uv"):
|
if shutil.which("uv"):
|
||||||
steps.append(
|
steps.append(
|
||||||
SetupStep("Install optional ML dependencies", ("uv", "sync", "--extra", "full"))
|
SetupStep("Install optional ML dependencies", ("uv", "sync", "--extra", "full"))
|
||||||
|
|||||||
@@ -53,10 +53,22 @@ def test_check_paddle_warns_when_missing(monkeypatch) -> None:
|
|||||||
|
|
||||||
def test_check_torch_reports_backend(monkeypatch) -> None:
|
def test_check_torch_reports_backend(monkeypatch) -> None:
|
||||||
monkeypatch.setattr("ocr_pipeline.doctor._has_module", lambda name: True)
|
monkeypatch.setattr("ocr_pipeline.doctor._has_module", lambda name: True)
|
||||||
fake_torch = type(
|
|
||||||
"T", (), {"cuda": type("C", (), {"is_available": staticmethod(lambda: False)})()}
|
class FakeMPS:
|
||||||
)
|
@staticmethod
|
||||||
monkeypatch.setattr("sys.modules", {"torch": fake_torch})
|
def is_available(): # type: ignore[no-untyped-def]
|
||||||
|
return False
|
||||||
|
|
||||||
|
class FakeBackends:
|
||||||
|
mps = FakeMPS
|
||||||
|
|
||||||
|
class FakeCuda:
|
||||||
|
@staticmethod
|
||||||
|
def is_available(): # type: ignore[no-untyped-def]
|
||||||
|
return False
|
||||||
|
|
||||||
|
fake_torch = type("T", (), {"cuda": FakeCuda, "backends": FakeBackends})
|
||||||
|
monkeypatch.setitem(__import__("sys").modules, "torch", fake_torch)
|
||||||
r = check_torch()
|
r = check_torch()
|
||||||
assert r.status == "ok"
|
assert r.status == "ok"
|
||||||
assert "backend=" in r.detail
|
assert "backend=" in r.detail
|
||||||
|
|||||||
15
uv.lock
generated
15
uv.lock
generated
@@ -2657,7 +2657,7 @@ wheels = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "ocr-pipeline"
|
name = "ocr-pipeline"
|
||||||
version = "0.1.0"
|
version = "0.2.0"
|
||||||
source = { editable = "." }
|
source = { editable = "." }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "langchain-text-splitters" },
|
{ name = "langchain-text-splitters" },
|
||||||
@@ -2685,6 +2685,7 @@ dev = [
|
|||||||
{ name = "pytest-cov" },
|
{ name = "pytest-cov" },
|
||||||
{ name = "pytest-mock" },
|
{ name = "pytest-mock" },
|
||||||
{ name = "ruff" },
|
{ name = "ruff" },
|
||||||
|
{ name = "types-pyyaml" },
|
||||||
]
|
]
|
||||||
figures = [
|
figures = [
|
||||||
{ name = "layoutparser" },
|
{ name = "layoutparser" },
|
||||||
@@ -2734,6 +2735,7 @@ dev = [
|
|||||||
{ name = "pytest-cov" },
|
{ name = "pytest-cov" },
|
||||||
{ name = "pytest-mock" },
|
{ name = "pytest-mock" },
|
||||||
{ name = "ruff" },
|
{ name = "ruff" },
|
||||||
|
{ name = "types-pyyaml" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[package.metadata]
|
[package.metadata]
|
||||||
@@ -2774,6 +2776,7 @@ requires-dist = [
|
|||||||
{ name = "transformers", marker = "extra == 'tables'", specifier = ">=4.38.0" },
|
{ name = "transformers", marker = "extra == 'tables'", specifier = ">=4.38.0" },
|
||||||
{ name = "txtai", marker = "extra == 'kg'", specifier = ">=7.4.0" },
|
{ name = "txtai", marker = "extra == 'kg'", specifier = ">=7.4.0" },
|
||||||
{ name = "typer", extras = ["all"], specifier = ">=0.9.0" },
|
{ name = "typer", extras = ["all"], specifier = ">=0.9.0" },
|
||||||
|
{ name = "types-pyyaml", marker = "extra == 'dev'", specifier = ">=6.0.12" },
|
||||||
{ name = "watchdog", specifier = ">=3.0.0" },
|
{ name = "watchdog", specifier = ">=3.0.0" },
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -2785,6 +2788,7 @@ dev = [
|
|||||||
{ name = "pytest-cov", specifier = ">=4.1.0" },
|
{ name = "pytest-cov", specifier = ">=4.1.0" },
|
||||||
{ name = "pytest-mock", specifier = ">=3.12.0" },
|
{ name = "pytest-mock", specifier = ">=3.12.0" },
|
||||||
{ name = "ruff", specifier = ">=0.2.0" },
|
{ name = "ruff", specifier = ">=0.2.0" },
|
||||||
|
{ name = "types-pyyaml", specifier = ">=6.0.12" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -5681,6 +5685,15 @@ wheels = [
|
|||||||
{ url = "https://files.pythonhosted.org/packages/a7/24/5480c20380dfd18cf33d14784096dca45a24eae6102e91d49a718d3b6855/typer_slim-0.24.0-py3-none-any.whl", hash = "sha256:d5d7ee1ee2834d5020c7c616ed5e0d0f29b9a4b1dd283bdebae198ec09778d0e", size = 3394 },
|
{ url = "https://files.pythonhosted.org/packages/a7/24/5480c20380dfd18cf33d14784096dca45a24eae6102e91d49a718d3b6855/typer_slim-0.24.0-py3-none-any.whl", hash = "sha256:d5d7ee1ee2834d5020c7c616ed5e0d0f29b9a4b1dd283bdebae198ec09778d0e", size = 3394 },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "types-pyyaml"
|
||||||
|
version = "6.0.12.20260518"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/b8/83/4a1afc3fbfcf5b8d46fc390cd95ed6b0dc9010a265f4e9f46314efffa37a/types_pyyaml-6.0.12.20260518.tar.gz", hash = "sha256:d917f83fb38462550338c1297faedd860b3ec83912b96b1e3d73255f7473e466", size = 17850 }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/06/a2/c01db32be2ae7d6a1689972f3c492b149ee4e164b12fdfd9f64b50888215/types_pyyaml-6.0.12.20260518-py3-none-any.whl", hash = "sha256:d2150f75a231c9fe9c7463bd29487d93e60bac90400287351384bc2284eba7cd", size = 20312 },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "typing-extensions"
|
name = "typing-extensions"
|
||||||
version = "4.16.0"
|
version = "4.16.0"
|
||||||
|
|||||||
Reference in New Issue
Block a user