improve the OCR pipeline processing and outputs formatting
This commit is contained in:
13
README.md
13
README.md
@@ -154,7 +154,7 @@ Lane 4: BRCA1 KO + pBRCA1-C61G mutant
|
||||
Anti-BRCA1 (1:1000), Anti-β-actin (1:5000)
|
||||
```
|
||||
|
||||
Files land in `data/ocr_output/<date>/run_NNN/`.
|
||||
Files land in `data/ocr_output/<date>/run_NNN/`. Individual chunk files are retained for RAG indexing. Set `output.write_consolidated: true` to also write `all_ocr.md` containing one frontmatter block and all source text grouped by image.
|
||||
|
||||
## Life science specifics
|
||||
|
||||
@@ -169,6 +169,17 @@ First run downloads what it needs, cached in `~/.cache/ocr_pipeline/`:
|
||||
- Table Transformer (~500MB)
|
||||
- LayoutParser PubLayNet (~300MB)
|
||||
|
||||
## Runtime requirements and health
|
||||
|
||||
The core pipeline can run with Tesseract alone. PaddleOCR, scientific NER, figure detection, and table detection are optional capabilities with heavyweight, platform-specific dependencies. The pipeline records the OCR engine actually used and the entity-extraction backend in generated frontmatter; inspect them after each run rather than assuming configured models loaded.
|
||||
|
||||
- **PaddleOCR:** the project pins the legacy 2.x API used by the pipeline. Install the locked environment with `uv sync`; a startup fallback to Tesseract is logged when Paddle cannot initialize.
|
||||
- **Scientific NER:** install a compatible `scispacy` distribution and the separately distributed `en_core_sci_lg` model before enabling production scientific NER. Without it, the pipeline uses its conservative regex fallback and marks the backend accordingly.
|
||||
- **Figures:** `layoutparser`'s `Detectron2LayoutModel` requires a Detectron2 build matching your Torch/Python platform. It is intentionally not forced as a universal dependency because no single wheel supports every platform.
|
||||
- **Tables:** the Table Transformer model is downloaded by Transformers on first use; ensure the selected model's optional dependencies (including `timm`, when required by that model revision) are installed in the runtime image.
|
||||
|
||||
Use `ocr-pipeline config` to verify effective settings. A missing optional model is logged and produces empty results for that detector instead of failing a complete batch.
|
||||
|
||||
## TODO
|
||||
|
||||
- [ ] Build a knowledge graph from extracted entities/citations
|
||||
|
||||
Reference in New Issue
Block a user