update README pyproject & config

README.md

@@ -12,15 +12,49 @@ Turns scientific screenshots into RAG-ready Markdown, with metadata attached.
 ## Quick start
 
 ```bash
-uv sync           # or: pip install -e .
+uv sync
 pre-commit install
 
-ocr-pipeline run                  # process screenshots
-ocr-pipeline watch                # watch a folder
-ocr-pipeline reprocess ocr_sc.ipynb   # pull screenshots out of a notebook
-ocr-pipeline status
+# Smoke test a small directory with the lightweight OCR path
+uv run ocr-pipeline run \
+  --input-dir ~/Pictures/test_screenshots \
+  --output-dir /tmp/ocr-output \
+  --engine tesseract \
+  --workers 1 \
+  --exclude "*.photoslibrary/*"
+
+uv run ocr-pipeline watch
+uv run ocr-pipeline reprocess ocr_sc.ipynb
+uv run ocr-pipeline status
 ```
 
+Use `--force` to reprocess files already recorded in the processing database. The
+`--exclude` option is repeatable and prevents recursive scans from entering bundles
+such as macOS `.photoslibrary` directories.
+
+### Scientific NER environment
+
+`en_core_sci_lg` is distributed separately from scispaCy and is not installed by
+`uv sync`. The compatible scispaCy 0.5.4 release requires spaCy 3.7.x, so install it
+in a dedicated Python 3.11 environment rather than mixing it with the main Python
+3.13 environment:
+
+```bash
+uv venv --python 3.11 .venv-scispacy
+source .venv-scispacy/bin/activate
+uv pip install "spacy>=3.7,<3.8" "scispacy==0.5.4" click
+uv pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_core_sci_lg-0.5.4.tar.gz
+uv pip install -e .
+uv pip install timm
+
+# Confirm the model and command use this same environment
+.venv-scispacy/bin/python -c "import spacy; spacy.load('en_core_sci_lg'); print('scispaCy OK')"
+.venv-scispacy/bin/ocr-pipeline run --help
+```
+
+If the two environments are kept separate, invoke the pipeline with the explicit
+`.venv-scispacy/bin/ocr-pipeline` path; `uv run` selects the project's default `.venv`.
+
 ## Docker
 
 ```bash
@@ -40,6 +74,7 @@ input:
   paths: ["~/Pictures", "/mnt/storage3/aman/screenshots"]
   patterns: ["SCR-*.png", "*.jpg", "*.jpeg", "*.tiff"]
   recursive: true
+  exclude_patterns: ["*.photoslibrary/*"]
 
 ocr:
   engine: "paddleocr"  # paddleocr | tesseract | auto
@@ -80,6 +115,8 @@ chunking:
 output:
   base_directory: "./data/ocr_output"
   organize_by: "date_run"  # date_run | source_dir | flat
+  write_consolidated: true
+  consolidated_filename: "all_ocr.md"
   frontmatter:
     - source_path
     - source_hash
@@ -88,6 +125,7 @@ output:
     - ocr_confidence_mean
     - language
     - detected_entities
+    - entity_extraction_backend
     - has_figures
     - has_tables
     - citations_found