remove notebooks

- Delete notebooks/ (personal scratch artifacts; gitignored)
- Move nbformat to --extra migrate; lazy imports in migrate.py and
  cli.py so base CLI loads without it
- Restore langchain-text-splitters in core (chunking dependency)

.gitignore

@@ -79,4 +79,5 @@ Thumbs.db
 
 # Project specific
 data/
+notebooks/
 .env

CHANGELOG.md

@@ -13,7 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `kg_ocr.embeddings.config_loader.KgConfig` for traversal tunables and Neo4j credentials from env.
 - `kg_ocr.export`: node-link JSON (lossless round-trip), GraphML for Gephi/Cytoscape, and batched MERGE export to Neo4j via `Neo4jExporter`.
 - CLI: `ocr-pipeline kg build|stats|query|export` subcommands with lazy kg_ocr imports.
-- Optional extras: `kg` (networkx + txtai + litellm + python-dotenv), `neo4j` (driver).
+- Optional extras: `kg` (networkx + txtai + litellm + python-dotenv), `neo4j` (driver), `migrate` (nbformat).
 - `init` command: writes a minimal `config.yaml` with platform-correct defaults and checks for Tesseract.
 - `demo` command: runs the pipeline on a bundled sample screenshot, prints the output Markdown, and verifies the install in <30s.
 - `setup [basic|full]` command: executes install steps with confirmation (`--yes`/`--dry-run`).
@@ -46,6 +46,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Dead `src/ocr_pipeline/watch.py` shim (shadowed by the `watch/` package).
 - Stub `kg_ocr/cli/` directory (commands now in the main `ocr-pipeline` CLI).
 - Untracked `src/ocr_pipeline.egg-info` from git (still gitignored).
+- `notebooks/` (personal scratch artifacts; gitignored).
+- `nbformat` from core dependencies (moved to `migrate` extra).
 - Unused core dependencies: `tqdm`, `scikit-image`, `sqlite-utils`, `xxhash`, `python-slugify`, `python-magic`, `legacy-cgi`.
 - Core dependency on `spacy` (moved to `scientific` extra).
 - Core dependency on `paddleocr`/`paddlepaddle` (moved to `paddle` extra).

notebooks/03_testing.ipynb

@@ -1,125 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "508336f4",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from kg_ocr import get_screenshots, extract_text, create_and_index, retrieve, ask_wllm"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "11055f85",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.\n"
-     ]
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "20e3c61fe191485da79a32e823ccd1ec",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Loading weights:   0%|          | 0/103 [00:00<?, ?it/s]"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "\u001b[1mBertModel LOAD REPORT\u001b[0m from: sentence-transformers/all-MiniLM-L6-v2\n",
-      "Key                     | Status     |  | \n",
-      "------------------------+------------+--+-\n",
-      "embeddings.position_ids | UNEXPECTED |  | \n",
-      "\n",
-      "\u001b[3mNotes:\n",
-      "- UNEXPECTED\u001b[3m\t:can be ignored when loading from different task/architecture; not ok if you expect identical arch.\u001b[0m\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "[0.739] This slide focuses on the effect of slow fluctuations in population size on the effective\n",
-      "population size (V.) and emphasizes the conditions under whi...\n",
-      "[0.722] This slide explains how rapid fluctuations in population size influence the effective population size\n",
-      "(N-), a key parameter in population genetics. Un...\n",
-      "[0.712] Variable population size\n",
-      "\n",
-      "Beyond the Standard Neutral Model\n",
-      "\n",
-      "Slow fluctuations\n",
-      "in population size : = =\n",
-      "\n",
-      "4 Need:\n",
-      "A, 7 T << min[N, |\n",
-      "\n",
-      "...\n",
-      "## How Population Size Fluctuations Affect Effective Population Size\n",
-      "\n",
-      "Based on the provided documents, population size fluctuations significantly impact effective population size (Ne) in the following ways:\n",
-      "\n",
-      "**1. Harmonic Mean Effect**\n",
-      "Unlike the arithmetic mean, Ne reflects the *harmonic mean* of population sizes over time. This is disproportionately affected by periods of small population size. For example, if a population fluctuates between N and N/4, the effective population size becomes N/2—\"significantly smaller than the actual average population size\" (first slide).\n",
-      "\n",
-      "**2. Reduction from Small Population Periods**\n",
-      "Smaller populations have higher probabilities of coalescence, which reduces genetic diversity. Even brief reductions in population size can greatly lower Ne. This is because \"smaller populations have higher probabilities of coalescence, reducing genetic diversity\" (first slide).\n",
-      "\n",
-      "**3. Time Scale Matters**\n",
-      "The effect depends on whether fluctuations are rapid or slow:\n",
-      "- **Rapid fluctuations**: The harmonic mean formula accurately represents Ne over the fluctuation period.\n",
-      "- **Slow fluctuations**: Occur when the time period of interest (T') is \"much shorter than the minimum population size (min[Nt]) across the fluctuation cycle\" (second slide). In this case, population size appears relatively stable, and \"the harmonic mean formula may not accurately represent the effective population size over longer periods\" (second slide).\n",
-      "\n",
-      "The key takeaway is that for meaningful harmonic mean calculation, \"the time scale of observation (T') must be significantly smaller than the scale of population size changes\" (second slide).\n"
-     ]
-    }
-   ],
-   "source": [
-    "screenshots = get_screenshots(\"/Users/Aman/Pictures\")\n",
-    "texts = extract_text(screenshots)\n",
-    "embeddings = create_and_index(texts)\n",
-    "results = retrieve(embeddings, \"population size fluctuations\")\n",
-    "for r in results:\n",
-    "    print(f\"[{r['score']:.3f}] {r['text'][:150]}...\")\n",
-    "\n",
-    "answer = ask_wllm(embeddings, \"How do population size fluctuations affect effective population size?\")\n",
-    "print(answer)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "base",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.12.7"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

pyproject.toml

@@ -24,15 +24,12 @@ dependencies = [
     "numpy>=1.26.0; python_version >= '3.12'",
     "pytesseract>=0.3.10",
 
-    # Text processing and notebook migration
-    "langchain-text-splitters>=0.0.2",
-    "nbformat>=5.9.0",
-
     # File watching
     "watchdog>=3.0.0",
 
     # Utilities
     "platformdirs>=4.2.0",
+    "langchain-text-splitters>=0.0.2",
 ]
 
 [project.optional-dependencies]
@@ -72,6 +69,9 @@ kg = [
 neo4j = [
     "neo4j>=5.0",
 ]
+migrate = [
+    "nbformat>=5.9.0",
+]
 grobid = []
 full = [
     "ocr-pipeline[paddle,tables,figures,scientific]",

src/ocr_pipeline/cli.py

@@ -21,7 +21,6 @@ from ocr_pipeline.onboarding import (
 from ocr_pipeline.pipeline import OCRPipeline
 from ocr_pipeline.setup_steps import run_steps, steps_for
 from ocr_pipeline.utils.logging import get_logger, setup_logging
-from ocr_pipeline.utils.migrate import migrate_notebook
 
 app = typer.Typer(
     name="ocr-pipeline",
@@ -218,6 +217,8 @@ def reprocess(
     notebook: Path = typer.Argument(..., help="Path to Jupyter notebook"),
 ):
     """Reprocess screenshots from a Jupyter notebook"""
+    from ocr_pipeline.utils.migrate import migrate_notebook
+
     pipeline = OCRPipeline()
     console.print(f"[green]Migrating from notebook: {notebook}[/green]")
     result = migrate_notebook(notebook, pipeline)

src/ocr_pipeline/utils/migrate.py

@@ -6,8 +6,6 @@ import platform
 from pathlib import Path
 from typing import Any
 
-import nbformat
-
 from ocr_pipeline.utils.logging import get_logger
 
 logger = get_logger(__name__)
@@ -36,6 +34,8 @@ def _get_pictures_dir() -> Path:
 def parse_notebook_for_images(notebook_path: Path) -> list[Path]:
     image_paths = []
     try:
+        import nbformat
+
         with open(notebook_path) as f:
             nb = nbformat.read(f, as_version=4)
 

uv.lock

@@ -1627,7 +1627,7 @@ wheels = [
 
 [[package]]
 name = "langsmith"
-version = "0.10.6"
+version = "0.10.9"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
     { name = "anyio" },
@@ -1645,9 +1645,9 @@ dependencies = [
     { name = "xxhash" },
     { name = "zstandard" },
 ]
-sdist = { url = "https://files.pythonhosted.org/packages/93/7e/05db6baaed1383386afd34e5d8fbf0b6cff1e87ed6b5c98d444511af6490/langsmith-0.10.6.tar.gz", hash = "sha256:6ec5b26bb57ee41363c07eb2884f24d69910b8d6bc26cae1f9e367af15259f1b", size = 4729044 }
+sdist = { url = "https://files.pythonhosted.org/packages/ed/6d/9ad4427662ef131878f0f928d5a9e9913e0dda4b6511bb03e8722f1dee8a/langsmith-0.10.9.tar.gz", hash = "sha256:195bc67c964a6370cb91742ce9fa07ce69bfae47977f0fb3f41d125b3435d03a", size = 4736750 }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/fb/d5/2d8d84f5330aa773bae18644aaf7e3288b2f082e31640080784a7d908a24/langsmith-0.10.6-py3-none-any.whl", hash = "sha256:094285b755224f49fd396c3672b0f747294c7b8d9e8278a81d76b487b5cfdb56", size = 661257 },
+    { url = "https://files.pythonhosted.org/packages/ed/54/509a1eb6b9e5572a4f3f4b087779d240c6b69f3d5247ffa21314f66155d9/langsmith-0.10.9-py3-none-any.whl", hash = "sha256:5e0e8ab0f8df05710809919184495e33c2a7c9a9a5e8861d63dd12c1226d9c79", size = 673326 },
 ]
 
 [[package]]
@@ -2661,7 +2661,6 @@ version = "0.2.0"
 source = { editable = "." }
 dependencies = [
     { name = "langchain-text-splitters" },
-    { name = "nbformat" },
     { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" },
     { name = "numpy", version = "2.5.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" },
     { name = "opencv-python-headless" },
@@ -2709,6 +2708,9 @@ kg = [
     { name = "python-dotenv" },
     { name = "txtai" },
 ]
+migrate = [
+    { name = "nbformat" },
+]
 neo4j = [
     { name = "neo4j" },
 ]
@@ -2745,7 +2747,7 @@ requires-dist = [
     { name = "legacy-cgi", marker = "python_full_version >= '3.13' and extra == 'figures'", specifier = ">=2.6.2" },
     { name = "litellm", marker = "extra == 'kg'", specifier = ">=1.40.0" },
     { name = "mypy", marker = "extra == 'dev'", specifier = ">=1.8.0" },
-    { name = "nbformat", specifier = ">=5.9.0" },
+    { name = "nbformat", marker = "extra == 'migrate'", specifier = ">=5.9.0" },
     { name = "neo4j", marker = "extra == 'neo4j'", specifier = ">=5.0" },
     { name = "networkx", marker = "extra == 'kg'", specifier = ">=3.2" },
     { name = "numpy", marker = "python_full_version < '3.12'", specifier = ">=1.26.0,<2.0" },