Import AITURK IDE 1.0.0-beta.1 from Hermes 63279301; preserve MIT license

skills/productivity/pdf/references/forms.md

@@ -0,0 +1,99 @@
+# Building Fillable Forms: spec format and workflow
+
+The same JSON spec drives both `pdf_form_layout.py` (design lint) and
+`pdf_make_form.py` (AcroForm build). Coordinates are PDF points, origin
+at the bottom-left of the page (1 pt = 1/72 inch; A4 is 595.27 x 841.89,
+letter is 612 x 792).
+
+## Spec shape
+
+```json
+{
+  "title": "Example Intake Form",
+  "author": "example-author",
+  "page_size": "A4",
+  "page_count": 1,
+  "fields": [
+    {"name": "surname", "type": "text", "page": 1,
+     "label": "Surname", "label_box": [72, 700, 150, 714],
+     "entry_box": [160, 696, 400, 716],
+     "value": "", "tooltip": "Family name"},
+
+    {"name": "agree", "type": "checkbox", "page": 1,
+     "label": "I agree", "label_box": [72, 660, 150, 674],
+     "entry_box": [160, 658, 176, 674], "checked": false},
+
+    {"name": "color", "type": "radio", "page": 1,
+     "label": "Color", "label_box": [72, 620, 150, 634],
+     "entry_box": [160, 616, 400, 636],
+     "options": ["red", "blue"], "value": "blue"},
+
+    {"name": "size", "type": "dropdown", "page": 1,
+     "label": "Size", "label_box": [72, 580, 150, 594],
+     "entry_box": [160, 576, 300, 596],
+     "options": ["small", "large"], "value": "small"}
+  ]
+}
+```
+
+- `page_size`: `"A4"`, `"letter"`, or `[width, height]` in points.
+- `page_count`: optional; extended automatically to the highest field page.
+- Boxes are `[x0, y0, x1, y1]` with `x0 < x1`, `y0 < y1`.
+- `label` is drawn as static text near `label_box`; omit it (and
+  `label_box`) for unlabeled fields.
+- `radio`: the buttons are laid out left-to-right inside `entry_box`,
+  one slot per option, each with a small static caption. `value`
+  pre-selects an option by its export name.
+- `dropdown` maps to an AcroForm choice (combo) field.
+
+## Field types → what pdf_read.py --fields reports
+
+| Spec type | /FT | value format after fill |
+|---|---|---|
+| text | /Tx (`text`) | the string |
+| checkbox | /Btn (`button`) | `/Yes` or `/Off` |
+| radio | /Btn (`button`) | `/<export>`, e.g. `/red` |
+| dropdown | /Ch (`choice`) | the option string |
+
+When filling with `pdf_fill_form.py`, checkboxes accept `true`/`false`;
+radio values need the leading slash (`"/red"`); dropdown values are the
+plain option string.
+
+## Layout lint rules (pdf_form_layout.py)
+
+Per field, on its declared page:
+
+- boxes must be well-formed and inside the page bounds;
+- entry boxes must be at least 8x8 pt (12 pt tall for text/dropdown);
+- no two entry boxes on the same page may overlap (the second and later
+  fields of an overlapping cluster are flagged);
+- a label must sit within 150 pt of its entry box and must not overlap it.
+
+Exit code 0 = clean, 1 = at least one problem; the JSON report lists
+per-field `problems`. Lint the spec BEFORE building — fixing numbers in
+JSON is cheaper than debugging a rendered PDF.
+
+## Visual review loop
+
+```bash
+python3 scripts/pdf_form_layout.py spec.json --render-overlay overlay.png [--pdf built.pdf]
+```
+
+Red rectangles = entry boxes (with field names), blue = label boxes.
+Without `--pdf` the overlay is drawn on a blank page (PIL-only, always
+works); with `--pdf` the real page is rasterized underneath
+(needs pypdfium2 or pdftoppm — otherwise the report says
+`"rendered": false` with install hints). Feed the PNG to `vision_analyze`
+and ask specifically about collisions, alignment, and stray labels.
+
+## Radio-group quirks (reportlab + pypdf)
+
+- reportlab requires at least two `radio()` calls per group; a
+  single-option radio group produces a broken field.
+- Pre-selecting is done at build time via `"value"`; changing selection
+  later via `pdf_fill_form.py` needs the slashed export name (`"/red"`).
+- Some viewers render reportlab radio appearances inconsistently after a
+  pypdf fill; verify with `--fields` (data truth) plus a rendered page
+  image (visual truth) rather than either alone.
+- Flattening radio groups is the least reliable flatten case — check the
+  output image before shipping.

skills/productivity/pdf/references/nano-pdf-editing.md

@@ -0,0 +1,40 @@
+# Natural-language PDF text editing with nano-pdf (merged from the nano-pdf skill)
+# nano-pdf
+
+Edit PDFs using natural-language instructions. Point it at a page and describe what to change. For structural PDF work (merge, split, forms, watermarks, creation), see the `pdf` skill; for text extraction from scans, see `ocr-and-documents`.
+
+## Prerequisites
+
+```bash
+# Install with uv (recommended — already available in Hermes)
+uv pip install nano-pdf
+
+# Or with pip
+pip install nano-pdf
+```
+
+## Usage
+
+```bash
+nano-pdf edit <file.pdf> <page_number> "<instruction>"
+```
+
+## Examples
+
+```bash
+# Change a title on page 1
+nano-pdf edit deck.pdf 1 "Change the title to 'Q3 Results' and fix the typo in the subtitle"
+
+# Update a date on a specific page
+nano-pdf edit report.pdf 3 "Update the date from January to February 2026"
+
+# Fix content
+nano-pdf edit contract.pdf 2 "Change the client name from 'Acme Corp' to 'Acme Industries'"
+```
+
+## Notes
+
+- Page numbers may be 0-based or 1-based depending on version — if the edit hits the wrong page, retry with ±1
+- Always verify the output PDF after editing (use `read_file` to check file size, or open it)
+- The tool uses an LLM under the hood — requires an API key (check `nano-pdf --help` for config)
+- Works well for text changes; complex layout modifications may need a different approach

skills/productivity/pdf/references/ocr-extraction.md

@@ -0,0 +1,165 @@
+# OCR & Document Text Extraction (merged from the ocr-and-documents skill)
+
+Scripts referenced below live in this skill's scripts/ directory.
+# PDF & Document Extraction
+
+For DOCX: see the `docx` skill (create/edit) or use `python-docx` for structured reads.
+For PPTX: see the `powerpoint` skill (full create/read/edit support).
+For PDF manipulation (merge, split, forms, watermarks, creation): see the `pdf` skill.
+This skill covers **text extraction from PDFs and scanned documents**.
+
+> **Coming from a `read_file` EXTRACTION COVERAGE WARNING?** `read_file` auto-converts local PDFs but reads the text layer only; the warning footer lists the pages that yielded no text (scanned images). For a handful of pages, render + vision is fastest: `pdftoppm -jpeg -r 150 -f N -l N file.pdf /tmp/page` then `vision_analyze` each image. For bulk OCR of many pages, use marker-pdf below (Step 2).
+
+## Step 1: Remote URL Available?
+
+If the document has a URL, **always try `web_extract` first**:
+
+```
+web_extract(urls=["https://arxiv.org/pdf/2402.03300"])
+web_extract(urls=["https://example.com/report.pdf"])
+```
+
+This handles PDF-to-markdown conversion via Firecrawl with no local dependencies.
+
+Only use local extraction when: the file is local, web_extract fails, or you need batch processing.
+
+## Step 2: Choose Local Extractor
+
+| Feature | pymupdf (~25MB) | marker-pdf (~3-5GB) |
+|---------|-----------------|---------------------|
+| **Text-based PDF** | ✅ | ✅ |
+| **Scanned PDF (OCR)** | ❌ | ✅ (90+ languages) |
+| **Tables** | ✅ (basic) | ✅ (high accuracy) |
+| **Equations / LaTeX** | ❌ | ✅ |
+| **Code blocks** | ❌ | ✅ |
+| **Forms** | ❌ | ✅ |
+| **Headers/footers removal** | ❌ | ✅ |
+| **Reading order detection** | ❌ | ✅ |
+| **Images extraction** | ✅ (embedded) | ✅ (with context) |
+| **Images → text (OCR)** | ❌ | ✅ |
+| **EPUB** | ✅ | ✅ |
+| **Markdown output** | ✅ (via pymupdf4llm) | ✅ (native, higher quality) |
+| **Install size** | ~25MB | ~3-5GB (PyTorch + models) |
+| **Speed** | Instant | ~1-14s/page (CPU), ~0.2s/page (GPU) |
+
+**Decision**: Use pymupdf unless you need OCR, equations, forms, or complex layout analysis.
+
+If the user needs marker capabilities but the system lacks ~5GB free disk:
+> "This document needs OCR/advanced extraction (marker-pdf), which requires ~5GB for PyTorch and models. Your system has [X]GB free. Options: free up space, provide a URL so I can use web_extract, or I can try pymupdf which works for text-based PDFs but not scanned documents or equations."
+
+---
+
+## pymupdf (lightweight)
+
+```bash
+pip install pymupdf pymupdf4llm
+```
+
+**Via helper script**:
+```bash
+python scripts/extract_pymupdf.py document.pdf              # Plain text
+python scripts/extract_pymupdf.py document.pdf --markdown    # Markdown
+python scripts/extract_pymupdf.py document.pdf --tables      # Tables
+python scripts/extract_pymupdf.py document.pdf --images out/ # Extract images
+python scripts/extract_pymupdf.py document.pdf --metadata    # Title, author, pages
+python scripts/extract_pymupdf.py document.pdf --pages 0-4   # Specific pages
+```
+
+**Inline**:
+```bash
+python -c "
+import pymupdf
+doc = pymupdf.open('document.pdf')
+for page in doc:
+    print(page.get_text())
+"
+```
+
+---
+
+## marker-pdf (high-quality OCR)
+
+```bash
+# Check disk space first
+python scripts/extract_marker.py --check
+
+pip install marker-pdf
+```
+
+**Via helper script**:
+```bash
+python scripts/extract_marker.py document.pdf                # Markdown
+python scripts/extract_marker.py document.pdf --json         # JSON with metadata
+python scripts/extract_marker.py document.pdf --output_dir out/  # Save images
+python scripts/extract_marker.py scanned.pdf                 # Scanned PDF (OCR)
+python scripts/extract_marker.py document.pdf --use_llm      # LLM-boosted accuracy
+```
+
+**CLI** (installed with marker-pdf):
+```bash
+marker_single document.pdf --output_dir ./output
+marker /path/to/folder --workers 4    # Batch
+```
+
+---
+
+## Arxiv Papers
+
+```
+# Abstract only (fast)
+web_extract(urls=["https://arxiv.org/abs/2402.03300"])
+
+# Full paper
+web_extract(urls=["https://arxiv.org/pdf/2402.03300"])
+
+# Search
+web_search(query="arxiv GRPO reinforcement learning 2026")
+```
+
+## Split, Merge & Search
+
+pymupdf handles these natively — use `execute_code` or inline Python:
+
+```python
+# Split: extract pages 1-5 to a new PDF
+import pymupdf
+doc = pymupdf.open("report.pdf")
+new = pymupdf.open()
+for i in range(5):
+    new.insert_pdf(doc, from_page=i, to_page=i)
+new.save("pages_1-5.pdf")
+```
+
+```python
+# Merge multiple PDFs
+import pymupdf
+result = pymupdf.open()
+for path in ["a.pdf", "b.pdf", "c.pdf"]:
+    result.insert_pdf(pymupdf.open(path))
+result.save("merged.pdf")
+```
+
+```python
+# Search for text across all pages
+import pymupdf
+doc = pymupdf.open("report.pdf")
+for i, page in enumerate(doc):
+    results = page.search_for("revenue")
+    if results:
+        print(f"Page {i+1}: {len(results)} match(es)")
+        print(page.get_text("text"))
+```
+
+No extra dependencies needed — pymupdf covers split, merge, search, and text extraction in one package.
+
+---
+
+## Notes
+
+- `web_extract` is always first choice for URLs
+- pymupdf is the safe default — instant, no models, works everywhere
+- marker-pdf is for OCR, scanned docs, equations, complex layouts — install only when needed
+- Both helper scripts accept `--help` for full usage
+- marker-pdf downloads ~2.5GB of models to `~/.cache/huggingface/` on first use
+- For Word docs: `pip install python-docx` (better than OCR — parses actual structure)
+- For PowerPoint: see the `powerpoint` skill (uses python-pptx)