feat: video-edit-planner skill — transcribe, extract frames, plan edits

A conversational video-editing planning assistant. Provides tools for: - Audio transcription via bundled funasr-script (Fun-ASR-Nano / SenseVoice) - On-demand clip extraction and frame sampling (ffmpeg, hardware-accelerated) - SQLite index to track all artifacts and avoid duplicate processing - Vision analysis guidance with binary-search-style frame sampling - Iterative Markdown-table edit plan output Agent-agnostic: no platform-specific tool names, works with any agent runtime. Dependencies checked at guidance level with OS package manager install hints.
2026-07-14 11:07:57 +08:00
commit a565382a52
13 changed files with 3371 additions and 0 deletions
@@ -0,0 +1,120 @@
+# Frame Extraction & Vision Token Guide
+
+## Vision model token costs at common resolutions
+
+All major vision models (OpenAI GPT-4o, Anthropic Claude, Google Gemini) process images as tiled patches. Higher resolution = more tokens = higher cost and latency.
+
+### OpenAI GPT-4o / GPT-4.1
+
+- **Low detail:** fixed 85 tokens per image.
+- **High detail:** fit inside 2048×2048 box, scale shortest side to ~768px, count 512×512 tiles, then `85 + 170 × tiles`.
+- 1024×1024 → ~765 tokens
+- 1280×800 → ~935 tokens
+- 1920×1080 → ~1105 tokens
+
+Source: [OpenAI vision docs](https://platform.openai.com/docs/guides/vision), [Spoold token estimator](https://www.spoold.com/tools/vision-tokens)
+
+### Anthropic Claude
+
+- Images are processed in 28×28 pixel patches (visual tokens).
+- Token cost: `⌈width / 28⌉ × ⌈height / 28⌉`
+- **Standard tier** (most models): max long edge 1568px, max 1568 visual tokens.
+- **High-res tier** (Opus 4.7/4.8, Sonnet 5, Fable 5, Mythos 5): max long edge 2576px, max 4784 visual tokens.
+- Images larger than either limit are downscaled before processing.
+- 1024×1024 → ~1296 tokens (standard)
+- 1280×800 → ~1334 tokens (standard)
+- 1920×1080 → ~1560 tokens (standard, capped), ~2691 tokens (high-res)
+- > 20 images per request: each image must be ≤2000px on both sides.
+- Max 100 images per API request (200k context models).
+
+Source: [Claude vision docs](https://platform.claude.com/docs/en/build-with-claude/vision)
+
+### Google Gemini
+
+- ≤384×384: 258 tokens per image.
+- Larger: 768×768 tiles × 258 tokens per tile.
+- 1280×800 → ~1032 tokens
+- 1920×1080 → ~1548 tokens
+
+Source: [Spoold token estimator](https://www.spoold.com/tools/vision-tokens)
+
+## Recommended resolution by use case
+
+| Clip duration | Recommended width | JPEG quality | Est. tokens/frame (GPT-4o) | Rationale                                               |
+| ------------- | ----------------- | ------------ | -------------------------- | ------------------------------------------------------- |
+| <15s          | 1920              | 2            | ~1105                      | Short clips need detail; few frames extracted           |
+| 15–60s        | 1280              | 3            | ~935                       | Balance detail vs. token cost                           |
+| 60–180s       | 1280              | 3            | ~935                       | Moderate length; standard default                       |
+| >180s         | 854               | 5            | ~680                       | Long clips produce many frames; minimize per-frame cost |
+
+## Batch size guidance
+
+| Model context | Max images/batch | Practical batch | Notes                                             |
+| ------------- | ---------------- | --------------- | ------------------------------------------------- |
+| 8k tokens     | 3–4              | 2–3             | Leave room for text prompt + response             |
+| 32k tokens    | 15–20            | 5–8             | Conservative; allows multi-turn follow-ups        |
+| 128k tokens   | 60+              | 10–15           | Still batch to maintain response quality          |
+| 200k tokens   | 100+             | 15–20           | Large batches ok but response quality may degrade |
+
+**Rule of thumb:** start with 5–8 frames per batch. If the model handles it well, increase. If responses are shallow or cut off, decrease.
+
+## Binary-search frame analysis workflow
+
+```
+1. Extract clip (e.g., 00:01:00–00:02:00, 60s)
+2. Extract frames: scene + uniform (fps=0.5) → ~30 frames + scene changes
+3. Select 5 representative frames (evenly spaced across the range)
+4. Send to vision: "What is happening in these frames? Summarize the visual content."
+5. Identify sub-range of interest (e.g., 00:01:20–00:01:35 looks relevant)
+6. Extract denser frames from sub-range (fps=2, threshold=0.15)
+7. Send sub-range frames to vision for detailed analysis
+8. Use combined transcript + visual understanding to answer user question
+```
+
+## FFmpeg scene detection parameters
+
+### Threshold guide
+
+| Content type              | Recommended `scene` threshold | Notes                                              |
+| ------------------------- | ----------------------------- | -------------------------------------------------- |
+| Gameplay (subtle changes) | 0.1–0.2                       | Cuts between similar-looking scenes                |
+| Standard video            | 0.25–0.35                     | Typical content with scene cuts                    |
+| Dynamic/high-action       | 0.35–0.45                     | Many scene changes; higher threshold reduces noise |
+
+### Hardware acceleration
+
+Check support:
+
+```bash
+ffmpeg -hwaccels
+```
+
+Common methods:
+
+- `cuda` — NVIDIA NVDEC/NVENC (preferred if available)
+- `vaapi` — Intel/AMD
+- `qsv` — Intel QuickSync
+- `vulkan` — cross-platform GPU
+
+### One-step vs two-step frame extraction
+
+**One-step (recommended):** decode once, filter for scene changes + uniform sampling simultaneously.
+
+```bash
+ffmpeg -hwaccel cuda -i clip.mkv \
+  -vf "scale=1280:-2,fps=0.5,select='gt(scene\,0.3)',showinfo" \
+  -vsync vfr -q:v 3 frame_%04d.jpg
+```
+
+**Two-step:** run `scdet` filter first to get timestamps, then extract frames at those timestamps. Slower for many frames but allows precise per-timestamp extraction.
+
+```bash
+# Step 1: detect scene changes
+ffmpeg -i clip.mkv -vf scdet -f null - 2> scdet.log
+# Step 2: extract frame at each timestamp
+for t in $(grep scdet.score scdet.log | ...); do
+  ffmpeg -ss $t -i clip.mkv -frames:v 1 frame_${t}.jpg
+done
+```
+
+Prefer one-step unless you need per-timestamp control.