video-edit-planner-skill/SKILL.md

---
name: video-edit-planner
description: "Use when planning video edits: transcribe audio, extract key frames, analyze visuals, and produce cut/transition/assembly plans via iterative dialogue."
license: MIT
tags:
  [
    video-editing,
    transcription,
    funasr,
    frame-extraction,
    vision,
    planning,
    ffmpeg,
  ]
---

# Video Edit Planner

## Overview

A conversational video-editing planning assistant. The user provides a video file and an audio track index; the skill transcribes the audio, extracts key frames on demand, analyzes them visually, and produces an iterative Markdown-table edit plan (timecodes, segment descriptions, transitions, notes).

The skill is **agent-driven, not scripted**: it provides tools (transcription, clipping, frame extraction, index management) and guidance for the agent to autonomously decide when and what to extract based on user needs.

## When to Use

- User provides a video path and wants help planning cuts, transitions, or assembly.
- User asks "help me edit this video" or "what parts should I cut from this recording."
- User needs to understand video content beyond what the transcript alone reveals.
- User wants an iterative dialogue to refine an edit plan over multiple turns.

**Don't use for:** actual video editing/rendering (this skill plans, it does not produce video files), or pure audio-only transcription.

## Workflow

### Step 1 — Check dependencies

Before any processing, verify the environment. Missing dependencies must be **installed only with explicit user consent**.

| Dependency           | Check command          | Install (prefer OS package manager; fallback to pip)                                                                                         |
| -------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `ffmpeg` + `ffprobe` | `which ffmpeg ffprobe` | Linux: `sudo pacman -S ffmpeg` / `sudo apt install ffmpeg`; Windows: `winget install Gyan.FFmpeg`; macOS: `brew install ffmpeg`              |
| `uv`                 | `which uv`             | Linux: `sudo pacman -S uv` / `sudo apt install uv` (if available); macOS: `brew install uv`; Windows: `winget install astral-sh.uv`; fallback: `pip install uv` |
| `python3`            | `which python3`        | Linux: `sudo pacman -S python`; Windows: `winget install Python.Python.3`; macOS: `brew install python`                                      |

**Completion criteria:** all three dependencies confirmed present, or user has explicitly declined installation (in which case stop — the skill cannot proceed).

### Step 2 — Gather inputs

Ask the user for:

1. **Video file path** (absolute path; translate Windows paths like `C:\Users\...\x.mkv` to `/mnt/c/Users/.../x.mkv` on WSL).
2. **Audio track index** — the ffmpeg stream index, not ordinal. If unknown, list tracks:
   ```bash
   ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout:stream_tags=language,title -of json VIDEO_PATH
   ```

**Completion criteria:** both video path and track index confirmed.

### Step 3 — Ask editing requirements

Before transcribing, ask the user what they want to do with the video. Examples:

- "Trim out dead air and boring parts"
- "Make a highlight reel of funny moments"
- "Plan transitions between scenes"
- "Find the best 30-second clip for a short"

User requirements are often vague in the first pass — that's expected. The plan is refined iteratively.

**Completion criteria:** user has described at least a rough editing goal.

### Step 4 — Transcribe audio

Use the bundled `funasr-script` in `scripts/transcription/`. **Prefer Fun-ASR-Nano** as the default transcription model — it has higher accuracy and sensitivity on Chinese audio. Use SenseVoice only for fast previews.

**First run** requires `uv sync` (installs funasr + torch, may take several minutes). Subsequent runs reuse the cached venv. Notify the user before first-time setup.

```bash
# Recommended default: Fun-ASR-Nano (high quality, Chinese-optimized)
uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --track TRACK_INDEX --output-dir VIDEO_DIR

# Fast preview only: SenseVoice
uv run --directory SKILL_DIR/scripts/transcription funasr-fast VIDEO_PATH --track TRACK_INDEX --output-dir VIDEO_DIR

# List tracks
uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --list-tracks
```

**Skip if cached:** check the index file (Step 5) for an existing transcription entry. If one exists and the video file hasn't changed, reuse it.

**Completion criteria:** a transcription JSON exists and is recorded in the index. JSON contains `segments` with timestamps and text.

### Step 5 — Manage index file

An SQLite database tracks all processing artifacts to avoid duplicate work. It lives **next to the video file**.

```
<video_dir>/<video_stem>.vedit.db
```

Schema (managed by `scripts/index/manage_index.py`):

```sql
CREATE TABLE IF NOT EXISTS transcription (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    json_path TEXT NOT NULL,
    track_index INTEGER NOT NULL,
    created_at REAL NOT NULL,    -- Unix timestamp
    duration_s REAL
);

CREATE TABLE IF NOT EXISTS clips (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    start_time REAL NOT NULL,    -- seconds from video start
    end_time REAL NOT NULL,
    path TEXT NOT NULL,
    created_at REAL NOT NULL,    -- Unix timestamp
    reason TEXT                  -- why this clip was extracted
);

CREATE TABLE IF NOT EXISTS frames (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    clip_id INTEGER NOT NULL REFERENCES clips(id),
    timestamp REAL NOT NULL,     -- seconds from video start
    path TEXT NOT NULL,
    scene_score REAL,            -- ffmpeg scene score if available
    method TEXT NOT NULL,        -- 'scene' | 'sample' | 'both'
    created_at REAL NOT NULL     -- Unix timestamp
);
```

Use `scripts/index/manage_index.py` for all CRUD operations:

```bash
# Initialize database (idempotent)
uv run --directory SKILL_DIR python scripts/index/manage_index.py init --video VIDEO_PATH

# Add transcription record
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-transcription --json-path PATH --track-index N --duration S

# Query existing transcription
uv run --directory SKILL_DIR python scripts/index/manage_index.py get-transcription

# Add a clip record
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-clip --start S --end E --path PATH --reason "user asked about this segment"

# Add frame records (batch from a directory)
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-frames --clip-id N --frames-dir DIR --method scene

# List all clips and frames
uv run --directory SKILL_DIR python scripts/index/manage_index.py list

# Check if a time range has already been extracted
uv run --directory SKILL_DIR python scripts/index/manage_index.py check-range --start S --end E
```

**Completion criteria:** index file exists and all produced artifacts are recorded in it.

### Step 6 — Extract clips and frames (on demand)

**When to extract:** the agent autonomously decides based on user needs. If the transcript alone is insufficient to answer (e.g., user asks about visual content, or a transcript segment is sparse/empty for a time range the user cares about), extract frames for that range.

**Binary-search-style frame analysis:**

1. Extract a clip for the target time range.
2. Extract scene-change frames + uniform samples from the clip.
3. Send a few representative frames to vision analysis.
4. If a sub-range needs deeper understanding, extract more frames from that sub-range.
5. Once the relevant range is identified, send all frames in that range (in batches if needed) to vision.

**Clip extraction** (keyframe-accurate, `-c copy`):

```bash
ffmpeg -hide_banner -y -ss START -to END -i VIDEO_PATH -c copy -map 0:v:0 CLIP_PATH
```

**Frame extraction** (scene detection + uniform sampling, hardware-accelerated, downsampled):

```bash
# Hardware-accelerated (CUDA/NVDEC preferred)
ffmpeg -hide_banner -y -hwaccel cuda -i CLIP_PATH \
  -vf "scale=1280:-2,fps=1/2,select='gt(scene,THRESHOLD)+gte(n,0)',showinfo" \
  -vsync vfr -q:v 3 FRAMES_DIR/frame_%04d.jpg

# CPU fallback
ffmpeg -hide_banner -y -i CLIP_PATH \
  -vf "scale=1280:-2,fps=1/2,select='gt(scene,THRESHOLD)',showinfo" \
  -vsync vfr -q:v 3 FRAMES_DIR/frame_%04d.jpg
```

**Parameters the agent chooses:**

| Parameter             | Default                | Guidance                                                                                                          |
| --------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `scale` width         | `1280`                 | Raise to `1920` for short clips (<30s) where detail matters; lower to `854` for long clips (>120s) to save tokens |
| `THRESHOLD`           | `0.3`                  | Lower to `0.1`–`0.2` for gameplay footage (subtle scene changes); raise to `0.4` for dynamic content              |
| `fps` sample rate     | `1/2` (1 frame per 2s) | Raise to `1` (1fps) for short clips; lower to `1/5` for long clips                                                |
| `-q:v` (JPEG quality) | `3`                    | Range 2–31, lower = higher quality. 3 is high quality; 5 is a good balance                                        |

**All clips and frames are saved next to the video file** (e.g., `<video_stem>_clips/` and `<video_stem>_frames/`). The user manages cleanup.

**Record every extracted clip and frame in the index** before proceeding.

**Completion criteria:** clips and frames exist on disk and are recorded in the index.

### Step 7 — Vision analysis

Use whatever vision capability the agent's runtime provides to analyze extracted frames. The agent chooses:

- **Which model** to use (depends on the agent's runtime and available vision capabilities).
- **Batch size** — depends on the model's context window. See `references/frame-extraction-guide.md` for token cost estimates at different resolutions.
- **How many frames** per analysis pass — use the binary-search approach from Step 6.

**Completion criteria:** the agent has enough visual understanding to answer the user's question or produce the requested edit plan.

### Step 8 — Produce edit plan

Output a Markdown table with overall recommendations, then let the user iterate. **Respond in the same language the user is using.**

```markdown
## Edit Plan

### Overall Recommendation

<1-3 sentences of high-level recommendation addressing the user's goal>

### Detailed Plan

| Timecode          | Duration | Segment Description             | Action        | Transition | Notes              |
| ----------------- | -------- | ------------------------------- | ------------- | ---------- | ------------------ |
| 00:01:23–00:01:45 | 22s      | Character enters town, dialogue | Keep          | Hard cut   | Can add subtitles  |
| 00:01:45–00:02:10 | 25s      | Walking, no dialogue or event   | Cut           | —          | —                  |
| 00:02:10–00:02:35 | 25s      | Combat starts, highlight moment | Keep, slow-mo | Fade in    | Sync to BGM rhythm |
| ...               | ...      | ...                             | ...           | ...        | ...                |
```

**Completion criteria:** user receives a Markdown table addressing their stated goal, and is invited to ask follow-up questions.

### Step 9 — Iterate

The user will likely ask follow-ups: "what about this section?", "add a transition here", "can you check what's happening at 5:30?". For each:

1. Check if the answer can be derived from existing transcript + frames (check index first).
2. If not, extract new clips/frames for the requested range (Step 6).
3. Analyze and update the plan.

**Completion criteria:** user is satisfied with the plan or explicitly ends the session.

## Finding stock materials (optional, on demand)

When the user needs editing materials — stickers, GIFs, meme video clips, B-roll — the agent can help search or provide links for the user to search themselves. This section is **only triggered when the user asks for materials**; it is never a mandatory step.

### Material sources

| Source | URL | Content | Best for |
|---|---|---|---|
| 爱给网 | https://www.aigei.com/ | Chinese meme video clips (热梗, 4.2万+), free download | Chinese-language creators (B站/抖音) |
| B站素材酷 | https://cool.bilibili.com/ | Bilibili platform-native assets (video, audio, BGM, templates, stickers) | B站 creators; large Chinese meme collection |
| Tenor | https://tenor.com/ | GIFs + stickers, 30+ languages incl. Chinese, free API | Quick GIF/sticker search |
| GIPHY | https://giphy.com/ | Largest GIF library, 30+ languages, sticker channel | English/multilingual GIF search |
| Klipy | https://klipy.com/ | Localized GIF/sticker/clip API, supports Chinese | API-integrated search, localization |

### Priority by language

If the user communicates in Chinese, **recommend 爱给网 and B站素材酷 first** — they have the most Chinese-relevant meme and sticker content. For non-Chinese users, Tenor and GIPHY are good defaults.

### Search strategy

- **Expand search terms** based on semantic understanding of the user's request. If the user says "I want a clip of someone looking shocked", search terms might include: "惊讶 / 震惊 / 吃惊 / 哇 / wtf / shocked / surprised". Use the user's language as the base, expand with synonyms and internet slang.
- The agent can either **search directly** (using web tools or API calls) or **suggest expanded search terms** for the user to search manually.
- If the user only wants links, provide the source URLs above and let the user search themselves.

### B站素材酷 special handling

B站素材酷 (cool.bilibili.com) has no built-in search function. Two options:

1. **Recommend the userscript**: [bcut-resource-search](https://git.nite07.com/nite/bcut-resource-search) — a Tampermonkey script that adds search to 素材酷, supporting video, sticker/image, music, sound effect, and template categories.
2. **Search via API directly**: the agent can read the userscript's source code to discover the API endpoints and make requests directly (via HTTP tools or curl). Note: the site has CORS restrictions and UA-based access control, so direct API calls may require appropriate headers. If the agent's HTTP tools are blocked, fall back to suggesting the userscript to the user.

### Anti-bot fallback

Some material sites may block automated access (anti-bot, Cloudflare, CAPTCHA). If a web request or HTTP tool is blocked:

1. Try with different headers (User-Agent, Accept-Language) if the tool allows.
2. Fall back to a browser-based tool (e.g., Playwright, Puppeteer, or any browser automation the agent's runtime provides) to access the content.
3. If browser tools are also blocked, inform the user and provide the direct URL for manual search.

## Long-running operations

Before any operation expected to take >1 minute (first-time `uv sync`, transcription of long videos, large frame extraction):

1. **Notify the user** that a long operation is about to start and give a rough time estimate.
2. Run the operation in a way that does not block the conversation.
3. Do not poll progress at high frequency — wait for completion notification rather than checking every few seconds.

## Self-contained setup

The skill bundles its own transcription project under `scripts/transcription/`. On first use:

```bash
cd SKILL_DIR/scripts/transcription && uv sync
```

This creates an isolated venv with `funasr`, `torch`, `torchaudio`. If the user has previously installed these packages via uv in other projects, uv's cache will reuse them — first-time cost is only the link step.

## Common pitfalls

1. **Installing dependencies without consent.** Always ask the user first. If they decline, stop — the skill cannot proceed without ffmpeg, uv, and python3.
2. **Re-processing already-transcribed videos.** Always check the index file first. If a transcription record exists and the video hasn't been modified, reuse it.
3. **Extracting frames from the entire video.** This is extremely slow for long videos. Always clip the target range first with `-c copy`, then extract frames from the clip.
4. **Using full-resolution frames.** A 2560×1600 PNG can cost 2000+ vision tokens. Downsample to 1280 width with `scale=1280:-2` and use JPEG (`-q:v 3`).
5. **Forgetting to record artifacts in the index.** Every clip and frame batch must be recorded. Otherwise subsequent turns will re-extract the same content.
6. **Assuming the transcript is sufficient.** Gameplay videos often have long stretches of action with no dialogue. The agent should proactively check whether visual analysis is needed for segments the user asks about.
7. **Mixing funasr-script flags.** The bundled `funasr-nano`/`funasr-fast` use `--track STREAM_INDEX` and `--output-dir DIR`. Do not use the legacy `~/.local/bin/funasr-transcribe` wrapper's flags.
8. **Hardcoding tool names.** This skill is agent-agnostic. Do not assume specific MCP tools or APIs exist. Use whatever the runtime provides for vision analysis, background execution, and user notification.
9. **Hardcoding output language.** The edit plan table and all user-facing text must follow the user's language, not a fixed language. The English table in Step 8 is a template — translate columns and content to match the user's language at runtime.

## Verification checklist

- [ ] All dependencies (ffmpeg, uv, python3) confirmed present or user declined.
- [ ] Video path and audio track index confirmed.
- [ ] User's editing goal understood.
- [ ] Transcription completed or reused from cache.
- [ ] Index file initialized at `<video_dir>/<video_stem>.vedit.db`.
- [ ] All extracted clips and frames recorded in the index.
- [ ] Vision analysis performed where needed.
- [ ] Edit plan produced as Markdown table with overall recommendation.
- [ ] User invited to iterate.