--- name: video-edit-planner description: "Use when planning video edits: transcribe audio, extract key frames, analyze visuals, and produce cut/transition/assembly plans via iterative dialogue." license: MIT tags: [ video-editing, transcription, funasr, frame-extraction, vision, planning, ffmpeg, ] --- # Video Edit Planner ## Overview A conversational video-editing planning assistant. The user provides a video file and an audio track index; the skill transcribes the audio, extracts key frames on demand, analyzes them visually, and produces an iterative Markdown-table edit plan (timecodes, segment descriptions, transitions, notes). The skill is **agent-driven, not scripted**: it provides tools (transcription, clipping, frame extraction, index management) and guidance for the agent to autonomously decide when and what to extract based on user needs. ## When to Use - User provides a video path and wants help planning cuts, transitions, or assembly. - User asks "help me edit this video" or "what parts should I cut from this recording." - User needs to understand video content beyond what the transcript alone reveals. - User wants an iterative dialogue to refine an edit plan over multiple turns. **Don't use for:** actual video editing/rendering (this skill plans, it does not produce video files), or pure audio-only transcription. ## Workflow ### Step 1 — Check dependencies Before any processing, verify the environment. Missing dependencies must be **installed only with explicit user consent**. | Dependency | Check command | Install (prefer OS package manager; fallback to pip) | | -------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `ffmpeg` + `ffprobe` | `which ffmpeg ffprobe` | Linux: `sudo pacman -S ffmpeg` / `sudo apt install ffmpeg`; Windows: `winget install Gyan.FFmpeg`; macOS: `brew install ffmpeg` | | `uv` | `which uv` | Linux: `sudo pacman -S uv` / `sudo apt install uv` (if available); macOS: `brew install uv`; Windows: `winget install astral-sh.uv`; fallback: `pip install uv` | | `python3` | `which python3` | Linux: `sudo pacman -S python`; Windows: `winget install Python.Python.3`; macOS: `brew install python` | **Completion criteria:** all three dependencies confirmed present, or user has explicitly declined installation (in which case stop — the skill cannot proceed). ### Step 2 — Gather inputs Ask the user for: 1. **Video file path** (absolute path; translate Windows paths like `C:\Users\...\x.mkv` to `/mnt/c/Users/.../x.mkv` on WSL). 2. **Audio track index** — the ffmpeg stream index, not ordinal. If unknown, list tracks: ```bash ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout:stream_tags=language,title -of json VIDEO_PATH ``` **Completion criteria:** both video path and track index confirmed. ### Step 3 — Ask editing requirements Before transcribing, ask the user what they want to do with the video. Examples: - "Trim out dead air and boring parts" - "Make a highlight reel of funny moments" - "Plan transitions between scenes" - "Find the best 30-second clip for a short" User requirements are often vague in the first pass — that's expected. The plan is refined iteratively. **Completion criteria:** user has described at least a rough editing goal. ### Step 4 — Transcribe audio Use the bundled `funasr-script` in `scripts/transcription/`. **First run** requires `uv sync` (installs funasr + torch, may take several minutes). Subsequent runs reuse the cached venv. Notify the user before first-time setup. ```bash # Default: Fun-ASR-Nano (high quality, Chinese-optimized) uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --track TRACK_INDEX --output-dir VIDEO_DIR # Fast preview: SenseVoice uv run --directory SKILL_DIR/scripts/transcription funasr-fast VIDEO_PATH --track TRACK_INDEX --output-dir VIDEO_DIR # List tracks uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --list-tracks ``` **Skip if cached:** check the index file (Step 5) for an existing transcription entry. If one exists and the video file hasn't changed, reuse it. **Completion criteria:** a transcription JSON exists and is recorded in the index. JSON contains `segments` with timestamps and text. ### Step 5 — Manage index file An SQLite database tracks all processing artifacts to avoid duplicate work. It lives **next to the video file**. ``` /.vedit.db ``` Schema (managed by `scripts/index/manage_index.py`): ```sql CREATE TABLE IF NOT EXISTS transcription ( id INTEGER PRIMARY KEY AUTOINCREMENT, json_path TEXT NOT NULL, track_index INTEGER NOT NULL, created_at REAL NOT NULL, -- Unix timestamp duration_s REAL ); CREATE TABLE IF NOT EXISTS clips ( id INTEGER PRIMARY KEY AUTOINCREMENT, start_time REAL NOT NULL, -- seconds from video start end_time REAL NOT NULL, path TEXT NOT NULL, created_at REAL NOT NULL, -- Unix timestamp reason TEXT -- why this clip was extracted ); CREATE TABLE IF NOT EXISTS frames ( id INTEGER PRIMARY KEY AUTOINCREMENT, clip_id INTEGER NOT NULL REFERENCES clips(id), timestamp REAL NOT NULL, -- seconds from video start path TEXT NOT NULL, scene_score REAL, -- ffmpeg scene score if available method TEXT NOT NULL, -- 'scene' | 'sample' | 'both' created_at REAL NOT NULL -- Unix timestamp ); ``` Use `scripts/index/manage_index.py` for all CRUD operations: ```bash # Initialize database (idempotent) uv run --directory SKILL_DIR python scripts/index/manage_index.py init --video VIDEO_PATH # Add transcription record uv run --directory SKILL_DIR python scripts/index/manage_index.py add-transcription --json-path PATH --track-index N --duration S # Query existing transcription uv run --directory SKILL_DIR python scripts/index/manage_index.py get-transcription # Add a clip record uv run --directory SKILL_DIR python scripts/index/manage_index.py add-clip --start S --end E --path PATH --reason "user asked about this segment" # Add frame records (batch from a directory) uv run --directory SKILL_DIR python scripts/index/manage_index.py add-frames --clip-id N --frames-dir DIR --method scene # List all clips and frames uv run --directory SKILL_DIR python scripts/index/manage_index.py list # Check if a time range has already been extracted uv run --directory SKILL_DIR python scripts/index/manage_index.py check-range --start S --end E ``` **Completion criteria:** index file exists and all produced artifacts are recorded in it. ### Step 6 — Extract clips and frames (on demand) **When to extract:** the agent autonomously decides based on user needs. If the transcript alone is insufficient to answer (e.g., user asks about visual content, or a transcript segment is sparse/empty for a time range the user cares about), extract frames for that range. **Binary-search-style frame analysis:** 1. Extract a clip for the target time range. 2. Extract scene-change frames + uniform samples from the clip. 3. Send a few representative frames to vision analysis. 4. If a sub-range needs deeper understanding, extract more frames from that sub-range. 5. Once the relevant range is identified, send all frames in that range (in batches if needed) to vision. **Clip extraction** (keyframe-accurate, `-c copy`): ```bash ffmpeg -hide_banner -y -ss START -to END -i VIDEO_PATH -c copy -map 0:v:0 CLIP_PATH ``` **Frame extraction** (scene detection + uniform sampling, hardware-accelerated, downsampled): ```bash # Hardware-accelerated (CUDA/NVDEC preferred) ffmpeg -hide_banner -y -hwaccel cuda -i CLIP_PATH \ -vf "scale=1280:-2,fps=1/2,select='gt(scene,THRESHOLD)+gte(n,0)',showinfo" \ -vsync vfr -q:v 3 FRAMES_DIR/frame_%04d.jpg # CPU fallback ffmpeg -hide_banner -y -i CLIP_PATH \ -vf "scale=1280:-2,fps=1/2,select='gt(scene,THRESHOLD)',showinfo" \ -vsync vfr -q:v 3 FRAMES_DIR/frame_%04d.jpg ``` **Parameters the agent chooses:** | Parameter | Default | Guidance | | --------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------- | | `scale` width | `1280` | Raise to `1920` for short clips (<30s) where detail matters; lower to `854` for long clips (>120s) to save tokens | | `THRESHOLD` | `0.3` | Lower to `0.1`–`0.2` for gameplay footage (subtle scene changes); raise to `0.4` for dynamic content | | `fps` sample rate | `1/2` (1 frame per 2s) | Raise to `1` (1fps) for short clips; lower to `1/5` for long clips | | `-q:v` (JPEG quality) | `3` | Range 2–31, lower = higher quality. 3 is high quality; 5 is a good balance | **All clips and frames are saved next to the video file** (e.g., `_clips/` and `_frames/`). The user manages cleanup. **Record every extracted clip and frame in the index** before proceeding. **Completion criteria:** clips and frames exist on disk and are recorded in the index. ### Step 7 — Vision analysis Use whatever vision capability the agent's runtime provides to analyze extracted frames. The agent chooses: - **Which model** to use (depends on the agent's runtime and available vision capabilities). - **Batch size** — depends on the model's context window. See `references/frame-extraction-guide.md` for token cost estimates at different resolutions. - **How many frames** per analysis pass — use the binary-search approach from Step 6. **Completion criteria:** the agent has enough visual understanding to answer the user's question or produce the requested edit plan. ### Step 8 — Produce edit plan Output a Markdown table with overall recommendations, then let the user iterate. **Respond in the same language the user is using.** ```markdown ## Edit Plan ### Overall Recommendation <1-3 sentences of high-level recommendation addressing the user's goal> ### Detailed Plan | Timecode | Duration | Segment Description | Action | Transition | Notes | | ----------------- | -------- | ------------------------------- | ------------- | ---------- | ------------------ | | 00:01:23–00:01:45 | 22s | Character enters town, dialogue | Keep | Hard cut | Can add subtitles | | 00:01:45–00:02:10 | 25s | Walking, no dialogue or event | Cut | — | — | | 00:02:10–00:02:35 | 25s | Combat starts, highlight moment | Keep, slow-mo | Fade in | Sync to BGM rhythm | | ... | ... | ... | ... | ... | ... | ``` **Completion criteria:** user receives a Markdown table addressing their stated goal, and is invited to ask follow-up questions. ### Step 9 — Iterate The user will likely ask follow-ups: "what about this section?", "add a transition here", "can you check what's happening at 5:30?". For each: 1. Check if the answer can be derived from existing transcript + frames (check index first). 2. If not, extract new clips/frames for the requested range (Step 6). 3. Analyze and update the plan. **Completion criteria:** user is satisfied with the plan or explicitly ends the session. ## Long-running operations Before any operation expected to take >1 minute (first-time `uv sync`, transcription of long videos, large frame extraction): 1. **Notify the user** that a long operation is about to start and give a rough time estimate. 2. Run the operation in a way that does not block the conversation. 3. Do not poll progress at high frequency — wait for completion notification rather than checking every few seconds. ## Self-contained setup The skill bundles its own transcription project under `scripts/transcription/`. On first use: ```bash cd SKILL_DIR/scripts/transcription && uv sync ``` This creates an isolated venv with `funasr`, `torch`, `torchaudio`. If the user has previously installed these packages via uv in other projects, uv's cache will reuse them — first-time cost is only the link step. ## Common pitfalls 1. **Installing dependencies without consent.** Always ask the user first. If they decline, stop — the skill cannot proceed without ffmpeg, uv, and python3. 2. **Re-processing already-transcribed videos.** Always check the index file first. If a transcription record exists and the video hasn't been modified, reuse it. 3. **Extracting frames from the entire video.** This is extremely slow for long videos. Always clip the target range first with `-c copy`, then extract frames from the clip. 4. **Using full-resolution frames.** A 2560×1600 PNG can cost 2000+ vision tokens. Downsample to 1280 width with `scale=1280:-2` and use JPEG (`-q:v 3`). 5. **Forgetting to record artifacts in the index.** Every clip and frame batch must be recorded. Otherwise subsequent turns will re-extract the same content. 6. **Assuming the transcript is sufficient.** Gameplay videos often have long stretches of action with no dialogue. The agent should proactively check whether visual analysis is needed for segments the user asks about. 7. **Mixing funasr-script flags.** The bundled `funasr-nano`/`funasr-fast` use `--track STREAM_INDEX` and `--output-dir DIR`. Do not use the legacy `~/.local/bin/funasr-transcribe` wrapper's flags. 8. **Hardcoding tool names.** This skill is agent-agnostic. Do not assume specific MCP tools or APIs exist. Use whatever the runtime provides for vision analysis, background execution, and user notification. 9. **Hardcoding output language.** The edit plan table and all user-facing text must follow the user's language, not a fixed language. The English table in Step 8 is a template — translate columns and content to match the user's language at runtime. ## Verification checklist - [ ] All dependencies (ffmpeg, uv, python3) confirmed present or user declined. - [ ] Video path and audio track index confirmed. - [ ] User's editing goal understood. - [ ] Transcription completed or reused from cache. - [ ] Index file initialized at `/.vedit.db`. - [ ] All extracted clips and frames recorded in the index. - [ ] Vision analysis performed where needed. - [ ] Edit plan produced as Markdown table with overall recommendation. - [ ] User invited to iterate.