nite/video-edit-planner-skill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: video-edit-planner skill — transcribe, extract frames, plan edits

Browse Source 
A conversational video-editing planning assistant. Provides tools for:
- Audio transcription via bundled funasr-script (Fun-ASR-Nano / SenseVoice)
- On-demand clip extraction and frame sampling (ffmpeg, hardware-accelerated)
- SQLite index to track all artifacts and avoid duplicate processing
- Vision analysis guidance with binary-search-style frame sampling
- Iterative Markdown-table edit plan output

Agent-agnostic: no platform-specific tool names, works with any agent runtime.
Dependencies checked at guidance level with OS package manager install hints.
This commit is contained in:
nite
2026-07-14 11:07:57 +08:00
commit a565382a52
13 changed files with 3371 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SKILL.md
+290
View File
@@ -0,0 +1,290 @@
---
name: video-edit-planner
description: "Use when planning video edits: transcribe audio, extract key frames, analyze visuals, and produce cut/transition/assembly plans via iterative dialogue."
license: MIT
tags:
[
video-editing,
transcription,
funasr,
frame-extraction,
vision,
planning,
ffmpeg,
]
---
# Video Edit Planner
## Overview
A conversational video-editing planning assistant. The user provides a video file and an audio track index; the skill transcribes the audio, extracts key frames on demand, analyzes them visually, and produces an iterative Markdown-table edit plan (timecodes, segment descriptions, transitions, notes).
The skill is **agent-driven, not scripted**: it provides tools (transcription, clipping, frame extraction, index management) and guidance for the agent to autonomously decide when and what to extract based on user needs.
## When to Use
- User provides a video path and wants help planning cuts, transitions, or assembly.
- User asks "help me edit this video" or "what parts should I cut from this recording."
- User needs to understand video content beyond what the transcript alone reveals.
- User wants an iterative dialogue to refine an edit plan over multiple turns.
**Don't use for:** actual video editing/rendering (this skill plans, it does not produce video files), or pure audio-only transcription.
## Workflow
### Step 1 — Check dependencies
Before any processing, verify the environment. Missing dependencies must be **installed only with explicit user consent**.
| Dependency | Check command | Install (prefer OS package manager; fallback to pip) |
| -------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `ffmpeg` + `ffprobe` | `which ffmpeg ffprobe` | Linux: `sudo pacman -S ffmpeg` / `sudo apt install ffmpeg`; Windows: `winget install Gyan.FFmpeg`; macOS: `brew install ffmpeg` |
| `uv` | `which uv` | Linux: `sudo pacman -S uv` / `sudo apt install uv` (if available); macOS: `brew install uv`; Windows: `winget install astral-sh.uv`; fallback: `pip install uv` |
| `python3` | `which python3` | Linux: `sudo pacman -S python`; Windows: `winget install Python.Python.3`; macOS: `brew install python` |
**Completion criteria:** all three dependencies confirmed present, or user has explicitly declined installation (in which case stop — the skill cannot proceed).
### Step 2 — Gather inputs
Ask the user for:
1. **Video file path** (absolute path; translate Windows paths like `C:\Users\...\x.mkv` to `/mnt/c/Users/.../x.mkv` on WSL).
2. **Audio track index** — the ffmpeg stream index, not ordinal. If unknown, list tracks:
```bash
ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout:stream_tags=language,title -of json VIDEO_PATH
```
**Completion criteria:** both video path and track index confirmed.
### Step 3 — Ask editing requirements
Before transcribing, ask the user what they want to do with the video. Examples:
- "Trim out dead air and boring parts"
- "Make a highlight reel of funny moments"
- "Plan transitions between scenes"
- "Find the best 30-second clip for a short"
User requirements are often vague in the first pass — that's expected. The plan is refined iteratively.
**Completion criteria:** user has described at least a rough editing goal.
### Step 4 — Transcribe audio
Use the bundled `funasr-script` in `scripts/transcription/`.
**First run** requires `uv sync` (installs funasr + torch, may take several minutes). Subsequent runs reuse the cached venv. Notify the user before first-time setup.
```bash
# Default: Fun-ASR-Nano (high quality, Chinese-optimized)
uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --track TRACK_INDEX --output-dir VIDEO_DIR
# Fast preview: SenseVoice
uv run --directory SKILL_DIR/scripts/transcription funasr-fast VIDEO_PATH --track TRACK_INDEX --output-dir VIDEO_DIR
# List tracks
uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --list-tracks
```
**Skip if cached:** check the index file (Step 5) for an existing transcription entry. If one exists and the video file hasn't changed, reuse it.
**Completion criteria:** a transcription JSON exists and is recorded in the index. JSON contains `segments` with timestamps and text.
### Step 5 — Manage index file
An SQLite database tracks all processing artifacts to avoid duplicate work. It lives **next to the video file**.
```
<video_dir>/<video_stem>.vedit.db
```
Schema (managed by `scripts/index/manage_index.py`):
```sql
CREATE TABLE IF NOT EXISTS transcription (
id INTEGER PRIMARY KEY AUTOINCREMENT,
json_path TEXT NOT NULL,
track_index INTEGER NOT NULL,
created_at REAL NOT NULL, -- Unix timestamp
duration_s REAL
);
CREATE TABLE IF NOT EXISTS clips (
id INTEGER PRIMARY KEY AUTOINCREMENT,
start_time REAL NOT NULL, -- seconds from video start
end_time REAL NOT NULL,
path TEXT NOT NULL,
created_at REAL NOT NULL, -- Unix timestamp
reason TEXT -- why this clip was extracted
);
CREATE TABLE IF NOT EXISTS frames (
id INTEGER PRIMARY KEY AUTOINCREMENT,
clip_id INTEGER NOT NULL REFERENCES clips(id),
timestamp REAL NOT NULL, -- seconds from video start
path TEXT NOT NULL,
scene_score REAL, -- ffmpeg scene score if available
method TEXT NOT NULL, -- 'scene' | 'sample' | 'both'
created_at REAL NOT NULL -- Unix timestamp
);
```
Use `scripts/index/manage_index.py` for all CRUD operations:
```bash
# Initialize database (idempotent)
uv run --directory SKILL_DIR python scripts/index/manage_index.py init --video VIDEO_PATH
# Add transcription record
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-transcription --json-path PATH --track-index N --duration S
# Query existing transcription
uv run --directory SKILL_DIR python scripts/index/manage_index.py get-transcription
# Add a clip record
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-clip --start S --end E --path PATH --reason "user asked about this segment"
# Add frame records (batch from a directory)
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-frames --clip-id N --frames-dir DIR --method scene
# List all clips and frames
uv run --directory SKILL_DIR python scripts/index/manage_index.py list
# Check if a time range has already been extracted
uv run --directory SKILL_DIR python scripts/index/manage_index.py check-range --start S --end E
```
**Completion criteria:** index file exists and all produced artifacts are recorded in it.
### Step 6 — Extract clips and frames (on demand)
**When to extract:** the agent autonomously decides based on user needs. If the transcript alone is insufficient to answer (e.g., user asks about visual content, or a transcript segment is sparse/empty for a time range the user cares about), extract frames for that range.
**Binary-search-style frame analysis:**
1. Extract a clip for the target time range.
2. Extract scene-change frames + uniform samples from the clip.
3. Send a few representative frames to vision analysis.
4. If a sub-range needs deeper understanding, extract more frames from that sub-range.
5. Once the relevant range is identified, send all frames in that range (in batches if needed) to vision.
**Clip extraction** (keyframe-accurate, `-c copy`):
```bash
ffmpeg -hide_banner -y -ss START -to END -i VIDEO_PATH -c copy -map 0:v:0 CLIP_PATH
```
**Frame extraction** (scene detection + uniform sampling, hardware-accelerated, downsampled):
```bash
# Hardware-accelerated (CUDA/NVDEC preferred)
ffmpeg -hide_banner -y -hwaccel cuda -i CLIP_PATH \
-vf "scale=1280:-2,fps=1/2,select='gt(scene,THRESHOLD)+gte(n,0)',showinfo" \
-vsync vfr -q:v 3 FRAMES_DIR/frame_%04d.jpg
# CPU fallback
ffmpeg -hide_banner -y -i CLIP_PATH \
-vf "scale=1280:-2,fps=1/2,select='gt(scene,THRESHOLD)',showinfo" \
-vsync vfr -q:v 3 FRAMES_DIR/frame_%04d.jpg
```
**Parameters the agent chooses:**
| Parameter | Default | Guidance |
| --------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `scale` width | `1280` | Raise to `1920` for short clips (<30s) where detail matters; lower to `854` for long clips (>120s) to save tokens |
| `THRESHOLD` | `0.3` | Lower to `0.1`–`0.2` for gameplay footage (subtle scene changes); raise to `0.4` for dynamic content |
| `fps` sample rate | `1/2` (1 frame per 2s) | Raise to `1` (1fps) for short clips; lower to `1/5` for long clips |
| `-q:v` (JPEG quality) | `3` | Range 2–31, lower = higher quality. 3 is high quality; 5 is a good balance |
**All clips and frames are saved next to the video file** (e.g., `<video_stem>_clips/` and `<video_stem>_frames/`). The user manages cleanup.
**Record every extracted clip and frame in the index** before proceeding.
**Completion criteria:** clips and frames exist on disk and are recorded in the index.
### Step 7 — Vision analysis
Use whatever vision capability the agent's runtime provides to analyze extracted frames. The agent chooses:
- **Which model** to use (depends on the agent's runtime and available vision capabilities).
- **Batch size** — depends on the model's context window. See `references/frame-extraction-guide.md` for token cost estimates at different resolutions.
- **How many frames** per analysis pass — use the binary-search approach from Step 6.
**Completion criteria:** the agent has enough visual understanding to answer the user's question or produce the requested edit plan.
### Step 8 — Produce edit plan
Output a Markdown table with overall recommendations, then let the user iterate. **Respond in the same language the user is using.**
```markdown
## Edit Plan
### Overall Recommendation
<1-3 sentences of high-level recommendation addressing the user's goal>
### Detailed Plan
| Timecode | Duration | Segment Description | Action | Transition | Notes |
| ----------------- | -------- | ------------------------------- | ------------- | ---------- | ------------------ |
| 00:01:23–00:01:45 | 22s | Character enters town, dialogue | Keep | Hard cut | Can add subtitles |
| 00:01:45–00:02:10 | 25s | Walking, no dialogue or event | Cut | — | — |
| 00:02:10–00:02:35 | 25s | Combat starts, highlight moment | Keep, slow-mo | Fade in | Sync to BGM rhythm |
| ... | ... | ... | ... | ... | ... |
```
**Completion criteria:** user receives a Markdown table addressing their stated goal, and is invited to ask follow-up questions.
### Step 9 — Iterate
The user will likely ask follow-ups: "what about this section?", "add a transition here", "can you check what's happening at 5:30?". For each:
1. Check if the answer can be derived from existing transcript + frames (check index first).
2. If not, extract new clips/frames for the requested range (Step 6).
3. Analyze and update the plan.
**Completion criteria:** user is satisfied with the plan or explicitly ends the session.
## Long-running operations
Before any operation expected to take >1 minute (first-time `uv sync`, transcription of long videos, large frame extraction):
1. **Notify the user** that a long operation is about to start and give a rough time estimate.
2. Run the operation in a way that does not block the conversation.
3. Do not poll progress at high frequency — wait for completion notification rather than checking every few seconds.
## Self-contained setup
The skill bundles its own transcription project under `scripts/transcription/`. On first use:
```bash
cd SKILL_DIR/scripts/transcription && uv sync
```
This creates an isolated venv with `funasr`, `torch`, `torchaudio`. If the user has previously installed these packages via uv in other projects, uv's cache will reuse them — first-time cost is only the link step.
## Common pitfalls
1. **Installing dependencies without consent.** Always ask the user first. If they decline, stop — the skill cannot proceed without ffmpeg, uv, and python3.
2. **Re-processing already-transcribed videos.** Always check the index file first. If a transcription record exists and the video hasn't been modified, reuse it.
3. **Extracting frames from the entire video.** This is extremely slow for long videos. Always clip the target range first with `-c copy`, then extract frames from the clip.
4. **Using full-resolution frames.** A 2560×1600 PNG can cost 2000+ vision tokens. Downsample to 1280 width with `scale=1280:-2` and use JPEG (`-q:v 3`).
5. **Forgetting to record artifacts in the index.** Every clip and frame batch must be recorded. Otherwise subsequent turns will re-extract the same content.
6. **Assuming the transcript is sufficient.** Gameplay videos often have long stretches of action with no dialogue. The agent should proactively check whether visual analysis is needed for segments the user asks about.
7. **Mixing funasr-script flags.** The bundled `funasr-nano`/`funasr-fast` use `--track STREAM_INDEX` and `--output-dir DIR`. Do not use the legacy `~/.local/bin/funasr-transcribe` wrapper's flags.
8. **Hardcoding tool names.** This skill is agent-agnostic. Do not assume specific MCP tools or APIs exist. Use whatever the runtime provides for vision analysis, background execution, and user notification.
9. **Hardcoding output language.** The edit plan table and all user-facing text must follow the user's language, not a fixed language. The English table in Step 8 is a template — translate columns and content to match the user's language at runtime.
## Verification checklist
- [ ] All dependencies (ffmpeg, uv, python3) confirmed present or user declined.
- [ ] Video path and audio track index confirmed.
- [ ] User's editing goal understood.
- [ ] Transcription completed or reused from cache.
- [ ] Index file initialized at `<video_dir>/<video_stem>.vedit.db`.
- [ ] All extracted clips and frames recorded in the index.
- [ ] Vision analysis performed where needed.
- [ ] Edit plan produced as Markdown table with overall recommendation.
- [ ] User invited to iterate.