nite/video-edit-planner-skill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
video-edit-planner-skill/SKILL.md
T

471 lines
36 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
name: video-edit-planner
description: "Use when planning video edits: transcribe audio, extract key frames, analyze visuals, and produce cut/transition/assembly plans via iterative dialogue."
license: MIT
tags:
[
video-editing,
transcription,
funasr,
frame-extraction,
vision,
planning,
ffmpeg,
]
---
# Video Edit Planner
## Overview
A conversational video-editing planning assistant. The user provides a video file and an audio track index; the skill transcribes the audio, extracts key frames on demand, analyzes them visually, and produces an iterative Markdown-table edit plan (timecodes, segment descriptions, transitions, notes).
The skill is **agent-driven, not scripted**: it provides tools (transcription, clipping, frame extraction, index management) and guidance for the agent to autonomously decide when and what to extract based on user needs.
## When to Use
- User provides a video path and wants help planning cuts, transitions, or assembly.
- User asks "help me edit this video" or "what parts should I cut from this recording."
- User needs to understand video content beyond what the transcript alone reveals.
- User wants an iterative dialogue to refine an edit plan over multiple turns.
- User only needs audio transcription from a video file.
- User needs to extract and analyze specific video segments visually.
- User needs help finding stock materials (stickers, GIFs, meme clips) for editing.
Any subset of these capabilities is a valid use case — not every session requires the full workflow.
**Don't use for:** actual video editing/rendering (this skill plans and transcribes, it does not produce rendered video files).
## Workflow
**Every step is optional.** The steps below describe the full end-to-end workflow, but the agent should only execute the steps relevant to the user's request. Users may need any subset of these capabilities — transcription only, frame extraction only, material search only, or a full planning session. Use the user's request to determine which steps apply; do not force the full pipeline when it is not needed.
### Step 1 — Check dependencies
Before any processing, verify the environment. Missing dependencies must be **installed only with explicit user consent**.
| Dependency | Check command | Install (prefer OS package manager; fallback to pip) |
| -------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `ffmpeg` + `ffprobe` | `which ffmpeg ffprobe` | Linux: `sudo pacman -S ffmpeg` / `sudo apt install ffmpeg`; Windows: `winget install Gyan.FFmpeg`; macOS: `brew install ffmpeg` |
| `uv` | `which uv` | Linux: `sudo pacman -S uv` / `sudo apt install uv` (if available); macOS: `brew install uv`; Windows: `winget install astral-sh.uv`; fallback: `pip install uv` |
| `python3` / Python 3.12 for transcription | `python --version` / `python3 --version` | Use Python 3.12 for the bundled FunASR project, especially on Windows. Linux: `sudo pacman -S python` / `sudo apt install python3`; Windows: `winget install Python.Python.3.12`; macOS: `brew install python@3.12` |
**Completion criteria:** all three dependencies confirmed present, or user has explicitly declined installation (in which case stop — the skill cannot proceed).
### Step 2 — Gather inputs
Ask the user for:
1. **Video file path** (absolute path; translate Windows paths like `C:\Users\...\x.mkv` to `/mnt/c/Users/.../x.mkv` on WSL).
2. **Audio track index(es)** — the ffmpeg stream index, not ordinal. The user may provide **multiple tracks** (e.g., track 1 = own mic, track 2 = Discord audio). Each track is transcribed separately and saved as a separate JSON file. If unknown, list tracks:
```bash
ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout:stream_tags=language,title -of json VIDEO_PATH
```
**Completion criteria:** video path and at least one track index confirmed.
### Step 3 — Ask editing requirements
Before transcribing, ask the user what they want to do with the video. User requirements are often vague in the first pass — that's expected. The plan is refined iteratively.
For full video-edit planning requests, ask early whether the user wants **one focused recommended cut** or **multiple alternative editing strategies**. If they want alternatives, analyze a wider set of candidate segments and present modular options the user can freely combine into their own final cut. Do not force the plan into a rigid one-path timeline where every section has only one fixed choice.
If the agent's runtime supports structured questioning (e.g., a `clarify` tool or a `grill-me` skill), use it to gather requirements systematically. Ask questions one at a time with multiple-choice options — this is much more effective than a free-text wall of questions.
**The requirements questionnaire below covers the key axes for gameplay/creator video editing.** Not every question applies to every session — pick the relevant ones based on the user's initial request. See `references/editing-requirements-questionnaire.md` for the full questionnaire with choices.
| Axis | Why it matters |
|---|---|
| Target platform (B站/抖音/YouTube/Shorts) | Determines aspect ratio, pacing, subtitle density, meme/热梗 density |
| Narrative style (搞笑集锦/故事线+搞笑/高光混剪) | Determines structure: pure clip reel vs. narrative arc with 起承转合 |
| Planning mode (single recommended cut / multiple alternatives) | Determines whether to produce one focused timeline or modular strategy options the user can mix and match |
| Role distribution (主视角/平权联机/主持人) | Determines whose voice/reactions to prioritize in cuts |
| Effect density (少量/中等/高密度) | Determines how many 表情包/贴纸/特效 to search for and apply |
| Editor tool (Premiere Pro/剪映/DaVinci/必剪) | Determines how specific transition/effect instructions are phrased |
| First edit or existing draft | Determines whether to plan from scratch or optimize existing cuts |
| BGM style preference | Determines BGM search/recommendation direction |
| Number of participants | Determines how to handle multi-person interaction and voice allocation |
| Audio handling (保留原声+字幕/重新配音/混合) | Determines subtitle work, BGM vs voice balance, and narration needs |
| Privacy (露脸+声音/不露脸声音OK/需打码变声) | Determines face-cam handling and any masking needs |
| Output format (时间戳笔记/脚本表格/分镜方案) | Determines deliverable detail level |
| Target duration | Determines how aggressively to cut and pace the content |
**Completion criteria:** user has described at least a rough editing goal, and the agent has probed the relevant axes above.
**Long requirement summaries:** if the synthesized requirements, transcript-derived notes, or edit-planning response will be long, do not dump the whole thing into chat by default. Ask the user whether to save it as a file instead. Markdown (`.md`) is the default format for requirement summaries and edit plans; if the user requests another format (JSON, CSV, DOCX, etc.), use the requested format when practical.
### Step 4 — Transcribe audio
Use the bundled `funasr-script` in `scripts/transcription/`. **Prefer Fun-ASR-Nano** as the default transcription model — it has higher accuracy and sensitivity on Chinese audio. Use SenseVoice only for fast previews.
**First run** requires `uv sync` (installs funasr + torch, may take several minutes). Subsequent runs reuse the cached venv.
**All output files (transcription JSON, clips, frames) go in the same directory as the video file.** Never use `~/whisperx_output/` or any other separate output directory — the `--output-dir` flag should point to the video's own directory. This keeps all artifacts co-located and portable.
**Choose the execution environment based on where the video file lives and where the command will run.** Keep heavy media I/O local to the filesystem that stores the video:
- If the video is on the Windows host filesystem (for example a WSL path under `/mnt/c/...`, `/mnt/d/...`, or a native Windows path like `C:\\\\...`, `D:\\\\...`), prefer running transcription on the Windows host. Give **PowerShell commands** with Windows paths. Do not hand the user `wsl ...` commands for the long transcription in this case; using WSL to read `/mnt/c` or `/mnt/d` defeats the storage-locality rule.
- If the video is inside the WSL/Linux filesystem (for example `/home/...`, `/var/...`, or another Linux mount), prefer running transcription in WSL/Linux. Give **Linux shell commands** with Linux paths.
- If the location is ambiguous, ask the user where the video is stored and where they want the workload to run.
**Agent should do the lightweight setup work.** Installing/checking `uv`, copying `scripts/transcription/` to a Windows-local directory, and running `uv sync` are setup tasks. If the agent can execute them through its available command interface, it should do them itself after getting any needed consent for dependency installation. Only the actual long-running transcription command should be handed to the user to execute manually.
**If the agent runs in WSL and Windows execution is chosen**, the agent can invoke the Windows host via `powershell.exe`. It should copy or clone the transcription project to a Windows-local path (for example `C:\\Users\\<user>\\AppData\\Local\\Temp\\vedit-transcription`) and run setup there. Do not ask the user to manually copy files or run `uv sync` unless the agent cannot access the host command line or lacks permission.
**Windows `uv` constraint:** Windows `uv` should not operate on WSL UNC paths like `\\\\wsl.localhost\\...` or `\\\\wsl$\\...`; it can fail while managing `.venv` directories. Therefore, when the chosen execution environment is Windows, the transcription project itself must also be available under a Windows-local path. Follow `references/wsl-windows-uv-transcription.md`.
**Transcription can be long-running.** After setup is complete, generate the final transcription command and give it to the user to execute manually. Use PowerShell syntax for Windows execution and Linux shell syntax for WSL/Linux execution.
**Critical: when you tell the user you will provide a command, include it immediately in the same response.** Do not say "I'll give you the transcription command" and then move on to other topics without actually printing the command. The user should never have to ask "you said you'd give me the command but didn't."
Command templates (adjust paths for the user's environment):
Always include an explicit transcription language in commands that run ASR. Use a neutral placeholder such as `AUDIO_LANGUAGE_CODE` in templates, then replace it with the language actually spoken in the audio (`zh`, `en`, `ja`, `ko`, `yue`, etc.). Do **not** infer the audio language solely from the user's chat language. If uncertain, ask; if the user wants automatic detection, explicitly pass `--language auto` rather than omitting the flag.
```bash
# Recommended default: Fun-ASR-Nano (high quality, Chinese-optimized)
uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --track TRACK_INDEX --language AUDIO_LANGUAGE_CODE --output-dir VIDEO_DIR
# Fast preview only: SenseVoice
uv run --directory SKILL_DIR/scripts/transcription funasr-fast VIDEO_PATH --track TRACK_INDEX --language AUDIO_LANGUAGE_CODE --output-dir VIDEO_DIR
# List tracks (no ASR is run, so no language flag is needed)
uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --list-tracks
```
Where `VIDEO_DIR` is the directory containing the video file, `SKILL_DIR` is the skill installation directory, and `AUDIO_LANGUAGE_CODE` is the explicit transcription language (for example `zh` for Mandarin Chinese).
**Common OBS multi-track audio layouts.** When the user says "multi-track" or mentions OBS recording, these are the typical stream layouts:
| Track | Typical content |
|---|---|
| Stream 1 (index 1) | Game audio (desktop/system sound) |
| Stream 2 (index 2) | Microphone (commentary/narration) |
| Stream 3 (index 3) | Discord/voice chat (other participants) |
Always verify with `ffprobe` before transcribing — the user may have a different layout. When the user says "transcribe the mic and Discord tracks", that typically means tracks 2 and 3.
**Multi-track transcription:** when the user provides multiple track indexes (e.g., track 1 = mic, track 2 = Discord), run the transcription command **once per track**. Each run produces a separate JSON file named `<video_stem>_track<INDEX>_<model>.json`. Record each transcription in the index (Step 5) with its corresponding track index. When answering user questions, the agent should cross-reference all available transcripts to understand the full conversation context.
**Note on Discord audio quality:** Discord tracks may contain noise, crosstalk, or non-participant chatter. This is normal — the ASR transcript will be noisier. When cross-referencing transcripts, treat Discord track content as supplementary context, not as the primary narrative source.
**ASR transcript reliability:** FunASR transcripts are useful for locating material, understanding the rough plot, and finding candidate time ranges, but they are not an authoritative text source for final deliverables. Misrecognized words are common when the original track has unclear pronunciation, overlap, noise, or game audio. Similar-sounding words may be substituted (for example, Chinese `帐篷` may be transcribed as `账本`). Before using transcript text as final copy, subtitles, quotes, captions, or narration, re-listen to the original audio around that time range and correct the text manually. When uncertain, mark the wording as uncertain rather than treating the ASR output as exact.
**Skip if cached:** check the index file (Step 5) for existing transcription entries. If a record exists for the same track index and the video file hasn't changed, reuse it.
**Completion criteria:** a transcription JSON exists for every requested track, and each is recorded in the index. Each JSON contains `segments` with timestamps and text.
### Step 5 — Manage index file
A JSON file tracks all processing artifacts to avoid duplicate work. It lives **next to the video file** and is human-readable and editable.
```
<video_dir>/<video_stem>.vedit.json
```
**All file paths are stored as relative paths** (relative to the video directory). Moving the entire directory does not break any references. If files are split across directories, paths outside the video directory are stored as absolute paths — the user can edit the JSON directly with a text editor to fix them.
Structure (managed by `scripts/index/manage_index.py`):
```json
{
"transcriptions": [
{
"id": 1,
"json_path": "video_track1_fun-asr-nano.json",
"track_index": 1,
"created_at": 1783997794.78,
"duration_s": 120.5
}
],
"clips": [
{
"id": 1,
"start_time": 30.0,
"end_time": 60.0,
"path": "video_clips/clip_30-60.mkv",
"created_at": 1783997794.85,
"reason": "user asked about this segment"
}
],
"frames": [
{
"id": 1,
"clip_id": 1,
"timestamp": 32.0,
"path": "video_frames/frame_0001.jpg",
"scene_score": 0.45,
"method": "scene",
"created_at": 1783997795.0
}
]
}
```
Use `scripts/index/manage_index.py` for all CRUD operations:
```bash
# Initialize index file (idempotent)
uv run --directory SKILL_DIR python scripts/index/manage_index.py init --video VIDEO_PATH
# Add transcription record
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-transcription --json-path PATH --track-index N --duration S
# Query existing transcription (all tracks)
uv run --directory SKILL_DIR python scripts/index/manage_index.py get-transcription
# Query specific track
uv run --directory SKILL_DIR python scripts/index/manage_index.py get-transcription --track 1
# Add a clip record
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-clip --start S --end E --path PATH --reason "user asked about this segment"
# Add frame records (batch from a directory)
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-frames --clip-id N --frames-dir DIR --method scene
# List all clips and frames
uv run --directory SKILL_DIR python scripts/index/manage_index.py list
# Check if a time range has already been extracted
uv run --directory SKILL_DIR python scripts/index/manage_index.py check-range --start S --end E
# Remove records for files that no longer exist on disk
uv run --directory SKILL_DIR python scripts/index/manage_index.py clean
```
**Completion criteria:** index file exists and all produced artifacts are recorded in it.
### Step 6 — Extract clips and frames (on demand)
**When to extract:** the agent autonomously decides based on user needs. If the transcript alone is insufficient to answer (e.g., user asks about visual content, or a transcript segment is sparse/empty for a time range the user cares about), extract frames for that range.
**Binary-search-style frame analysis:**
1. Extract a clip for the target time range.
2. Extract scene-change frames + uniform samples from the clip.
3. Send a few representative frames to vision analysis.
4. If a sub-range needs deeper understanding, extract more frames from that sub-range.
5. Once the relevant range is identified, send all frames in that range (in batches if needed) to vision.
**Clip extraction** (keyframe-accurate, `-c copy`):
For frame extraction purposes, only the video stream is needed:
```bash
ffmpeg -hide_banner -y -ss START -to END -i VIDEO_PATH -c copy -map 0:v:0 CLIP_PATH
```
For clips that the user may want to import into a video editor or play back, **preserve all audio tracks** by mapping all streams instead of only the video:
```bash
ffmpeg -hide_banner -y -ss START -to END -i VIDEO_PATH -c copy -map 0 CLIP_PATH
```
`-map 0` copies all streams (video + all audio tracks + subtitles). The agent should ask the user whether they need a full-multitrack clip or a video-only clip for frame extraction. When in doubt, produce both: a video-only clip for fast frame extraction, and a full-multitrack clip if the user plans to use it in an editor.
**Note:** When using `-c copy` with `-ss` before `-i`, seek accuracy is keyframe-level. If the user needs frame-exact cuts, re-encode is required (significantly slower). Keyframe accuracy is sufficient for planning purposes.
**Frame extraction** (scene detection + uniform sampling, hardware-accelerated, downsampled):
```bash
# Hardware-accelerated (CUDA/NVDEC preferred)
ffmpeg -hide_banner -y -hwaccel cuda -i CLIP_PATH \
-vf "scale=1280:-2,fps=1/2,select='gt(scene,THRESHOLD)+gte(n,0)',showinfo" \
-vsync vfr -q:v 3 FRAMES_DIR/frame_%04d.jpg
# CPU fallback
ffmpeg -hide_banner -y -i CLIP_PATH \
-vf "scale=1280:-2,fps=1/2,select='gt(scene,THRESHOLD)',showinfo" \
-vsync vfr -q:v 3 FRAMES_DIR/frame_%04d.jpg
```
**Parameters the agent chooses:**
| Parameter | Default | Guidance |
| --------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `scale` width | `1280` | Raise to `1920` for short clips (<30s) where detail matters; lower to `854` for long clips (>120s) to save tokens |
| `THRESHOLD` | `0.3` | Lower to `0.1`–`0.2` for gameplay footage (subtle scene changes); raise to `0.4` for dynamic content |
| `fps` sample rate | `1/2` (1 frame per 2s) | Raise to `1` (1fps) for short clips; lower to `1/5` for long clips |
| `-q:v` (JPEG quality) | `3` | Range 2–31, lower = higher quality. 3 is high quality; 5 is a good balance |
**All clips and frames are saved next to the video file** (e.g., `<video_stem>_clips/` and `<video_stem>_frames/`). The user manages cleanup.
**Record every extracted clip and frame in the index** before proceeding.
**Completion criteria:** clips and frames exist on disk and are recorded in the index.
### Step 7 — Vision analysis
Use whatever vision capability the agent's runtime provides to analyze extracted frames. The agent chooses:
- **Which model** to use (depends on the agent's runtime and available vision capabilities).
- **Batch size** — depends on the model's context window. See `references/frame-extraction-guide.md` for token cost estimates at different resolutions.
- **Windows setup troubleshooting** — see `references/windows-funasr-uv-setup.md` for Windows-host PowerShell transcription commands, Python wheel issues such as `editdistance` source builds, and PyTorch CUDA-vs-CPU build checks.
- **How many frames** per analysis pass — use the binary-search approach from Step 6.
**Completion criteria:** the agent has enough visual understanding to answer the user's question or produce the requested edit plan.
### Step 8 — Produce edit plan
#### Transcript-first candidate mining for a bounded review window
When the user asks for **clip candidates from a specific time range** of an already-transcribed video (for example, `22:00–45:00`) rather than a full edit plan for the whole recording, do a lightweight **transcript-first mining pass** before any heavy extraction:
1. Load every relevant transcript track covering that window and slice to the requested range.
2. Compute simple density signals per minute or rolling window (segment count, character count, keyword hits, overlapping reactions across tracks).
3. Print and review the top windows plus manually chosen story pivots (setup / escalation / payoff), not just the numerically densest span.
4. Cross-read the main mic track with secondary chat/Discord tracks — use side tracks as corroboration and reaction context, not as the sole source of truth.
5. For the shortlisted windows, do a **spot visual check** with a few representative frames (for example 3–4 timestamps per candidate, optionally stitched into a contact sheet) instead of extracting long clips immediately.
6. Only extract full clips if the transcript and spot frames are still insufficient.
For multi-track recordings with crosstalk or duplicated speech, treat ASR as a **locator index**. Merge tracks by time, de-duplicate repeated lines mentally, and prefer moments where both dialogue and visuals support a self-contained mini-story. Secondary tracks may contain unrelated chatter; they are useful for reactions but should not dominate the narrative. When presenting the plan, call out which suggestions are transcript-grounded and which were visually spot-checked.
For fast-paced story + comedy gameplay videos, cull hard: keep combat, danger, reactions, relationship banter, and short setup/payoff beats; compress crafting, shopping, and inventory UI into 0.5–2 second bridge shots unless the item/choice itself is the punchline. A 7–8 minute plan should usually be a chaptered structure, not a flat list of every candidate.
This workflow is especially effective for requests like **“find fast-paced funny + story-driven candidate segments”**. It keeps token/runtime cost low while still verifying whether a transcript-highlighted moment actually has usable on-screen action.
Output a Markdown table with overall recommendations, then let the user iterate. **Respond in the same language the user is using.** If the user requested multiple alternative strategies, structure the plan as modular options rather than a single rigid timeline: offer alternative openings, narrative arcs, pacing choices, candidate segments, transitions, BGM/effect directions, and optional inserts the user can mix and match. In this mode, analyze more candidate segments than strictly needed for one final cut, and clearly label tradeoffs (e.g., funniest, clearest story, fastest pacing, most emotional, best visual action). If the plan is likely to be long (large tables, detailed scripts, full requirement summaries, or multi-section recommendations), ask whether the user wants it saved as a file instead of pasted into chat. Default to Markdown (`.md`) for saved plans unless the user requests another format.
```markdown
## Edit Plan
### Overall Recommendation
<1-3 sentences of high-level recommendation addressing the user's goal>
### Detailed Plan
| Timecode | Duration | Segment Description | Action | Transition | Notes |
| ----------------- | -------- | ------------------------------- | ------------- | ---------- | ------------------ |
| 00:01:23–00:01:45 | 22s | Character enters town, dialogue | Keep | Hard cut | Can add subtitles |
| 00:01:45–00:02:10 | 25s | Walking, no dialogue or event | Cut | — | — |
| 00:02:10–00:02:35 | 25s | Combat starts, highlight moment | Keep, slow-mo | Fade in | Sync to BGM rhythm |
| ... | ... | ... | ... | ... | ... |
### Alternative Strategy Options (if requested)
| Module | Option | Best for | Candidate segments | Tradeoffs |
| ------ | ------ | -------- | ------------------ | --------- |
| Opening | Cold open with funniest quote | Fast hook | 00:03:10–00:03:24 / 00:18:40–00:19:05 | Strong hook, weaker story setup |
| Story arc | Slow setup → escalation → payoff | Clear narrative | 00:01:20–00:02:30 / 00:12:00–00:14:10 / 00:22:30–00:23:20 | Better story, slower pacing |
| Insert | Short visual gag bridge | Rhythm reset | 00:08:10–00:08:16 | Optional; use only if pacing feels too dense |
```
**Completion criteria:** user receives a Markdown table addressing their stated goal, and is invited to ask follow-up questions.
### Step 9 — Iterate
The user will likely ask follow-ups: "what about this section?", "add a transition here", "can you check what's happening at 5:30?". For each:
1. Check if the answer can be derived from existing transcript + frames (check index first).
2. If not, extract new clips/frames for the requested range (Step 6).
3. Analyze and update the plan.
**Completion criteria:** user is satisfied with the plan or explicitly ends the session.
## Finding stock materials (optional, on demand)
When the user needs editing materials — stickers, GIFs, meme video clips, B-roll — the agent can help search or provide links for the user to search themselves. This section is **only triggered when the user asks for materials**; it is never a mandatory step.
### Material sources
| Source | URL | Content | Best for |
|---|---|---|---|
| 爱给网 | https://www.aigei.com/ | Chinese meme video clips (热梗, 4.2万+), free download | Chinese-language creators (B站/抖音) |
| B站素材酷 | https://cool.bilibili.com/ | Bilibili platform-native assets (video, audio, BGM, templates, stickers) | B站 creators; large Chinese meme collection |
| Tenor | https://tenor.com/ | GIFs + stickers, 30+ languages incl. Chinese, free API | Quick GIF/sticker search |
| GIPHY | https://giphy.com/ | Largest GIF library, 30+ languages, sticker channel | English/multilingual GIF search |
| Klipy | https://klipy.com/ | Localized GIF/sticker/clip API, supports Chinese | API-integrated search, localization |
### Priority by language
If the user communicates in Chinese, **recommend 爱给网 and B站素材酷 first** — they have the most Chinese-relevant meme and sticker content. For non-Chinese users, Tenor and GIPHY are good defaults.
### Search strategy
- **Expand search terms** based on semantic understanding of the user's request. If the user says "I want a clip of someone looking shocked", search terms might include: "惊讶 / 震惊 / 吃惊 / 哇 / wtf / shocked / surprised". Use the user's language as the base, expand with synonyms and internet slang.
- The agent can either **search directly** (using web tools or API calls) or **suggest expanded search terms** for the user to search manually.
- If the user only wants links, provide the source URLs above and let the user search themselves.
### B站素材酷 special handling
B站素材酷 (cool.bilibili.com) has no built-in search function. Two options:
1. **Recommend the userscript**: [bcut-resource-search](https://git.nite07.com/nite/bcut-resource-search) — a Tampermonkey script that adds search to 素材酷, supporting video, sticker/image, music, sound effect, and template categories.
2. **Search via API directly**: the agent can read the userscript's source code to discover the API endpoints and make requests directly (via HTTP tools or curl). Note: the site has CORS restrictions and UA-based access control, so direct API calls may require appropriate headers. If the agent's HTTP tools are blocked, fall back to suggesting the userscript to the user.
### Anti-bot fallback
Some material sites may block automated access (anti-bot, Cloudflare, CAPTCHA). If a web request or HTTP tool is blocked:
1. Try with different headers (User-Agent, Accept-Language) if the tool allows.
2. Fall back to a browser-based tool (e.g., Playwright, Puppeteer, or any browser automation the agent's runtime provides) to access the content.
3. If browser tools are also blocked, inform the user and provide the direct URL for manual search.
## Long-running operations
Before any operation expected to take >1 minute (transcription of long videos, large frame extraction):
1. Do lightweight setup directly when possible: dependency checks, copying scripts, preparing a Windows-local transcription directory, and running `uv sync`.
2. Ask for consent before installing missing system dependencies (`ffmpeg`, `uv`, Python), then install them if the runtime can do so.
3. Hand off only genuinely long-running media operations to the user. Generate the command and give it to the user to execute manually.
4. Match command syntax to the chosen execution environment: PowerShell commands for Windows execution; Linux shell commands for WSL/Linux execution.
5. After the user reports completion, the agent should verify the output exists before proceeding.
## Self-contained setup
The skill bundles its own transcription project under `scripts/transcription/`. On first use, the agent should prepare the environment before handing off the transcription command.
For WSL/Linux execution, run setup in the bundled transcription directory:
```bash
cd SKILL_DIR/scripts/transcription && uv sync
```
For Windows execution, ensure the transcription project is in a Windows-local directory before running setup. If the agent itself is running in WSL, it can use `powershell.exe` to run host commands and should do this setup itself when possible:
```powershell
# Example Windows-local setup directory
$TransDir = "$env:LOCALAPPDATA\\Temp\\vedit-transcription"
New-Item -ItemType Directory -Force -Path $TransDir | Out-Null
# Copy or clone scripts/transcription into $TransDir before this step.
Set-Location $TransDir
uv sync
```
This creates an isolated venv with `funasr`, `torch`, `torchaudio`. If the user has previously installed these packages via uv in other projects, uv's cache will reuse them — first-time cost is often only the link step. Do not ask the user to manually copy files or run `uv sync` unless the agent cannot access the host command line or lacks permission.
## Common pitfalls
1. **Installing dependencies without consent.** Always ask the user first. If they decline, stop — the skill cannot proceed without ffmpeg, uv, and python3.
2. **Re-processing already-transcribed videos.** Always check the index file first. If a transcription record exists and the video hasn't been modified, reuse it.
3. **Extracting frames from the entire video.** This is extremely slow for long videos. Always clip the target range first with `-c copy`, then extract frames from the clip.
4. **Using full-resolution frames.** A 2560×1600 PNG can cost 2000+ vision tokens. Downsample to 1280 width with `scale=1280:-2` and use JPEG (`-q:v 3`).
5. **Forgetting to record artifacts in the index.** Every clip and frame batch must be recorded. Otherwise subsequent turns will re-extract the same content.
6. **Assuming the transcript is sufficient.** Gameplay videos often have long stretches of action with no dialogue. The agent should proactively check whether visual analysis is needed for segments the user asks about.
7. **Treating ASR text as final copy.** FunASR transcripts can contain wrong characters/words caused by unclear speech, noise, overlap, or similar-sounding terms (e.g., `帐篷` → `账本`). Use transcripts for locating material and understanding context, not as the absolute wording for final captions, quotes, narration, or published copy. Re-listen and manually correct any text that will appear in the final product.
8. **Mixing funasr-script flags.** The bundled `funasr-nano`/`funasr-fast` use `--track STREAM_INDEX`, explicit `--language AUDIO_LANGUAGE_CODE`, and `--output-dir DIR`. Do not use the legacy `~/.local/bin/funasr-transcribe` wrapper's flags.
9. **Hardcoding tool names.** This skill is agent-agnostic. Do not assume specific MCP tools or APIs exist. Use whatever the runtime provides for vision analysis, background execution, and user notification.
10. **Hardcoding output language.** The edit plan table and all user-facing text must follow the user's language, not a fixed language. The English table in Step 8 is a template — translate columns and content to match the user's language at runtime.
11. **Dropping audio tracks during clip extraction.** The default clip command maps only `0:v:0` (video-only) for fast frame extraction. If the user plans to import the clip into a video editor, use `--all-streams` (or `-map 0`) to preserve all audio tracks. Always ask the user which they need.
12. **Using `~/whisperx_output/` or other separate output directories.** All artifacts (transcription JSON, clips, frames, index file) must go in the same directory as the video file. The `--output-dir` flag should always point to the video's own directory. This keeps everything co-located and portable.
13. **Choosing command syntax or execution environment solely from where the agent runs.** Choose based on where the video file lives and where the command will execute. Windows execution → PowerShell commands and Windows paths; WSL/Linux execution → Linux shell commands and Linux paths. If the agent runs in WSL but Windows execution is chosen, use `powershell.exe` to do lightweight setup (copy files, install/check dependencies, run `uv sync`) before handing only the long transcription command to the user. Never ask Windows `uv` to run from `\\\\wsl.localhost\\...` / `\\\\wsl$\\...`.
14. **Saying you'll provide a command but not including it.** If you mention giving the user a transcription command, include it immediately in that same response. Do not move on to other content without printing the actual command. The user should never have to ask for it.
15. **Sending a wall of free-text questions.** When gathering editing requirements, use structured one-at-a-time multiple-choice questions via the runtime's clarify/prompt tool. This is far more effective than dumping 10 questions as a paragraph.
16. **Making full edit plans too rigid.** For complete planning requests, ask whether the user wants multiple alternative strategies. If yes, provide modular options and extra candidate segments that the user can combine, rather than a single fixed linear plan where every section has exactly one prescribed choice.
## Verification checklist
Check only the items relevant to the steps actually executed. Not all items apply to every session — the user may only need a subset of the skill's capabilities.
- [ ] Dependencies (ffmpeg, uv, python3) confirmed present or user declined. *(Step 1 — always required)*
- [ ] Video path and audio track index(es) confirmed. *(Step 2 — required when video processing is needed)*
- [ ] User's editing goal understood. *(Step 3 — skip if not relevant to the user's request)*
- [ ] Transcription completed for all requested tracks, or reused from cache. *(Step 4)*
- [ ] Index file initialized at `<video_dir>/<video_stem>.vedit.json`. *(Step 5 — always required when any processing is done)*
- [ ] All extracted clips and frames recorded in the index. *(Step 6 — only if frames were extracted)*
- [ ] Vision analysis performed where needed. *(Step 7 — only if frames were extracted)*
- [ ] Edit plan produced as Markdown table with overall recommendation. *(Step 8 — skip if not relevant to the user's request)*
- [ ] User invited to iterate. *(Step 9)*
Reference in New Issue View Git Blame Copy Permalink