diff --git a/README.md b/README.md index 1ba3cd7..b83320a 100644 --- a/README.md +++ b/README.md @@ -69,7 +69,7 @@ For Hermes Agent specifically, clone into `~/.hermes/skills/media/`. For other a video-edit-planner/ ├── SKILL.md # Skill definition (workflow, guidance, pitfalls) ├── README.md # This file (English) -├── README.zh.md # 中文说明 +├── README.zh.md # Chinese README ├── scripts/ │ ├── transcription/ # Bundled funasr-script (self-contained uv project) │ │ ├── pyproject.toml diff --git a/SKILL.md b/SKILL.md index f05762e..b056f44 100644 --- a/SKILL.md +++ b/SKILL.md @@ -76,18 +76,18 @@ If the agent's runtime supports structured questioning (e.g., a `clarify` tool o | Axis | Why it matters | |---|---| -| Target platform (B站/抖音/YouTube/Shorts) | Determines aspect ratio, pacing, subtitle density, meme/热梗 density | -| Narrative style (搞笑集锦/故事线+搞笑/高光混剪) | Determines structure: pure clip reel vs. narrative arc with 起承转合 | +| Target platform (Bilibili / Douyin / YouTube / Shorts) | Determines aspect ratio, pacing, subtitle density, meme density | +| Narrative style (funny compilation / story + comedy / highlights montage) | Determines structure: pure clip reel vs. narrative arc | | Planning mode (single recommended cut / multiple alternatives) | Determines whether to produce one focused timeline or modular strategy options the user can mix and match | -| Role distribution (主视角/平权联机/主持人) | Determines whose voice/reactions to prioritize in cuts | -| Effect density (少量/中等/高密度) | Determines how many 表情包/贴纸/特效 to search for and apply | -| Editor tool (Premiere Pro/剪映/DaVinci/必剪) | Determines how specific transition/effect instructions are phrased | +| Role distribution (main POV / equal co-op / host / commentator) | Determines whose voice/reactions to prioritize in cuts | +| Effect density (minimal / medium / high) | Determines how many stickers, overlays, and effects to search for and apply | +| Editor tool (Premiere Pro / CapCut / DaVinci / other) | Determines how specific transition/effect instructions are phrased | | First edit or existing draft | Determines whether to plan from scratch or optimize existing cuts | | BGM style preference | Determines BGM search/recommendation direction | | Number of participants | Determines how to handle multi-person interaction and voice allocation | -| Audio handling (保留原声+字幕/重新配音/混合) | Determines subtitle work, BGM vs voice balance, and narration needs | -| Privacy (露脸+声音/不露脸声音OK/需打码变声) | Determines face-cam handling and any masking needs | -| Output format (时间戳笔记/脚本表格/分镜方案) | Determines deliverable detail level | +| Audio handling (keep original + subtitles / voice-over / mixed) | Determines subtitle work, BGM vs voice balance, and narration needs | +| Privacy (face + voice OK / no face / need masking) | Determines face-cam handling and any masking needs | +| Output format (timestamp notes / script table / storyboard plan) | Determines deliverable detail level | | Target duration | Determines how aggressively to cut and pace the content | **Completion criteria:** user has described at least a rough editing goal, and the agent has probed the relevant axes above. @@ -150,7 +150,7 @@ Always verify with `ffprobe` before transcribing — the user may have a differe **Note on Discord audio quality:** Discord tracks may contain noise, crosstalk, or non-participant chatter. This is normal — the ASR transcript will be noisier. When cross-referencing transcripts, treat Discord track content as supplementary context, not as the primary narrative source. -**ASR transcript reliability:** FunASR transcripts are useful for locating material, understanding the rough plot, and finding candidate time ranges, but they are not an authoritative text source for final deliverables. Misrecognized words are common when the original track has unclear pronunciation, overlap, noise, or game audio. Similar-sounding words may be substituted (for example, Chinese `帐篷` may be transcribed as `账本`). Before using transcript text as final copy, subtitles, quotes, captions, or narration, re-listen to the original audio around that time range and correct the text manually. When uncertain, mark the wording as uncertain rather than treating the ASR output as exact. +**ASR transcript reliability:** FunASR transcripts are useful for locating material, understanding the rough plot, and finding candidate time ranges, but they are not an authoritative text source for final deliverables. Misrecognized words are common when the original track has unclear pronunciation, overlap, noise, or background audio. Similar-sounding words may be substituted. Before using transcript text as final copy, subtitles, quotes, captions, or narration, re-listen to the original audio around that time range and correct the text manually. When uncertain, mark the wording as uncertain rather than treating the ASR output as exact. **ASR JSON format:** FunASR output JSON files contain a `segments` array where each segment has `start` and `end` timestamps in **milliseconds** (not seconds), plus `text` and `speaker` fields. When filtering segments by time range, convert `mm:ss` to milliseconds first (e.g., `49:30` = `2970000` ms). The top-level `audio_duration_s` field is in seconds. Always verify units before using timestamps for frame extraction. @@ -259,12 +259,12 @@ When verifying specific transcript claims (e.g., "is there really a boss fight a ffmpeg -y -ss TIMESTAMP -i VIDEO_PATH -frames:v 1 -q:v 3 OUTPUT.jpg ``` -This is much faster than clip+scene-detection for verifying a handful of specific points. Send each frame to vision analysis with a targeted question about what the transcript claims is happening at that moment (e.g., "The transcript says '他二阶段了' — can you see a boss in phase 2?"). +This is much faster than clip+scene-detection for verifying a handful of specific points. Send each frame to vision analysis with a targeted question about what the transcript claims is happening at that moment (e.g., "The transcript says the boss entered phase 2 — can you see a boss with a phase indicator?"). Key lessons from spot-verification: - **ASR timestamps can be off by several seconds** from the visual event they describe. If a frame at the ASR timestamp doesn't show the claimed content, try ±5–10 seconds before concluding the claim is wrong. -- **ASR proper nouns are unreliable.** Boss names, item names, and location names may be homophone errors (e.g., "沙漠巨凤" Desert Giant Phoenix was transcribed as "沙漠巨风" Desert Giant Wind). Treat proper nouns in transcripts as phonetic hints, not authoritative labels. -- **Visual confirmation can upgrade or downgrade segment priority.** A "待核验 S" segment with confirmed boss visuals becomes confirmed S; a segment where the transcript claims action but frames show only menus/UI should be downgraded. +- **ASR proper nouns are unreliable.** Boss names, item names, and location names may be homophone errors. Treat proper nouns in transcripts as phonetic hints, not authoritative labels. +- **Visual confirmation can upgrade or downgrade segment priority.** A segment with confirmed visuals becomes higher priority; a segment where the transcript claims action but frames show only menus/UI should be downgraded. **Clip extraction** (keyframe-accurate, `-c copy`): @@ -365,8 +365,8 @@ Each segment below is self-contained: timecode, suggested duration, content/reas | Suggested duration | ~22s | | Content / reason | Character enters town, NPC dialogue reveals quest hook | | Edit technique | Hard cut in, match-on-action to next segment | -| Visual materials | Sticker: "震惊" reaction face; search: 震惊 / surprised / shocked meme | -| Sound effects | Whoosh on cut; search: whoosh / 转场音效 | +| Visual materials | Sticker: shocked reaction face; search: shocked / surprised meme | +| Sound effects | Whoosh on cut; search: whoosh / transition sound effect | | BGM | Light exploration theme, fade in at 00:01:25 | #### Segment 2 — @@ -389,8 +389,8 @@ Each segment below is self-contained: timecode, suggested duration, content/reas | Suggested duration | ~25s | | Content / reason | Combat starts, highlight moment with reaction | | Edit technique | Keep, slow-motion on hit, fade in from black | -| Visual materials | Zoom-in effect on impact; search: 放大镜 / zoom meme | -| Sound effects | Impact hit + record scratch on slow-mo; search: impact / punch / 打击音效 / record scratch | +| Visual materials | Zoom-in effect on impact; search: zoom-in / magnify meme | +| Sound effects | Impact hit + record scratch on slow-mo; search: impact / punch / hit sound effect / record scratch | | BGM | Switch to combat track, sync hit to beat drop | ### Alternative Strategy Options (if requested) @@ -440,19 +440,19 @@ When the user needs editing materials — stickers, GIFs, meme video clips, B-ro | Source | URL | Content | Best for | |---|---|---|---| -| 爱给网 | https://www.aigei.com/ | Chinese meme video clips (热梗, 4.2万+), free download | Chinese-language creators (B站/抖音) | -| B站素材酷 | https://cool.bilibili.com/ | Bilibili platform-native assets (video, audio, BGM, templates, stickers) | B站 creators; large Chinese meme collection | +| Aigei | https://www.aigei.com/ | Chinese meme video clips, free download | Chinese-language creators (Bilibili / Douyin) | +| Bilibili Material Hub | https://cool.bilibili.com/ | Bilibili platform-native assets (video, audio, BGM, templates, stickers) | Bilibili creators; large Chinese meme collection | | Tenor | https://tenor.com/ | GIFs + stickers, 30+ languages incl. Chinese, free API | Quick GIF/sticker search | | GIPHY | https://giphy.com/ | Largest GIF library, 30+ languages, sticker channel | English/multilingual GIF search | | Klipy | https://klipy.com/ | Localized GIF/sticker/clip API, supports Chinese | API-integrated search, localization | ### Priority by language -If the user communicates in Chinese, **recommend 爱给网 and B站素材酷 first** — they have the most Chinese-relevant meme and sticker content. For non-Chinese users, Tenor and GIPHY are good defaults. +If the user communicates in Chinese, **recommend Aigei and Bilibili Material Hub first** — they have the most Chinese-relevant meme and sticker content. For non-Chinese users, Tenor and GIPHY are good defaults. ### Search strategy -- **Expand search terms** based on semantic understanding of the user's request. If the user says "I want a clip of someone looking shocked", search terms might include: "惊讶 / 震惊 / 吃惊 / 哇 / wtf / shocked / surprised". Use the user's language as the base, expand with synonyms and internet slang. +- **Expand search terms** based on semantic understanding of the user's request. If the user says "I want a clip of someone looking shocked", search terms might include: "shocked / surprised / stunned / wtf". Use the user's language as the base, expand with synonyms and internet slang. - The agent can either **search directly** (using web tools or API calls) or **suggest expanded search terms** for the user to search manually. - If the user only wants links, provide the source URLs above and let the user search themselves. @@ -464,22 +464,22 @@ Common sound-effect categories for gameplay/creator edits: | Category | Example search terms (expand by language) | |---|---| -| Whoosh / transition | "whoosh", "swoosh", "风声", "转场音效" | -| Record scratch / stop | "record scratch", "vinyl stop", "唱片刮擦" | -| Impact / hit | "impact", "punch", "slap", "打击音效", "拳到肉" | -| Alert / alarm | "alert", "alarm", "siren", "警报音效" | -| Loot / pickup | "loot ping", "item pickup", "拾取音效", "开箱音效" | -| Comedy / meme | "meme sound", "vine boom", "搞笑音效", "梗音效" | -| Heartbeat / tension | "heartbeat", "tension", "心跳加速", "紧张音效" | -| Level up / success | "level up", "fanfare", "升级音效", "成功音效" | +| Whoosh / transition | "whoosh", "swoosh", "transition sound effect" | +| Record scratch / stop | "record scratch", "vinyl stop", "record stop" | +| Impact / hit | "impact", "punch", "slap", "hit sound effect" | +| Alert / alarm | "alert", "alarm", "siren", "warning sound" | +| Loot / pickup | "loot ping", "item pickup", "pickup sound effect" | +| Comedy / meme | "meme sound", "vine boom", "comedy sound effect" | +| Heartbeat / tension | "heartbeat", "tension", "suspense sound effect" | +| Level up / success | "level up", "fanfare", "success sound effect" | -Recommend sound effects per scene alongside visual materials, not in a separate chapter. See pitfall #24 about splitting advice into separate top-level chapters. +Recommend sound effects per scene alongside visual materials, not in a separate chapter. See pitfall #23 about splitting advice into separate top-level chapters. -### B站素材酷 special handling +### Bilibili Material Hub special handling -B站素材酷 (cool.bilibili.com) has no built-in search function. Two options: +Bilibili Material Hub (cool.bilibili.com) has no built-in search function. Two options: -1. **Recommend the userscript**: [bcut-resource-search](https://git.nite07.com/nite/bcut-resource-search) — a Tampermonkey script that adds search to 素材酷, supporting video, sticker/image, music, sound effect, and template categories. +1. **Recommend the userscript**: [bcut-resource-search](https://git.nite07.com/nite/bcut-resource-search) — a Tampermonkey script that adds search to the Material Hub, supporting video, sticker/image, music, sound effect, and template categories. 2. **Search via API directly**: the agent can read the userscript's source code to discover the API endpoints and make requests directly (via HTTP tools or curl). Note: the site has CORS restrictions and UA-based access control, so direct API calls may require appropriate headers. If the agent's HTTP tools are blocked, fall back to suggesting the userscript to the user. ### Anti-bot fallback @@ -531,7 +531,7 @@ This creates an isolated venv with `funasr`, `torch`, `torchaudio`. If the user 4. **Using full-resolution frames.** A 2560×1600 PNG can cost 2000+ vision tokens. Downsample to 1280 width with `scale=1280:-2` and use JPEG (`-q:v 3`). 5. **Forgetting to record artifacts in the index.** Every clip and frame batch must be recorded. Otherwise subsequent turns will re-extract the same content. 6. **Assuming the transcript is sufficient.** Gameplay videos often have long stretches of action with no dialogue. The agent should proactively check whether visual analysis is needed for segments the user asks about. -7. **Treating ASR text as final copy.** FunASR transcripts can contain wrong characters/words caused by unclear speech, noise, overlap, or similar-sounding terms (e.g., `帐篷` → `账本`). Use transcripts for locating material and understanding context, not as the absolute wording for final captions, quotes, narration, or published copy. Re-listen and manually correct any text that will appear in the final product. +7. **Treating ASR text as final copy.** FunASR transcripts can contain wrong characters/words caused by unclear speech, noise, overlap, or similar-sounding terms. Use transcripts for locating material and understanding context, not as the absolute wording for final captions, quotes, narration, or published copy. Re-listen and manually correct any text that will appear in the final product. 8. **Mixing funasr-script flags.** The bundled `funasr-nano`/`funasr-fast` use `--track STREAM_INDEX`, explicit `--language AUDIO_LANGUAGE_CODE`, and `--output-dir DIR`. Do not use the legacy `~/.local/bin/funasr-transcribe` wrapper's flags. 9. **Hardcoding tool names.** This skill is agent-agnostic. Do not assume specific MCP tools or APIs exist. Use whatever the runtime provides for vision analysis, background execution, and user notification. 10. **Hardcoding output language.** The edit plan table and all user-facing text must follow the user's language, not a fixed language. The English table in Step 8 is a template — translate columns and content to match the user's language at runtime. @@ -543,11 +543,11 @@ This creates an isolated venv with `funasr`, `torch`, `torchaudio`. If the user 16. **Making full edit plans too rigid.** For complete planning requests, ask whether the user wants multiple alternative strategies. If yes, provide modular options and extra candidate segments that the user can combine, rather than a single fixed linear plan where every section has exactly one prescribed choice. 17. **Assuming ASR timestamps are in seconds.** FunASR JSON segment `start`/`end` fields are in **milliseconds**. Filtering by `2970` (seconds) when the data is `2970000` (ms) silently returns zero results. Always check the first segment's timestamp magnitude before filtering. 18. **Trusting ASR timestamps as frame-exact.** ASR segment timestamps can be off by several seconds from the visual event they describe. When doing spot-verification, if the frame at the ASR timestamp doesn't match the transcript claim, try ±5–10 seconds before concluding the claim is wrong. -19. **Treating ASR proper nouns as authoritative.** Boss names, item names, location names, and character names in ASR transcripts may be homophone errors (e.g., "沙漠巨凤" → "沙漠巨风", "帐篷" → "账本"). Use transcript proper nouns as phonetic hints for searching, but verify against visual/UI evidence before using them in final copy. -20. **Assuming transcript-claimed action is visually present.** A transcript saying "鸟追着打" (bird chasing) does not guarantee the bird is visible on screen at that timestamp — the player may be in a menu, or the action may have happened off-screen. Always spot-verify with frames before promising visual content in the edit plan. Distinguish "transcript-grounded" from "visually confirmed" segments in the plan. -21. **Inferring psychological state from isolated ASR sentences.** ASR transcripts capture *what was said*, not *how it was said* — tone, emotion, and situational context are invisible in the text. A phrase like "笑死" (laugh to death) might mean the speaker is speechless/frustrated (无语), not genuinely amused. "你要害死我" (you're going to kill me) might be about one person entering a dungeon room first while the other stopped to eat a healing item — a coordination timing issue, not blame. Inferring psychological state or intent from context IS allowed, but it must be based on surrounding dialogue and situational context, not a single sentence in isolation. When a single transcript line could be misread, check nearby segments for context, ask the user for clarification, or mark the description as uncertain. +19. **Treating ASR proper nouns as authoritative.** Boss names, item names, location names, and character names in ASR transcripts may be homophone errors. Use transcript proper nouns as phonetic hints for searching, but verify against visual/UI evidence before using them in final copy. +20. **Assuming transcript-claimed action is visually present.** A transcript saying an action is happening (e.g., a creature chasing the player) does not guarantee the action is visible on screen at that timestamp — the player may be in a menu, or the action may have happened off-screen. Always spot-verify with frames before promising visual content in the edit plan. Distinguish "transcript-grounded" from "visually confirmed" segments in the plan. +21. **Inferring psychological state from isolated ASR sentences.** ASR transcripts capture *what was said*, not *how it was said* — tone, emotion, and situational context are invisible in the text. A phrase that looks like amusement in text might actually express frustration or speechlessness in context. A phrase that looks like blame might actually be a coordination timing issue. Inferring psychological state or intent from context IS allowed, but it must be based on surrounding dialogue and situational context, not a single sentence in isolation. When a single transcript line could be misread, check nearby segments for context, ask the user for clarification, or mark the description as uncertain. 22. **Including non-game content in a game edit plan.** If the video is a gameplay recording, the edit plan should focus on game-related content only. Off-topic segments (microphone crosstalk, refund discussions, personal banter unrelated to the game) should not be included unless the user specifically requests them. The user's chat language being Chinese does not mean they want non-game Chinese-language banter in the plan. -23. **Splitting technique/BGM/material advice into separate top-level chapters.** The edit-plan document should be organized around the script table (each scene/segment as a subsection), with BGM search terms, sound-effect search terms, visual-material search terms, Premiere technique notes (放大镜, 屏幕震动, 快速回放), and any other per-scene guidance embedded directly under each subsection — NOT split into separate top-level sections (e.g., "§4 Premiere 技巧", "§5 素材关键词", "§6 BGM 策略"). The user reads the plan scene-by-scene; scattering technique advice across detached chapters forces them to cross-reference. +23. **Splitting technique/BGM/material advice into separate top-level chapters.** The edit-plan document should be organized around the script table (each scene/segment as a subsection), with BGM search terms, sound-effect search terms, visual-material search terms, technique notes (zoom-in, screen shake, quick replay), and any other per-scene guidance embedded directly under each subsection — NOT split into separate top-level sections (e.g., "Section 4: Techniques", "Section 5: Material Keywords", "Section 6: BGM Strategy"). The user reads the plan scene-by-scene; scattering technique advice across detached chapters forces them to cross-reference. 24. **Recommending optional clips that are too fragmented to use.** Optional clip candidates with timecodes of only 3–15 seconds are often too short to form a coherent, flowable segment when inserted into a video. Prefer self-contained segments of 20+ seconds. When the only interesting moment in a time range is a single 5-second quote, recommend that the user expand the selection range in Premiere (include ±30–60 seconds of surrounding context) to form a usable segment. Note this explicitly in the optional clip library. Short merchant/dialogue segments (30–60s) tend to be more self-contained and usable than short combat fragments. 25. **Recommending visual materials without corresponding audio effects.** Each scene's material recommendations should include both visual search terms (for stickers, GIFs, overlays) AND audio/sound-effect search terms (for whoosh, record scratch, alarm, loot ping, etc.). The user needs both to execute the edit in Premiere. 26. **Saving frames to `/tmp/` or other temporary directories.** All extracted frames must go in the video file's directory (e.g., `_frames/`). When using `execute_code` or direct `ffmpeg` commands, always set the output path to the video directory — not `/tmp/frames/` or any other temp directory. Temporary directories are not portable, get cleaned up by the OS, and break the self-contained workflow. This applies to both spot-verification frames and full clip extraction. diff --git a/references/spot-verification-workflow.md b/references/spot-verification-workflow.md index fdc96dc..0a6b122 100644 --- a/references/spot-verification-workflow.md +++ b/references/spot-verification-workflow.md @@ -6,18 +6,20 @@ When verifying ASR transcript claims against actual video visuals, use this ligh - You have a transcript with timestamps and need to confirm what's visually happening at specific moments. - You need to upgrade/downgrade segment priority based on visual evidence. -- You want to check whether a transcript-claimed event (boss fight, chest opening, character death) actually has corresponding on-screen visuals. +- You want to check whether a transcript-claimed event (combat, item discovery, character death, etc.) actually has corresponding on-screen visuals. ## Procedure 1. **Extract frames at ASR-mentioned timestamps.** For each key transcript claim, extract a single frame: ```bash - ffmpeg -y -ss -i -frames:v 1 -q:v 3 /tmp/frames/frame_.jpg + ffmpeg -y -ss -i -frames:v 1 -q:v 3 /_frames/verify_.jpg ``` Use `-ss` before `-i` for fast keyframe seeking. This takes <1s per frame. + **Important:** Save frames to the video file's directory, never to `/tmp/` or other temporary paths. + 2. **Send frames to vision analysis with targeted questions.** For each frame, ask about the specific transcript claim: - "The transcript says 'X' at this time. Can you see X happening?" - "Is there a boss/enemy visible? Is combat happening?" @@ -31,26 +33,23 @@ When verifying ASR transcript claims against actual video visuals, use this ligh 4. **Record verification results.** For each verified segment, note: - Confirmed: visual matches transcript claim → keep or upgrade priority - Discrepancy: visual differs from transcript (e.g., menu open instead of combat) → downgrade or reclassify - - Not found: no visual evidence for the claimed content → mark as "voice-only" or "needs Premiere manual check" + - Not found: no visual evidence for the claimed content → mark as "voice-only" or "needs manual check in NLE" 5. **Distinguish in the edit plan.** Label each segment as: - **Visually confirmed** — frames show the claimed content - **Transcript-grounded** — only ASR evidence; visual not yet checked - - **Needs manual check** — frames inconclusive; must be verified in Premiere + - **Needs manual check** — frames inconclusive; must be verified in the NLE -## Concrete example (Romestead session 2026-07-14) +## Verification table template | ASR timestamp | Transcript claim | Frame result | Verification | |---|---|---|---| -| 49:08 | "他跟第一关的boss怎么一样" (boss comparison) | Boss "沙漠巨凤" visible with health bar, combat active | ✅ Confirmed S | -| 50:56 | "他二阶段了" (phase 2) | Boss visible, health bar nearly depleted, intense effects | ✅ Confirmed | -| 51:26 | "他打我有点疼" (hits hard) | Boss visible, player HP low (~20-30%), damage numbers | ✅ Confirmed | -| 52:28 | "打死了" (killed) | Boss still visible at 52:28; boss gone by 52:40 | ⚠️ Kill ~52:30-52:40 | -| 55:35 | "鸟追着打" (bird chasing) | Player in base/village with inventory open, no bird visible | ❌ Not confirmed | +| MM:SS | Brief description of what transcript says | Description of what frame shows | Confirmed / Discrepancy / Not found | +| MM:SS | ... | ... | ... | ## Key lessons - ASR segment `start`/`end` are in **milliseconds**, not seconds. Convert before filtering. -- ASR proper nouns are phonetic hints: "沙漠巨凤" (Phoenix) was transcribed as "沙漠巨风" (Wind). Verify names against on-screen UI. -- "打死了" (killed it) at 52:28 was actually ~52:35–52:40 visually — a 7–12 second offset. -- "鸟追着打" (bird chase) had zero visual evidence in the 55:00–56:00 range — the player was in a menu. The "凤凰之翼" (Phoenix Wing) item existed in inventory, but the chase itself was voice-only commentary or happened earlier. \ No newline at end of file +- ASR proper nouns are phonetic hints, not authoritative labels. Verify names against on-screen UI. +- ASR timestamps can be off by 5–12 seconds from the visual event. Always try ±5–10 seconds before concluding a claim is wrong. +- A transcript claiming an action is happening does not guarantee the action is visible on screen — the player may be in a menu, or the action may have happened off-screen or earlier. \ No newline at end of file