feat: multi-track transcription support

SKILL.md:
- Step 2: user can provide multiple track indexes (e.g. mic + Discord)
- Step 4: transcribe each track separately, cross-reference all transcripts
- Step 5: get-transcription supports --track filter

manage_index.py:
- get-transcription: optional --track flag to filter by track index
- Without --track, returns all transcription records (was: latest only)
This commit is contained in:
2026-07-14 12:19:42 +08:00
parent 8fbb7f176e
commit e56723a68f
2 changed files with 19 additions and 7 deletions
+10 -5
View File
@@ -50,12 +50,12 @@ Before any processing, verify the environment. Missing dependencies must be **in
Ask the user for: Ask the user for:
1. **Video file path** (absolute path; translate Windows paths like `C:\Users\...\x.mkv` to `/mnt/c/Users/.../x.mkv` on WSL). 1. **Video file path** (absolute path; translate Windows paths like `C:\Users\...\x.mkv` to `/mnt/c/Users/.../x.mkv` on WSL).
2. **Audio track index** — the ffmpeg stream index, not ordinal. If unknown, list tracks: 2. **Audio track index(es)** — the ffmpeg stream index, not ordinal. The user may provide **multiple tracks** (e.g., track 1 = own mic, track 2 = Discord audio). Each track is transcribed separately and saved as a separate JSON file. If unknown, list tracks:
```bash ```bash
ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout:stream_tags=language,title -of json VIDEO_PATH ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout:stream_tags=language,title -of json VIDEO_PATH
``` ```
**Completion criteria:** both video path and track index confirmed. **Completion criteria:** video path and at least one track index confirmed.
### Step 3 — Ask editing requirements ### Step 3 — Ask editing requirements
@@ -87,9 +87,11 @@ uv run --directory SKILL_DIR/scripts/transcription funasr-fast VIDEO_PATH --trac
uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --list-tracks uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --list-tracks
``` ```
**Skip if cached:** check the index file (Step 5) for an existing transcription entry. If one exists and the video file hasn't changed, reuse it. **Multi-track transcription:** when the user provides multiple track indexes (e.g., track 1 = mic, track 2 = Discord), run the transcription command **once per track**. Each run produces a separate JSON file named `<video_stem>_track<INDEX>_<model>.json`. Record each transcription in the index (Step 5) with its corresponding track index. When answering user questions, the agent should cross-reference all available transcripts to understand the full conversation context.
**Completion criteria:** a transcription JSON exists and is recorded in the index. JSON contains `segments` with timestamps and text. **Skip if cached:** check the index file (Step 5) for existing transcription entries. If a record exists for the same track index and the video file hasn't changed, reuse it.
**Completion criteria:** a transcription JSON exists for every requested track, and each is recorded in the index. Each JSON contains `segments` with timestamps and text.
### Step 5 — Manage index file ### Step 5 — Manage index file
@@ -139,9 +141,12 @@ uv run --directory SKILL_DIR python scripts/index/manage_index.py init --video V
# Add transcription record # Add transcription record
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-transcription --json-path PATH --track-index N --duration S uv run --directory SKILL_DIR python scripts/index/manage_index.py add-transcription --json-path PATH --track-index N --duration S
# Query existing transcription # Query existing transcription (all tracks)
uv run --directory SKILL_DIR python scripts/index/manage_index.py get-transcription uv run --directory SKILL_DIR python scripts/index/manage_index.py get-transcription
# Query specific track
uv run --directory SKILL_DIR python scripts/index/manage_index.py get-transcription --track 1
# Add a clip record # Add a clip record
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-clip --start S --end E --path PATH --reason "user asked about this segment" uv run --directory SKILL_DIR python scripts/index/manage_index.py add-clip --start S --end E --path PATH --reason "user asked about this segment"
+8 -1
View File
@@ -97,6 +97,12 @@ def cmd_add_transcription(args: argparse.Namespace) -> None:
def cmd_get_transcription(args: argparse.Namespace) -> None: def cmd_get_transcription(args: argparse.Namespace) -> None:
video = Path(args.video).expanduser().resolve() video = Path(args.video).expanduser().resolve()
conn = connect(video) conn = connect(video)
if args.track is not None:
rows = conn.execute(
"SELECT * FROM transcription WHERE track_index = ? ORDER BY created_at DESC",
(args.track,),
).fetchall()
else:
rows = conn.execute("SELECT * FROM transcription ORDER BY created_at DESC").fetchall() rows = conn.execute("SELECT * FROM transcription ORDER BY created_at DESC").fetchall()
conn.close() conn.close()
if not rows: if not rows:
@@ -258,8 +264,9 @@ def main() -> None:
p_at.add_argument("--duration", type=float, default=None) p_at.add_argument("--duration", type=float, default=None)
# get-transcription # get-transcription
p_gt = sub.add_parser("get-transcription", help="Get latest transcription record") p_gt = sub.add_parser("get-transcription", help="Get transcription records (all or by track)")
p_gt.add_argument("--video", required=True) p_gt.add_argument("--video", required=True)
p_gt.add_argument("--track", type=int, default=None, help="Filter by track index (omit to get all)")
# add-clip # add-clip
p_ac = sub.add_parser("add-clip", help="Add a clip record") p_ac = sub.add_parser("add-clip", help="Add a clip record")