feat: multi-track transcription support

SKILL.md:
- Step 2: user can provide multiple track indexes (e.g. mic + Discord)
- Step 4: transcribe each track separately, cross-reference all transcripts
- Step 5: get-transcription supports --track filter

manage_index.py:
- get-transcription: optional --track flag to filter by track index
- Without --track, returns all transcription records (was: latest only)
This commit is contained in:
2026-07-14 12:19:42 +08:00
parent 8fbb7f176e
commit e56723a68f
2 changed files with 19 additions and 7 deletions
+10 -5
View File
@@ -50,12 +50,12 @@ Before any processing, verify the environment. Missing dependencies must be **in
Ask the user for:
1. **Video file path** (absolute path; translate Windows paths like `C:\Users\...\x.mkv` to `/mnt/c/Users/.../x.mkv` on WSL).
2. **Audio track index** — the ffmpeg stream index, not ordinal. If unknown, list tracks:
2. **Audio track index(es)** — the ffmpeg stream index, not ordinal. The user may provide **multiple tracks** (e.g., track 1 = own mic, track 2 = Discord audio). Each track is transcribed separately and saved as a separate JSON file. If unknown, list tracks:
```bash
ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout:stream_tags=language,title -of json VIDEO_PATH
```
**Completion criteria:** both video path and track index confirmed.
**Completion criteria:** video path and at least one track index confirmed.
### Step 3 — Ask editing requirements
@@ -87,9 +87,11 @@ uv run --directory SKILL_DIR/scripts/transcription funasr-fast VIDEO_PATH --trac
uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --list-tracks
```
**Skip if cached:** check the index file (Step 5) for an existing transcription entry. If one exists and the video file hasn't changed, reuse it.
**Multi-track transcription:** when the user provides multiple track indexes (e.g., track 1 = mic, track 2 = Discord), run the transcription command **once per track**. Each run produces a separate JSON file named `<video_stem>_track<INDEX>_<model>.json`. Record each transcription in the index (Step 5) with its corresponding track index. When answering user questions, the agent should cross-reference all available transcripts to understand the full conversation context.
**Completion criteria:** a transcription JSON exists and is recorded in the index. JSON contains `segments` with timestamps and text.
**Skip if cached:** check the index file (Step 5) for existing transcription entries. If a record exists for the same track index and the video file hasn't changed, reuse it.
**Completion criteria:** a transcription JSON exists for every requested track, and each is recorded in the index. Each JSON contains `segments` with timestamps and text.
### Step 5 — Manage index file
@@ -139,9 +141,12 @@ uv run --directory SKILL_DIR python scripts/index/manage_index.py init --video V
# Add transcription record
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-transcription --json-path PATH --track-index N --duration S
# Query existing transcription
# Query existing transcription (all tracks)
uv run --directory SKILL_DIR python scripts/index/manage_index.py get-transcription
# Query specific track
uv run --directory SKILL_DIR python scripts/index/manage_index.py get-transcription --track 1
# Add a clip record
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-clip --start S --end E --path PATH --reason "user asked about this segment"
+9 -2
View File
@@ -97,7 +97,13 @@ def cmd_add_transcription(args: argparse.Namespace) -> None:
def cmd_get_transcription(args: argparse.Namespace) -> None:
video = Path(args.video).expanduser().resolve()
conn = connect(video)
rows = conn.execute("SELECT * FROM transcription ORDER BY created_at DESC").fetchall()
if args.track is not None:
rows = conn.execute(
"SELECT * FROM transcription WHERE track_index = ? ORDER BY created_at DESC",
(args.track,),
).fetchall()
else:
rows = conn.execute("SELECT * FROM transcription ORDER BY created_at DESC").fetchall()
conn.close()
if not rows:
print(json.dumps({"found": False}))
@@ -258,8 +264,9 @@ def main() -> None:
p_at.add_argument("--duration", type=float, default=None)
# get-transcription
p_gt = sub.add_parser("get-transcription", help="Get latest transcription record")
p_gt = sub.add_parser("get-transcription", help="Get transcription records (all or by track)")
p_gt.add_argument("--video", required=True)
p_gt.add_argument("--track", type=int, default=None, help="Filter by track index (omit to get all)")
# add-clip
p_ac = sub.add_parser("add-clip", help="Add a clip record")