feat: multi-track transcription support
SKILL.md: - Step 2: user can provide multiple track indexes (e.g. mic + Discord) - Step 4: transcribe each track separately, cross-reference all transcripts - Step 5: get-transcription supports --track filter manage_index.py: - get-transcription: optional --track flag to filter by track index - Without --track, returns all transcription records (was: latest only)
This commit is contained in:
@@ -50,12 +50,12 @@ Before any processing, verify the environment. Missing dependencies must be **in
|
||||
Ask the user for:
|
||||
|
||||
1. **Video file path** (absolute path; translate Windows paths like `C:\Users\...\x.mkv` to `/mnt/c/Users/.../x.mkv` on WSL).
|
||||
2. **Audio track index** — the ffmpeg stream index, not ordinal. If unknown, list tracks:
|
||||
2. **Audio track index(es)** — the ffmpeg stream index, not ordinal. The user may provide **multiple tracks** (e.g., track 1 = own mic, track 2 = Discord audio). Each track is transcribed separately and saved as a separate JSON file. If unknown, list tracks:
|
||||
```bash
|
||||
ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout:stream_tags=language,title -of json VIDEO_PATH
|
||||
```
|
||||
|
||||
**Completion criteria:** both video path and track index confirmed.
|
||||
**Completion criteria:** video path and at least one track index confirmed.
|
||||
|
||||
### Step 3 — Ask editing requirements
|
||||
|
||||
@@ -87,9 +87,11 @@ uv run --directory SKILL_DIR/scripts/transcription funasr-fast VIDEO_PATH --trac
|
||||
uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --list-tracks
|
||||
```
|
||||
|
||||
**Skip if cached:** check the index file (Step 5) for an existing transcription entry. If one exists and the video file hasn't changed, reuse it.
|
||||
**Multi-track transcription:** when the user provides multiple track indexes (e.g., track 1 = mic, track 2 = Discord), run the transcription command **once per track**. Each run produces a separate JSON file named `<video_stem>_track<INDEX>_<model>.json`. Record each transcription in the index (Step 5) with its corresponding track index. When answering user questions, the agent should cross-reference all available transcripts to understand the full conversation context.
|
||||
|
||||
**Completion criteria:** a transcription JSON exists and is recorded in the index. JSON contains `segments` with timestamps and text.
|
||||
**Skip if cached:** check the index file (Step 5) for existing transcription entries. If a record exists for the same track index and the video file hasn't changed, reuse it.
|
||||
|
||||
**Completion criteria:** a transcription JSON exists for every requested track, and each is recorded in the index. Each JSON contains `segments` with timestamps and text.
|
||||
|
||||
### Step 5 — Manage index file
|
||||
|
||||
@@ -139,9 +141,12 @@ uv run --directory SKILL_DIR python scripts/index/manage_index.py init --video V
|
||||
# Add transcription record
|
||||
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-transcription --json-path PATH --track-index N --duration S
|
||||
|
||||
# Query existing transcription
|
||||
# Query existing transcription (all tracks)
|
||||
uv run --directory SKILL_DIR python scripts/index/manage_index.py get-transcription
|
||||
|
||||
# Query specific track
|
||||
uv run --directory SKILL_DIR python scripts/index/manage_index.py get-transcription --track 1
|
||||
|
||||
# Add a clip record
|
||||
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-clip --start S --end E --path PATH --reason "user asked about this segment"
|
||||
|
||||
|
||||
@@ -97,7 +97,13 @@ def cmd_add_transcription(args: argparse.Namespace) -> None:
|
||||
def cmd_get_transcription(args: argparse.Namespace) -> None:
|
||||
video = Path(args.video).expanduser().resolve()
|
||||
conn = connect(video)
|
||||
rows = conn.execute("SELECT * FROM transcription ORDER BY created_at DESC").fetchall()
|
||||
if args.track is not None:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM transcription WHERE track_index = ? ORDER BY created_at DESC",
|
||||
(args.track,),
|
||||
).fetchall()
|
||||
else:
|
||||
rows = conn.execute("SELECT * FROM transcription ORDER BY created_at DESC").fetchall()
|
||||
conn.close()
|
||||
if not rows:
|
||||
print(json.dumps({"found": False}))
|
||||
@@ -258,8 +264,9 @@ def main() -> None:
|
||||
p_at.add_argument("--duration", type=float, default=None)
|
||||
|
||||
# get-transcription
|
||||
p_gt = sub.add_parser("get-transcription", help="Get latest transcription record")
|
||||
p_gt = sub.add_parser("get-transcription", help="Get transcription records (all or by track)")
|
||||
p_gt.add_argument("--video", required=True)
|
||||
p_gt.add_argument("--track", type=int, default=None, help="Filter by track index (omit to get all)")
|
||||
|
||||
# add-clip
|
||||
p_ac = sub.add_parser("add-clip", help="Add a clip record")
|
||||
|
||||
Reference in New Issue
Block a user