feat: video-edit-planner skill — transcribe, extract frames, plan edits
A conversational video-editing planning assistant. Provides tools for: - Audio transcription via bundled funasr-script (Fun-ASR-Nano / SenseVoice) - On-demand clip extraction and frame sampling (ffmpeg, hardware-accelerated) - SQLite index to track all artifacts and avoid duplicate processing - Vision analysis guidance with binary-search-style frame sampling - Iterative Markdown-table edit plan output Agent-agnostic: no platform-specific tool names, works with any agent runtime. Dependencies checked at guidance level with OS package manager install hints.
This commit is contained in:
+28
@@ -0,0 +1,28 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
.venv/
|
||||
venv/
|
||||
|
||||
# uv
|
||||
scripts/transcription/.venv/
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Editor
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# Index databases (runtime artifacts, not source)
|
||||
*.vedit.db
|
||||
|
||||
# Extracted clips and frames (runtime artifacts)
|
||||
*_clips/
|
||||
*_frames/
|
||||
|
||||
# Showinfo logs
|
||||
showinfo.log
|
||||
@@ -0,0 +1,290 @@
|
||||
---
|
||||
name: video-edit-planner
|
||||
description: "Use when planning video edits: transcribe audio, extract key frames, analyze visuals, and produce cut/transition/assembly plans via iterative dialogue."
|
||||
license: MIT
|
||||
tags:
|
||||
[
|
||||
video-editing,
|
||||
transcription,
|
||||
funasr,
|
||||
frame-extraction,
|
||||
vision,
|
||||
planning,
|
||||
ffmpeg,
|
||||
]
|
||||
---
|
||||
|
||||
# Video Edit Planner
|
||||
|
||||
## Overview
|
||||
|
||||
A conversational video-editing planning assistant. The user provides a video file and an audio track index; the skill transcribes the audio, extracts key frames on demand, analyzes them visually, and produces an iterative Markdown-table edit plan (timecodes, segment descriptions, transitions, notes).
|
||||
|
||||
The skill is **agent-driven, not scripted**: it provides tools (transcription, clipping, frame extraction, index management) and guidance for the agent to autonomously decide when and what to extract based on user needs.
|
||||
|
||||
## When to Use
|
||||
|
||||
- User provides a video path and wants help planning cuts, transitions, or assembly.
|
||||
- User asks "help me edit this video" or "what parts should I cut from this recording."
|
||||
- User needs to understand video content beyond what the transcript alone reveals.
|
||||
- User wants an iterative dialogue to refine an edit plan over multiple turns.
|
||||
|
||||
**Don't use for:** actual video editing/rendering (this skill plans, it does not produce video files), or pure audio-only transcription.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1 — Check dependencies
|
||||
|
||||
Before any processing, verify the environment. Missing dependencies must be **installed only with explicit user consent**.
|
||||
|
||||
| Dependency | Check command | Install (prefer OS package manager; fallback to pip) |
|
||||
| -------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `ffmpeg` + `ffprobe` | `which ffmpeg ffprobe` | Linux: `sudo pacman -S ffmpeg` / `sudo apt install ffmpeg`; Windows: `winget install Gyan.FFmpeg`; macOS: `brew install ffmpeg` |
|
||||
| `uv` | `which uv` | Linux: `sudo pacman -S uv` / `sudo apt install uv` (if available); macOS: `brew install uv`; Windows: `winget install astral-sh.uv`; fallback: `pip install uv` |
|
||||
| `python3` | `which python3` | Linux: `sudo pacman -S python`; Windows: `winget install Python.Python.3`; macOS: `brew install python` |
|
||||
|
||||
**Completion criteria:** all three dependencies confirmed present, or user has explicitly declined installation (in which case stop — the skill cannot proceed).
|
||||
|
||||
### Step 2 — Gather inputs
|
||||
|
||||
Ask the user for:
|
||||
|
||||
1. **Video file path** (absolute path; translate Windows paths like `C:\Users\...\x.mkv` to `/mnt/c/Users/.../x.mkv` on WSL).
|
||||
2. **Audio track index** — the ffmpeg stream index, not ordinal. If unknown, list tracks:
|
||||
```bash
|
||||
ffprobe -v error -select_streams a -show_entries stream=index,codec_name,channels,channel_layout:stream_tags=language,title -of json VIDEO_PATH
|
||||
```
|
||||
|
||||
**Completion criteria:** both video path and track index confirmed.
|
||||
|
||||
### Step 3 — Ask editing requirements
|
||||
|
||||
Before transcribing, ask the user what they want to do with the video. Examples:
|
||||
|
||||
- "Trim out dead air and boring parts"
|
||||
- "Make a highlight reel of funny moments"
|
||||
- "Plan transitions between scenes"
|
||||
- "Find the best 30-second clip for a short"
|
||||
|
||||
User requirements are often vague in the first pass — that's expected. The plan is refined iteratively.
|
||||
|
||||
**Completion criteria:** user has described at least a rough editing goal.
|
||||
|
||||
### Step 4 — Transcribe audio
|
||||
|
||||
Use the bundled `funasr-script` in `scripts/transcription/`.
|
||||
|
||||
**First run** requires `uv sync` (installs funasr + torch, may take several minutes). Subsequent runs reuse the cached venv. Notify the user before first-time setup.
|
||||
|
||||
```bash
|
||||
# Default: Fun-ASR-Nano (high quality, Chinese-optimized)
|
||||
uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --track TRACK_INDEX --output-dir VIDEO_DIR
|
||||
|
||||
# Fast preview: SenseVoice
|
||||
uv run --directory SKILL_DIR/scripts/transcription funasr-fast VIDEO_PATH --track TRACK_INDEX --output-dir VIDEO_DIR
|
||||
|
||||
# List tracks
|
||||
uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --list-tracks
|
||||
```
|
||||
|
||||
**Skip if cached:** check the index file (Step 5) for an existing transcription entry. If one exists and the video file hasn't changed, reuse it.
|
||||
|
||||
**Completion criteria:** a transcription JSON exists and is recorded in the index. JSON contains `segments` with timestamps and text.
|
||||
|
||||
### Step 5 — Manage index file
|
||||
|
||||
An SQLite database tracks all processing artifacts to avoid duplicate work. It lives **next to the video file**.
|
||||
|
||||
```
|
||||
<video_dir>/<video_stem>.vedit.db
|
||||
```
|
||||
|
||||
Schema (managed by `scripts/index/manage_index.py`):
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS transcription (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
json_path TEXT NOT NULL,
|
||||
track_index INTEGER NOT NULL,
|
||||
created_at REAL NOT NULL, -- Unix timestamp
|
||||
duration_s REAL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS clips (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
start_time REAL NOT NULL, -- seconds from video start
|
||||
end_time REAL NOT NULL,
|
||||
path TEXT NOT NULL,
|
||||
created_at REAL NOT NULL, -- Unix timestamp
|
||||
reason TEXT -- why this clip was extracted
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS frames (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
clip_id INTEGER NOT NULL REFERENCES clips(id),
|
||||
timestamp REAL NOT NULL, -- seconds from video start
|
||||
path TEXT NOT NULL,
|
||||
scene_score REAL, -- ffmpeg scene score if available
|
||||
method TEXT NOT NULL, -- 'scene' | 'sample' | 'both'
|
||||
created_at REAL NOT NULL -- Unix timestamp
|
||||
);
|
||||
```
|
||||
|
||||
Use `scripts/index/manage_index.py` for all CRUD operations:
|
||||
|
||||
```bash
|
||||
# Initialize database (idempotent)
|
||||
uv run --directory SKILL_DIR python scripts/index/manage_index.py init --video VIDEO_PATH
|
||||
|
||||
# Add transcription record
|
||||
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-transcription --json-path PATH --track-index N --duration S
|
||||
|
||||
# Query existing transcription
|
||||
uv run --directory SKILL_DIR python scripts/index/manage_index.py get-transcription
|
||||
|
||||
# Add a clip record
|
||||
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-clip --start S --end E --path PATH --reason "user asked about this segment"
|
||||
|
||||
# Add frame records (batch from a directory)
|
||||
uv run --directory SKILL_DIR python scripts/index/manage_index.py add-frames --clip-id N --frames-dir DIR --method scene
|
||||
|
||||
# List all clips and frames
|
||||
uv run --directory SKILL_DIR python scripts/index/manage_index.py list
|
||||
|
||||
# Check if a time range has already been extracted
|
||||
uv run --directory SKILL_DIR python scripts/index/manage_index.py check-range --start S --end E
|
||||
```
|
||||
|
||||
**Completion criteria:** index file exists and all produced artifacts are recorded in it.
|
||||
|
||||
### Step 6 — Extract clips and frames (on demand)
|
||||
|
||||
**When to extract:** the agent autonomously decides based on user needs. If the transcript alone is insufficient to answer (e.g., user asks about visual content, or a transcript segment is sparse/empty for a time range the user cares about), extract frames for that range.
|
||||
|
||||
**Binary-search-style frame analysis:**
|
||||
|
||||
1. Extract a clip for the target time range.
|
||||
2. Extract scene-change frames + uniform samples from the clip.
|
||||
3. Send a few representative frames to vision analysis.
|
||||
4. If a sub-range needs deeper understanding, extract more frames from that sub-range.
|
||||
5. Once the relevant range is identified, send all frames in that range (in batches if needed) to vision.
|
||||
|
||||
**Clip extraction** (keyframe-accurate, `-c copy`):
|
||||
|
||||
```bash
|
||||
ffmpeg -hide_banner -y -ss START -to END -i VIDEO_PATH -c copy -map 0:v:0 CLIP_PATH
|
||||
```
|
||||
|
||||
**Frame extraction** (scene detection + uniform sampling, hardware-accelerated, downsampled):
|
||||
|
||||
```bash
|
||||
# Hardware-accelerated (CUDA/NVDEC preferred)
|
||||
ffmpeg -hide_banner -y -hwaccel cuda -i CLIP_PATH \
|
||||
-vf "scale=1280:-2,fps=1/2,select='gt(scene,THRESHOLD)+gte(n,0)',showinfo" \
|
||||
-vsync vfr -q:v 3 FRAMES_DIR/frame_%04d.jpg
|
||||
|
||||
# CPU fallback
|
||||
ffmpeg -hide_banner -y -i CLIP_PATH \
|
||||
-vf "scale=1280:-2,fps=1/2,select='gt(scene,THRESHOLD)',showinfo" \
|
||||
-vsync vfr -q:v 3 FRAMES_DIR/frame_%04d.jpg
|
||||
```
|
||||
|
||||
**Parameters the agent chooses:**
|
||||
|
||||
| Parameter | Default | Guidance |
|
||||
| --------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------- |
|
||||
| `scale` width | `1280` | Raise to `1920` for short clips (<30s) where detail matters; lower to `854` for long clips (>120s) to save tokens |
|
||||
| `THRESHOLD` | `0.3` | Lower to `0.1`–`0.2` for gameplay footage (subtle scene changes); raise to `0.4` for dynamic content |
|
||||
| `fps` sample rate | `1/2` (1 frame per 2s) | Raise to `1` (1fps) for short clips; lower to `1/5` for long clips |
|
||||
| `-q:v` (JPEG quality) | `3` | Range 2–31, lower = higher quality. 3 is high quality; 5 is a good balance |
|
||||
|
||||
**All clips and frames are saved next to the video file** (e.g., `<video_stem>_clips/` and `<video_stem>_frames/`). The user manages cleanup.
|
||||
|
||||
**Record every extracted clip and frame in the index** before proceeding.
|
||||
|
||||
**Completion criteria:** clips and frames exist on disk and are recorded in the index.
|
||||
|
||||
### Step 7 — Vision analysis
|
||||
|
||||
Use whatever vision capability the agent's runtime provides to analyze extracted frames. The agent chooses:
|
||||
|
||||
- **Which model** to use (depends on the agent's runtime and available vision capabilities).
|
||||
- **Batch size** — depends on the model's context window. See `references/frame-extraction-guide.md` for token cost estimates at different resolutions.
|
||||
- **How many frames** per analysis pass — use the binary-search approach from Step 6.
|
||||
|
||||
**Completion criteria:** the agent has enough visual understanding to answer the user's question or produce the requested edit plan.
|
||||
|
||||
### Step 8 — Produce edit plan
|
||||
|
||||
Output a Markdown table with overall recommendations, then let the user iterate. **Respond in the same language the user is using.**
|
||||
|
||||
```markdown
|
||||
## Edit Plan
|
||||
|
||||
### Overall Recommendation
|
||||
|
||||
<1-3 sentences of high-level recommendation addressing the user's goal>
|
||||
|
||||
### Detailed Plan
|
||||
|
||||
| Timecode | Duration | Segment Description | Action | Transition | Notes |
|
||||
| ----------------- | -------- | ------------------------------- | ------------- | ---------- | ------------------ |
|
||||
| 00:01:23–00:01:45 | 22s | Character enters town, dialogue | Keep | Hard cut | Can add subtitles |
|
||||
| 00:01:45–00:02:10 | 25s | Walking, no dialogue or event | Cut | — | — |
|
||||
| 00:02:10–00:02:35 | 25s | Combat starts, highlight moment | Keep, slow-mo | Fade in | Sync to BGM rhythm |
|
||||
| ... | ... | ... | ... | ... | ... |
|
||||
```
|
||||
|
||||
**Completion criteria:** user receives a Markdown table addressing their stated goal, and is invited to ask follow-up questions.
|
||||
|
||||
### Step 9 — Iterate
|
||||
|
||||
The user will likely ask follow-ups: "what about this section?", "add a transition here", "can you check what's happening at 5:30?". For each:
|
||||
|
||||
1. Check if the answer can be derived from existing transcript + frames (check index first).
|
||||
2. If not, extract new clips/frames for the requested range (Step 6).
|
||||
3. Analyze and update the plan.
|
||||
|
||||
**Completion criteria:** user is satisfied with the plan or explicitly ends the session.
|
||||
|
||||
## Long-running operations
|
||||
|
||||
Before any operation expected to take >1 minute (first-time `uv sync`, transcription of long videos, large frame extraction):
|
||||
|
||||
1. **Notify the user** that a long operation is about to start and give a rough time estimate.
|
||||
2. Run the operation in a way that does not block the conversation.
|
||||
3. Do not poll progress at high frequency — wait for completion notification rather than checking every few seconds.
|
||||
|
||||
## Self-contained setup
|
||||
|
||||
The skill bundles its own transcription project under `scripts/transcription/`. On first use:
|
||||
|
||||
```bash
|
||||
cd SKILL_DIR/scripts/transcription && uv sync
|
||||
```
|
||||
|
||||
This creates an isolated venv with `funasr`, `torch`, `torchaudio`. If the user has previously installed these packages via uv in other projects, uv's cache will reuse them — first-time cost is only the link step.
|
||||
|
||||
## Common pitfalls
|
||||
|
||||
1. **Installing dependencies without consent.** Always ask the user first. If they decline, stop — the skill cannot proceed without ffmpeg, uv, and python3.
|
||||
2. **Re-processing already-transcribed videos.** Always check the index file first. If a transcription record exists and the video hasn't been modified, reuse it.
|
||||
3. **Extracting frames from the entire video.** This is extremely slow for long videos. Always clip the target range first with `-c copy`, then extract frames from the clip.
|
||||
4. **Using full-resolution frames.** A 2560×1600 PNG can cost 2000+ vision tokens. Downsample to 1280 width with `scale=1280:-2` and use JPEG (`-q:v 3`).
|
||||
5. **Forgetting to record artifacts in the index.** Every clip and frame batch must be recorded. Otherwise subsequent turns will re-extract the same content.
|
||||
6. **Assuming the transcript is sufficient.** Gameplay videos often have long stretches of action with no dialogue. The agent should proactively check whether visual analysis is needed for segments the user asks about.
|
||||
7. **Mixing funasr-script flags.** The bundled `funasr-nano`/`funasr-fast` use `--track STREAM_INDEX` and `--output-dir DIR`. Do not use the legacy `~/.local/bin/funasr-transcribe` wrapper's flags.
|
||||
8. **Hardcoding tool names.** This skill is agent-agnostic. Do not assume specific MCP tools or APIs exist. Use whatever the runtime provides for vision analysis, background execution, and user notification.
|
||||
9. **Hardcoding output language.** The edit plan table and all user-facing text must follow the user's language, not a fixed language. The English table in Step 8 is a template — translate columns and content to match the user's language at runtime.
|
||||
|
||||
## Verification checklist
|
||||
|
||||
- [ ] All dependencies (ffmpeg, uv, python3) confirmed present or user declined.
|
||||
- [ ] Video path and audio track index confirmed.
|
||||
- [ ] User's editing goal understood.
|
||||
- [ ] Transcription completed or reused from cache.
|
||||
- [ ] Index file initialized at `<video_dir>/<video_stem>.vedit.db`.
|
||||
- [ ] All extracted clips and frames recorded in the index.
|
||||
- [ ] Vision analysis performed where needed.
|
||||
- [ ] Edit plan produced as Markdown table with overall recommendation.
|
||||
- [ ] User invited to iterate.
|
||||
@@ -0,0 +1,120 @@
|
||||
# Frame Extraction & Vision Token Guide
|
||||
|
||||
## Vision model token costs at common resolutions
|
||||
|
||||
All major vision models (OpenAI GPT-4o, Anthropic Claude, Google Gemini) process images as tiled patches. Higher resolution = more tokens = higher cost and latency.
|
||||
|
||||
### OpenAI GPT-4o / GPT-4.1
|
||||
|
||||
- **Low detail:** fixed 85 tokens per image.
|
||||
- **High detail:** fit inside 2048×2048 box, scale shortest side to ~768px, count 512×512 tiles, then `85 + 170 × tiles`.
|
||||
- 1024×1024 → ~765 tokens
|
||||
- 1280×800 → ~935 tokens
|
||||
- 1920×1080 → ~1105 tokens
|
||||
|
||||
Source: [OpenAI vision docs](https://platform.openai.com/docs/guides/vision), [Spoold token estimator](https://www.spoold.com/tools/vision-tokens)
|
||||
|
||||
### Anthropic Claude
|
||||
|
||||
- Images are processed in 28×28 pixel patches (visual tokens).
|
||||
- Token cost: `⌈width / 28⌉ × ⌈height / 28⌉`
|
||||
- **Standard tier** (most models): max long edge 1568px, max 1568 visual tokens.
|
||||
- **High-res tier** (Opus 4.7/4.8, Sonnet 5, Fable 5, Mythos 5): max long edge 2576px, max 4784 visual tokens.
|
||||
- Images larger than either limit are downscaled before processing.
|
||||
- 1024×1024 → ~1296 tokens (standard)
|
||||
- 1280×800 → ~1334 tokens (standard)
|
||||
- 1920×1080 → ~1560 tokens (standard, capped), ~2691 tokens (high-res)
|
||||
- > 20 images per request: each image must be ≤2000px on both sides.
|
||||
- Max 100 images per API request (200k context models).
|
||||
|
||||
Source: [Claude vision docs](https://platform.claude.com/docs/en/build-with-claude/vision)
|
||||
|
||||
### Google Gemini
|
||||
|
||||
- ≤384×384: 258 tokens per image.
|
||||
- Larger: 768×768 tiles × 258 tokens per tile.
|
||||
- 1280×800 → ~1032 tokens
|
||||
- 1920×1080 → ~1548 tokens
|
||||
|
||||
Source: [Spoold token estimator](https://www.spoold.com/tools/vision-tokens)
|
||||
|
||||
## Recommended resolution by use case
|
||||
|
||||
| Clip duration | Recommended width | JPEG quality | Est. tokens/frame (GPT-4o) | Rationale |
|
||||
| ------------- | ----------------- | ------------ | -------------------------- | ------------------------------------------------------- |
|
||||
| <15s | 1920 | 2 | ~1105 | Short clips need detail; few frames extracted |
|
||||
| 15–60s | 1280 | 3 | ~935 | Balance detail vs. token cost |
|
||||
| 60–180s | 1280 | 3 | ~935 | Moderate length; standard default |
|
||||
| >180s | 854 | 5 | ~680 | Long clips produce many frames; minimize per-frame cost |
|
||||
|
||||
## Batch size guidance
|
||||
|
||||
| Model context | Max images/batch | Practical batch | Notes |
|
||||
| ------------- | ---------------- | --------------- | ------------------------------------------------- |
|
||||
| 8k tokens | 3–4 | 2–3 | Leave room for text prompt + response |
|
||||
| 32k tokens | 15–20 | 5–8 | Conservative; allows multi-turn follow-ups |
|
||||
| 128k tokens | 60+ | 10–15 | Still batch to maintain response quality |
|
||||
| 200k tokens | 100+ | 15–20 | Large batches ok but response quality may degrade |
|
||||
|
||||
**Rule of thumb:** start with 5–8 frames per batch. If the model handles it well, increase. If responses are shallow or cut off, decrease.
|
||||
|
||||
## Binary-search frame analysis workflow
|
||||
|
||||
```
|
||||
1. Extract clip (e.g., 00:01:00–00:02:00, 60s)
|
||||
2. Extract frames: scene + uniform (fps=0.5) → ~30 frames + scene changes
|
||||
3. Select 5 representative frames (evenly spaced across the range)
|
||||
4. Send to vision: "What is happening in these frames? Summarize the visual content."
|
||||
5. Identify sub-range of interest (e.g., 00:01:20–00:01:35 looks relevant)
|
||||
6. Extract denser frames from sub-range (fps=2, threshold=0.15)
|
||||
7. Send sub-range frames to vision for detailed analysis
|
||||
8. Use combined transcript + visual understanding to answer user question
|
||||
```
|
||||
|
||||
## FFmpeg scene detection parameters
|
||||
|
||||
### Threshold guide
|
||||
|
||||
| Content type | Recommended `scene` threshold | Notes |
|
||||
| ------------------------- | ----------------------------- | -------------------------------------------------- |
|
||||
| Gameplay (subtle changes) | 0.1–0.2 | Cuts between similar-looking scenes |
|
||||
| Standard video | 0.25–0.35 | Typical content with scene cuts |
|
||||
| Dynamic/high-action | 0.35–0.45 | Many scene changes; higher threshold reduces noise |
|
||||
|
||||
### Hardware acceleration
|
||||
|
||||
Check support:
|
||||
|
||||
```bash
|
||||
ffmpeg -hwaccels
|
||||
```
|
||||
|
||||
Common methods:
|
||||
|
||||
- `cuda` — NVIDIA NVDEC/NVENC (preferred if available)
|
||||
- `vaapi` — Intel/AMD
|
||||
- `qsv` — Intel QuickSync
|
||||
- `vulkan` — cross-platform GPU
|
||||
|
||||
### One-step vs two-step frame extraction
|
||||
|
||||
**One-step (recommended):** decode once, filter for scene changes + uniform sampling simultaneously.
|
||||
|
||||
```bash
|
||||
ffmpeg -hwaccel cuda -i clip.mkv \
|
||||
-vf "scale=1280:-2,fps=0.5,select='gt(scene\,0.3)',showinfo" \
|
||||
-vsync vfr -q:v 3 frame_%04d.jpg
|
||||
```
|
||||
|
||||
**Two-step:** run `scdet` filter first to get timestamps, then extract frames at those timestamps. Slower for many frames but allows precise per-timestamp extraction.
|
||||
|
||||
```bash
|
||||
# Step 1: detect scene changes
|
||||
ffmpeg -i clip.mkv -vf scdet -f null - 2> scdet.log
|
||||
# Step 2: extract frame at each timestamp
|
||||
for t in $(grep scdet.score scdet.log | ...); do
|
||||
ffmpeg -ss $t -i clip.mkv -frames:v 1 frame_${t}.jpg
|
||||
done
|
||||
```
|
||||
|
||||
Prefer one-step unless you need per-timestamp control.
|
||||
@@ -0,0 +1,221 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Extract clips and frames from a video for visual analysis.
|
||||
|
||||
This script wraps ffmpeg to:
|
||||
1. Extract a time range as a clip (keyframe-accurate, -c copy).
|
||||
2. Extract frames from that clip (scene detection + uniform sampling).
|
||||
|
||||
It is designed to be called by the agent, not directly by the user.
|
||||
The agent decides parameters based on clip length, content type, and user needs.
|
||||
|
||||
Usage:
|
||||
# Extract a clip
|
||||
python extract_frames.py clip --video VIDEO --start 60 --end 90 --output clip.mkv
|
||||
|
||||
# Extract frames from a clip (or full video)
|
||||
python extract_frames.py frames --input clip.mkv --output-dir frames/ \\
|
||||
--width 1280 --threshold 0.3 --fps 0.5 --quality 3
|
||||
|
||||
# Check hardware acceleration support
|
||||
python extract_frames.py check-hwaccel
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def run_command(cmd: list[str], *, capture: bool = True) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run(
|
||||
cmd,
|
||||
check=True,
|
||||
text=True,
|
||||
stdout=subprocess.PIPE if capture else None,
|
||||
stderr=subprocess.PIPE if capture else None,
|
||||
)
|
||||
|
||||
|
||||
def check_hwaccel() -> dict:
|
||||
"""Check available hardware acceleration methods."""
|
||||
if not shutil.which("ffmpeg"):
|
||||
return {"error": "ffmpeg not found"}
|
||||
proc = subprocess.run(
|
||||
["ffmpeg", "-hide_banner", "-hwaccels"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
hwaccels = [
|
||||
line.strip()
|
||||
for line in proc.stdout.splitlines()
|
||||
if line.strip() and not line.startswith("Hardware")
|
||||
]
|
||||
return {"hwaccels": hwaccels, "cuda": "cuda" in hwaccels, "vaapi": "vaapi" in hwaccels}
|
||||
|
||||
|
||||
def cmd_clip(args: argparse.Namespace) -> None:
|
||||
"""Extract a clip using -c copy (keyframe-accurate)."""
|
||||
video = Path(args.video).expanduser().resolve()
|
||||
output = Path(args.output).expanduser().resolve()
|
||||
|
||||
if not video.exists():
|
||||
sys.exit(f"Video not found: {video}")
|
||||
|
||||
cmd = [
|
||||
"ffmpeg", "-hide_banner", "-y",
|
||||
"-ss", str(args.start),
|
||||
"-to", str(args.end),
|
||||
"-i", str(video),
|
||||
"-c", "copy",
|
||||
"-map", "0:v:0",
|
||||
str(output),
|
||||
]
|
||||
|
||||
try:
|
||||
run_command(cmd)
|
||||
print(json.dumps({
|
||||
"status": "ok",
|
||||
"clip_path": str(output),
|
||||
"start": args.start,
|
||||
"end": args.end,
|
||||
"duration": args.end - args.start,
|
||||
}))
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(json.dumps({"status": "error", "stderr": e.stderr}))
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def cmd_frames(args: argparse.Namespace) -> None:
|
||||
"""Extract frames from a video/clip with scene detection + uniform sampling."""
|
||||
input_path = Path(args.input).expanduser().resolve()
|
||||
output_dir = Path(args.output_dir).expanduser().resolve()
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if not input_path.exists():
|
||||
sys.exit(f"Input not found: {input_path}")
|
||||
|
||||
# Build the scale filter
|
||||
scale_filter = f"scale={args.width}:-2"
|
||||
|
||||
# Build the select filter: scene change OR uniform sampling
|
||||
# fps filter handles uniform sampling; select filter handles scene detection
|
||||
# We use fps for uniform sampling and select for scene changes, combined
|
||||
select_filter = f"select='gt(scene\\,{args.threshold})'"
|
||||
vf_parts = [scale_filter]
|
||||
|
||||
if args.fps > 0:
|
||||
# fps filter for uniform sampling
|
||||
vf_parts.append(f"fps={args.fps}")
|
||||
|
||||
# Add scene detection select filter
|
||||
vf_parts.append(select_filter)
|
||||
|
||||
# Add showinfo for scene score logging
|
||||
if args.showinfo:
|
||||
vf_parts.append("showinfo")
|
||||
|
||||
vf = ",".join(vf_parts)
|
||||
|
||||
# Build command
|
||||
cmd = ["ffmpeg", "-hide_banner", "-y"]
|
||||
|
||||
# Hardware acceleration
|
||||
if args.hwaccel:
|
||||
hwaccel_info = check_hwaccel()
|
||||
if hwaccel_info.get("cuda") and args.hwaccel == "cuda":
|
||||
cmd.extend(["-hwaccel", "cuda"])
|
||||
elif args.hwaccel != "auto":
|
||||
# Try the requested hwaccel, fall back to none
|
||||
cmd.extend(["-hwaccel", args.hwaccel])
|
||||
|
||||
cmd.extend(["-i", str(input_path)])
|
||||
cmd.extend(["-vf", vf])
|
||||
cmd.extend(["-vsync", "vfr"])
|
||||
cmd.extend(["-q:v", str(args.quality)])
|
||||
|
||||
frame_pattern = str(output_dir / "frame_%04d.jpg")
|
||||
cmd.append(frame_pattern)
|
||||
|
||||
# Capture showinfo log
|
||||
if args.showinfo:
|
||||
log_file = output_dir / "showinfo.log"
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
log_file.write_text(result.stderr)
|
||||
if result.returncode != 0:
|
||||
print(json.dumps({"status": "error", "stderr": result.stderr[-2000:]}))
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
print(json.dumps({"status": "error", "error": str(e)}))
|
||||
sys.exit(1)
|
||||
else:
|
||||
try:
|
||||
run_command(cmd)
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(json.dumps({"status": "error", "stderr": e.stderr[-2000:] if e.stderr else ""}))
|
||||
sys.exit(1)
|
||||
|
||||
# Count extracted frames
|
||||
frame_files = sorted(output_dir.glob("frame_*.jpg"))
|
||||
print(json.dumps({
|
||||
"status": "ok",
|
||||
"frames_dir": str(output_dir),
|
||||
"frame_count": len(frame_files),
|
||||
"width": args.width,
|
||||
"threshold": args.threshold,
|
||||
"fps": args.fps,
|
||||
"quality": args.quality,
|
||||
}))
|
||||
|
||||
|
||||
def cmd_check_hwaccel(args: argparse.Namespace) -> None:
|
||||
"""Print available hardware acceleration methods."""
|
||||
info = check_hwaccel()
|
||||
print(json.dumps(info, indent=2))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Extract clips and frames for video edit planning.")
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
# clip
|
||||
p_clip = sub.add_parser("clip", help="Extract a time range as a clip")
|
||||
p_clip.add_argument("--video", required=True, help="Source video path")
|
||||
p_clip.add_argument("--start", required=True, type=float, help="Start time in seconds")
|
||||
p_clip.add_argument("--end", required=True, type=float, help="End time in seconds")
|
||||
p_clip.add_argument("--output", required=True, help="Output clip path")
|
||||
|
||||
# frames
|
||||
p_frames = sub.add_parser("frames", help="Extract frames from a video/clip")
|
||||
p_frames.add_argument("--input", required=True, help="Input video/clip path")
|
||||
p_frames.add_argument("--output-dir", required=True, help="Output directory for frames")
|
||||
p_frames.add_argument("--width", type=int, default=1280, help="Output width in pixels (height auto-scaled)")
|
||||
p_frames.add_argument("--threshold", type=float, default=0.3, help="Scene change threshold (0-1)")
|
||||
p_frames.add_argument("--fps", type=float, default=0.5, help="Uniform sampling fps (0 = scene-only)")
|
||||
p_frames.add_argument("--quality", type=int, default=3, help="JPEG quality (2-31, lower=better)")
|
||||
p_frames.add_argument("--hwaccel", default="cuda", help="Hardware acceleration (cuda/vaapi/auto/none)")
|
||||
p_frames.add_argument("--showinfo", action="store_true", default=True, help="Log scene scores to showinfo.log")
|
||||
|
||||
# check-hwaccel
|
||||
sub.add_parser("check-hwaccel", help="Check available hardware acceleration")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
commands = {
|
||||
"clip": cmd_clip,
|
||||
"frames": cmd_frames,
|
||||
"check-hwaccel": cmd_check_hwaccel,
|
||||
}
|
||||
commands[args.command](args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,311 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Manage the video-edit-planner SQLite index.
|
||||
|
||||
The index file lives next to the video as <video_stem>.vedit.db.
|
||||
It tracks transcriptions, clips, and frames to avoid duplicate processing.
|
||||
|
||||
Usage:
|
||||
python manage_index.py init --video VIDEO_PATH
|
||||
python manage_index.py add-transcription --json-path PATH --track-index N --duration S
|
||||
python manage_index.py get-transcription
|
||||
python manage_index.py add-clip --start S --end E --path PATH --reason "..."
|
||||
python manage_index.py add-frames --clip-id N --frames-dir DIR --method scene
|
||||
python manage_index.py list
|
||||
python manage_index.py check-range --start S --end E
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS transcription (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
json_path TEXT NOT NULL,
|
||||
track_index INTEGER NOT NULL,
|
||||
created_at REAL NOT NULL,
|
||||
duration_s REAL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS clips (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
start_time REAL NOT NULL,
|
||||
end_time REAL NOT NULL,
|
||||
path TEXT NOT NULL,
|
||||
created_at REAL NOT NULL,
|
||||
reason TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS frames (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
clip_id INTEGER NOT NULL REFERENCES clips(id),
|
||||
timestamp REAL NOT NULL,
|
||||
path TEXT NOT NULL,
|
||||
scene_score REAL,
|
||||
method TEXT NOT NULL,
|
||||
created_at REAL NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_frames_clip_id ON frames(clip_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_clips_time ON clips(start_time, end_time);
|
||||
"""
|
||||
|
||||
|
||||
def db_path_for_video(video_path: Path) -> Path:
|
||||
"""Return the index database path for a given video file."""
|
||||
return video_path.parent / f"{video_path.stem}.vedit.db"
|
||||
|
||||
|
||||
def connect(video_path: Path) -> sqlite3.Connection:
|
||||
db = db_path_for_video(video_path)
|
||||
if not db.exists():
|
||||
sys.exit(f"Index not found: {db}. Run 'init' first.")
|
||||
conn = sqlite3.connect(str(db))
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("PRAGMA foreign_keys = ON")
|
||||
return conn
|
||||
|
||||
|
||||
def cmd_init(args: argparse.Namespace) -> None:
|
||||
video = Path(args.video).expanduser().resolve()
|
||||
db = db_path_for_video(video)
|
||||
conn = sqlite3.connect(str(db))
|
||||
conn.executescript(SCHEMA)
|
||||
conn.close()
|
||||
print(json.dumps({"status": "ok", "db_path": str(db)}))
|
||||
|
||||
|
||||
def cmd_add_transcription(args: argparse.Namespace) -> None:
|
||||
video = Path(args.video).expanduser().resolve()
|
||||
conn = connect(video)
|
||||
conn.execute(
|
||||
"INSERT INTO transcription (json_path, track_index, created_at, duration_s) VALUES (?, ?, ?, ?)",
|
||||
(args.json_path, args.track_index, time.time(), args.duration),
|
||||
)
|
||||
conn.commit()
|
||||
cur = conn.execute("SELECT last_insert_rowid()")
|
||||
row_id = cur.fetchone()[0]
|
||||
conn.close()
|
||||
print(json.dumps({"status": "ok", "id": row_id}))
|
||||
|
||||
|
||||
def cmd_get_transcription(args: argparse.Namespace) -> None:
|
||||
video = Path(args.video).expanduser().resolve()
|
||||
conn = connect(video)
|
||||
rows = conn.execute("SELECT * FROM transcription ORDER BY created_at DESC").fetchall()
|
||||
conn.close()
|
||||
if not rows:
|
||||
print(json.dumps({"found": False}))
|
||||
return
|
||||
result = [dict(r) for r in rows]
|
||||
print(json.dumps({"found": True, "records": result}, ensure_ascii=False))
|
||||
|
||||
|
||||
def cmd_add_clip(args: argparse.Namespace) -> None:
|
||||
video = Path(args.video).expanduser().resolve()
|
||||
conn = connect(video)
|
||||
cur = conn.execute(
|
||||
"INSERT INTO clips (start_time, end_time, path, created_at, reason) VALUES (?, ?, ?, ?, ?)",
|
||||
(args.start, args.end, args.path, time.time(), args.reason),
|
||||
)
|
||||
conn.commit()
|
||||
clip_id = cur.lastrowid
|
||||
conn.close()
|
||||
print(json.dumps({"status": "ok", "clip_id": clip_id}))
|
||||
|
||||
|
||||
def cmd_add_frames(args: argparse.Namespace) -> None:
|
||||
"""Add frame records for all images in a directory, associating with a clip."""
|
||||
video = Path(args.video).expanduser().resolve()
|
||||
frames_dir = Path(args.frames_dir).expanduser().resolve()
|
||||
if not frames_dir.is_dir():
|
||||
sys.exit(f"Frames directory not found: {frames_dir}")
|
||||
|
||||
conn = connect(video)
|
||||
|
||||
# Read scene scores from showinfo log if available
|
||||
scene_scores: dict[str, float] = {}
|
||||
log_file = frames_dir / "showinfo.log"
|
||||
if log_file.exists():
|
||||
for line in log_file.read_text(errors="replace").splitlines():
|
||||
# Parse lavfi.scd.score and lavfi.scd.time from showinfo output
|
||||
score = None
|
||||
t = None
|
||||
for part in line.split():
|
||||
if part.startswith("score:"):
|
||||
score = float(part.split(":")[1])
|
||||
elif part.startswith("pts_time:"):
|
||||
t = float(part.split(":")[1])
|
||||
if score is not None and t is not None:
|
||||
scene_scores[f"{t:.3f}"] = score
|
||||
|
||||
# Parse timestamps from filenames: frame_XXXX.jpg
|
||||
# The agent should pass --fps and --start to compute timestamps
|
||||
fps = args.fps if args.fps else 0.5
|
||||
start_offset = args.start if args.start else 0.0
|
||||
|
||||
frame_files = sorted(frames_dir.glob("frame_*.jpg"))
|
||||
if not frame_files:
|
||||
# Try other extensions
|
||||
frame_files = sorted(frames_dir.glob("frame_*.png"))
|
||||
|
||||
if not frame_files:
|
||||
sys.exit(f"No frame files found in {frames_dir}")
|
||||
|
||||
count = 0
|
||||
for i, frame_path in enumerate(frame_files):
|
||||
# Estimate timestamp: frame number * (1/fps) + start_offset
|
||||
# This is approximate; the agent can refine with --timestamps
|
||||
timestamp = start_offset + (i / fps)
|
||||
|
||||
# Try to match with scene score
|
||||
scene_score = scene_scores.get(f"{timestamp:.3f}")
|
||||
|
||||
conn.execute(
|
||||
"INSERT INTO frames (clip_id, timestamp, path, scene_score, method, created_at) VALUES (?, ?, ?, ?, ?, ?)",
|
||||
(args.clip_id, timestamp, str(frame_path), scene_score, args.method, time.time()),
|
||||
)
|
||||
count += 1
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
print(json.dumps({"status": "ok", "frames_added": count}))
|
||||
|
||||
|
||||
def cmd_list(args: argparse.Namespace) -> None:
|
||||
video = Path(args.video).expanduser().resolve()
|
||||
conn = connect(video)
|
||||
|
||||
transcriptions = conn.execute("SELECT * FROM transcription ORDER BY created_at DESC").fetchall()
|
||||
clips = conn.execute("SELECT * FROM clips ORDER BY start_time").fetchall()
|
||||
|
||||
result = {"transcriptions": [dict(r) for r in transcriptions], "clips": []}
|
||||
for clip in clips:
|
||||
clip_dict = dict(clip)
|
||||
frames = conn.execute(
|
||||
"SELECT * FROM frames WHERE clip_id = ? ORDER BY timestamp", (clip["id"],)
|
||||
).fetchall()
|
||||
clip_dict["frames"] = [dict(f) for f in frames]
|
||||
result["clips"].append(clip_dict)
|
||||
|
||||
conn.close()
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
def cmd_check_range(args: argparse.Namespace) -> None:
|
||||
"""Check if a time range overlaps with any existing clips."""
|
||||
video = Path(args.video).expanduser().resolve()
|
||||
conn = connect(video)
|
||||
# Find clips that overlap with the requested range
|
||||
rows = conn.execute(
|
||||
"""SELECT * FROM clips
|
||||
WHERE start_time < ? AND end_time > ?
|
||||
ORDER BY start_time""",
|
||||
(args.end, args.start),
|
||||
).fetchall()
|
||||
conn.close()
|
||||
if not rows:
|
||||
print(json.dumps({"found": False}))
|
||||
else:
|
||||
print(json.dumps({"found": True, "clips": [dict(r) for r in rows]}, ensure_ascii=False))
|
||||
|
||||
|
||||
def cmd_clean(args: argparse.Namespace) -> None:
|
||||
"""Remove frame and clip records for files that no longer exist on disk."""
|
||||
video = Path(args.video).expanduser().resolve()
|
||||
conn = connect(video)
|
||||
|
||||
# Check frames
|
||||
frame_rows = conn.execute("SELECT id, path FROM frames").fetchall()
|
||||
frames_removed = 0
|
||||
for row in frame_rows:
|
||||
if not Path(row["path"]).exists():
|
||||
conn.execute("DELETE FROM frames WHERE id = ?", (row["id"],))
|
||||
frames_removed += 1
|
||||
|
||||
# Check clips
|
||||
clip_rows = conn.execute("SELECT id, path FROM clips").fetchall()
|
||||
clips_removed = 0
|
||||
for row in clip_rows:
|
||||
if not Path(row["path"]).exists():
|
||||
conn.execute("DELETE FROM frames WHERE clip_id = ?", (row["id"],))
|
||||
conn.execute("DELETE FROM clips WHERE id = ?", (row["id"],))
|
||||
clips_removed += 1
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
print(json.dumps({"status": "ok", "frames_removed": frames_removed, "clips_removed": clips_removed}))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Manage video-edit-planner index database.")
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
# init
|
||||
p_init = sub.add_parser("init", help="Initialize index database")
|
||||
p_init.add_argument("--video", required=True, help="Path to the video file")
|
||||
|
||||
# add-transcription
|
||||
p_at = sub.add_parser("add-transcription", help="Add a transcription record")
|
||||
p_at.add_argument("--video", required=True)
|
||||
p_at.add_argument("--json-path", required=True)
|
||||
p_at.add_argument("--track-index", required=True, type=int)
|
||||
p_at.add_argument("--duration", type=float, default=None)
|
||||
|
||||
# get-transcription
|
||||
p_gt = sub.add_parser("get-transcription", help="Get latest transcription record")
|
||||
p_gt.add_argument("--video", required=True)
|
||||
|
||||
# add-clip
|
||||
p_ac = sub.add_parser("add-clip", help="Add a clip record")
|
||||
p_ac.add_argument("--video", required=True)
|
||||
p_ac.add_argument("--start", required=True, type=float, help="Start time in seconds")
|
||||
p_ac.add_argument("--end", required=True, type=float, help="End time in seconds")
|
||||
p_ac.add_argument("--path", required=True, help="Path to the clip file")
|
||||
p_ac.add_argument("--reason", default=None)
|
||||
|
||||
# add-frames
|
||||
p_af = sub.add_parser("add-frames", help="Add frame records from a directory")
|
||||
p_af.add_argument("--video", required=True)
|
||||
p_af.add_argument("--clip-id", required=True, type=int)
|
||||
p_af.add_argument("--frames-dir", required=True)
|
||||
p_af.add_argument("--method", required=True, choices=["scene", "sample", "both"])
|
||||
p_af.add_argument("--fps", type=float, default=0.5, help="Sampling fps for timestamp estimation")
|
||||
p_af.add_argument("--start", type=float, default=0.0, help="Start offset in seconds")
|
||||
|
||||
# list
|
||||
p_list = sub.add_parser("list", help="List all records")
|
||||
p_list.add_argument("--video", required=True)
|
||||
|
||||
# check-range
|
||||
p_cr = sub.add_parser("check-range", help="Check if a time range has existing clips")
|
||||
p_cr.add_argument("--video", required=True)
|
||||
p_cr.add_argument("--start", required=True, type=float)
|
||||
p_cr.add_argument("--end", required=True, type=float)
|
||||
|
||||
# clean
|
||||
p_clean = sub.add_parser("clean", help="Remove records for files that no longer exist")
|
||||
p_clean.add_argument("--video", required=True)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
commands = {
|
||||
"init": cmd_init,
|
||||
"add-transcription": cmd_add_transcription,
|
||||
"get-transcription": cmd_get_transcription,
|
||||
"add-clip": cmd_add_clip,
|
||||
"add-frames": cmd_add_frames,
|
||||
"list": cmd_list,
|
||||
"check-range": cmd_check_range,
|
||||
"clean": cmd_clean,
|
||||
}
|
||||
commands[args.command](args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1 @@
|
||||
3.13
|
||||
@@ -0,0 +1,365 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
REGULAR_MODEL_CONFIGS: dict[str, dict[str, Any]] = {
|
||||
"sensevoice": {
|
||||
"model": "iic/SenseVoiceSmall",
|
||||
"vad_model": "fsmn-vad",
|
||||
"vad_kwargs": {"max_single_segment_time": 30000},
|
||||
},
|
||||
"paraformer": {
|
||||
"model": "paraformer-zh",
|
||||
"vad_model": "fsmn-vad",
|
||||
"punc_model": "ct-punc",
|
||||
},
|
||||
"paraformer-en": {
|
||||
"model": "paraformer-en",
|
||||
"vad_model": "fsmn-vad",
|
||||
},
|
||||
}
|
||||
|
||||
NANO_MODEL_CONFIG: dict[str, Any] = {
|
||||
"model": "FunAudioLLM/Fun-ASR-Nano-2512",
|
||||
"vad_model": "fsmn-vad",
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudioTrack:
|
||||
index: int
|
||||
codec: str | None
|
||||
channels: int | None
|
||||
channel_layout: str | None
|
||||
language: str | None
|
||||
title: str | None
|
||||
|
||||
|
||||
def require_command(name: str) -> None:
|
||||
if shutil.which(name) is None:
|
||||
raise SystemExit(f"缺少命令: {name}")
|
||||
|
||||
|
||||
def run_command(cmd: list[str], *, capture: bool = True) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run(
|
||||
cmd,
|
||||
check=True,
|
||||
text=True,
|
||||
stdout=subprocess.PIPE if capture else None,
|
||||
stderr=subprocess.PIPE if capture else None,
|
||||
)
|
||||
|
||||
|
||||
def list_audio_tracks(media_path: Path) -> list[AudioTrack]:
|
||||
require_command("ffprobe")
|
||||
proc = run_command(
|
||||
[
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"error",
|
||||
"-select_streams",
|
||||
"a",
|
||||
"-show_entries",
|
||||
"stream=index,codec_name,channels,channel_layout:stream_tags=language,title",
|
||||
"-of",
|
||||
"json",
|
||||
str(media_path),
|
||||
]
|
||||
)
|
||||
data = json.loads(proc.stdout)
|
||||
tracks: list[AudioTrack] = []
|
||||
for stream in data.get("streams", []):
|
||||
tags = stream.get("tags") or {}
|
||||
tracks.append(
|
||||
AudioTrack(
|
||||
index=int(stream["index"]),
|
||||
codec=stream.get("codec_name"),
|
||||
channels=stream.get("channels"),
|
||||
channel_layout=stream.get("channel_layout"),
|
||||
language=tags.get("language"),
|
||||
title=tags.get("title"),
|
||||
)
|
||||
)
|
||||
return tracks
|
||||
|
||||
|
||||
def print_audio_tracks(media_path: Path) -> None:
|
||||
tracks = list_audio_tracks(media_path)
|
||||
if not tracks:
|
||||
print("未找到音频轨")
|
||||
return
|
||||
for i, track in enumerate(tracks, 1):
|
||||
parts = [
|
||||
f"#{i}",
|
||||
f"stream_index={track.index}",
|
||||
f"codec={track.codec or '?'}",
|
||||
f"channels={track.channels or '?'}",
|
||||
]
|
||||
if track.channel_layout:
|
||||
parts.append(f"layout={track.channel_layout}")
|
||||
if track.language:
|
||||
parts.append(f"lang={track.language}")
|
||||
if track.title:
|
||||
parts.append(f"title={track.title}")
|
||||
print(" ".join(parts))
|
||||
|
||||
|
||||
def resolve_track(media_path: Path, requested: int | None) -> int:
|
||||
tracks = list_audio_tracks(media_path)
|
||||
if not tracks:
|
||||
raise SystemExit(f"未找到音频轨: {media_path}")
|
||||
if requested is None:
|
||||
return tracks[0].index
|
||||
valid_indexes = {track.index for track in tracks}
|
||||
if requested in valid_indexes:
|
||||
return requested
|
||||
raise SystemExit(
|
||||
f"音轨 stream index 不存在: {requested}\n"
|
||||
f"可用音轨: {', '.join(str(t.index) for t in tracks)}\n"
|
||||
"提示: --track 使用 ffprobe/ffmpeg 的 stream index，不是第几条音轨的序号。"
|
||||
)
|
||||
|
||||
|
||||
def extract_audio(media_path: Path, wav_path: Path, track_index: int) -> None:
|
||||
require_command("ffmpeg")
|
||||
wav_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
cmd = [
|
||||
"ffmpeg",
|
||||
"-hide_banner",
|
||||
"-y",
|
||||
"-i",
|
||||
str(media_path),
|
||||
"-map",
|
||||
f"0:{track_index}",
|
||||
"-vn",
|
||||
"-ar",
|
||||
"16000",
|
||||
"-ac",
|
||||
"1",
|
||||
"-c:a",
|
||||
"pcm_s16le",
|
||||
str(wav_path),
|
||||
]
|
||||
subprocess.run(cmd, check=True)
|
||||
|
||||
|
||||
def clean_text(text: str, *, sensevoice: bool) -> str:
|
||||
if sensevoice:
|
||||
try:
|
||||
from funasr.utils.postprocess_utils import rich_transcription_postprocess
|
||||
|
||||
text = rich_transcription_postprocess(text)
|
||||
except Exception:
|
||||
# Keep transcription usable even if FunASR changes this helper.
|
||||
pass
|
||||
return re.sub(r"<\|[^|]*\|>", "", text).strip()
|
||||
|
||||
|
||||
def get_audio_duration(audio_path: Path) -> float | None:
|
||||
try:
|
||||
import soundfile as sf
|
||||
|
||||
return round(float(sf.info(str(audio_path)).duration), 3)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def normalize_result(
|
||||
raw_result: list[dict[str, Any]],
|
||||
*,
|
||||
audio_path: Path,
|
||||
source_media: Path,
|
||||
track_index: int,
|
||||
model_name: str,
|
||||
model_id: str,
|
||||
language: str | None,
|
||||
diarize: bool,
|
||||
elapsed: float,
|
||||
sensevoice: bool,
|
||||
) -> dict[str, Any]:
|
||||
first = raw_result[0] if raw_result else {}
|
||||
text = clean_text(str(first.get("text", "")), sensevoice=sensevoice)
|
||||
segments: list[dict[str, Any]] = []
|
||||
for seg in first.get("sentence_info") or []:
|
||||
item: dict[str, Any] = {
|
||||
"start": seg.get("start", 0),
|
||||
"end": seg.get("end", 0),
|
||||
"text": clean_text(str(seg.get("sentence") or seg.get("text") or ""), sensevoice=sensevoice),
|
||||
}
|
||||
if diarize and "spk" in seg:
|
||||
item["speaker"] = seg["spk"]
|
||||
segments.append(item)
|
||||
|
||||
output: dict[str, Any] = {
|
||||
"text": text,
|
||||
"segments": segments,
|
||||
"file": source_media.name,
|
||||
"audio_file": audio_path.name,
|
||||
"track": track_index,
|
||||
"model": model_name,
|
||||
"model_id": model_id,
|
||||
"language": language or "auto",
|
||||
"audio_duration_s": get_audio_duration(audio_path),
|
||||
"processing_s": round(elapsed, 3),
|
||||
}
|
||||
for key in ("timestamps", "timestamp"):
|
||||
if key in first:
|
||||
output[key] = first[key]
|
||||
return output
|
||||
|
||||
|
||||
def write_json(path: Path, data: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
|
||||
def make_output_path(output_dir: Path, media_path: Path, track_index: int, suffix: str) -> Path:
|
||||
return output_dir / f"{media_path.stem}_track{track_index}_{suffix}.json"
|
||||
|
||||
|
||||
def resolve_output_dir(output_dir: Path | None, media_path: Path) -> Path:
|
||||
if output_dir is None:
|
||||
return media_path.parent
|
||||
return output_dir.expanduser().resolve()
|
||||
|
||||
|
||||
def add_common_args(parser: argparse.ArgumentParser) -> None:
|
||||
parser.add_argument("media", type=Path, help="视频/音频文件路径")
|
||||
parser.add_argument("--track", type=int, default=None, help="ffmpeg stream index；默认第一条音频轨")
|
||||
parser.add_argument("--list-tracks", action="store_true", help="列出音轨后退出")
|
||||
parser.add_argument("--output-dir", "-o", type=Path, default=None, help="输出目录；默认源视频/音频文件所在目录")
|
||||
parser.add_argument("--language", "-l", default="auto", help="语言，如 auto/zh/en/ja/ko/yue；默认 auto")
|
||||
parser.add_argument("--device", default=None, help="设备，如 cuda:0/cpu；默认自动")
|
||||
parser.add_argument("--no-diarize", action="store_true", help="禁用 cam++ 说话人分离")
|
||||
parser.add_argument("--keep-wav", action="store_true", help="保留提取出的 16k wav 文件")
|
||||
parser.add_argument("--wav-dir", type=Path, default=None, help="临时 wav 输出目录；默认系统临时目录")
|
||||
parser.add_argument("--verbose", "-v", action="store_true", help="输出更多进度信息")
|
||||
|
||||
|
||||
def prepare_audio(args: argparse.Namespace) -> tuple[Path, int, tempfile.TemporaryDirectory[str] | None]:
|
||||
media_path: Path = args.media.expanduser().resolve()
|
||||
if not media_path.exists():
|
||||
raise SystemExit(f"文件不存在: {media_path}")
|
||||
|
||||
if args.list_tracks:
|
||||
print_audio_tracks(media_path)
|
||||
raise SystemExit(0)
|
||||
|
||||
track_index = resolve_track(media_path, args.track)
|
||||
temp_dir: tempfile.TemporaryDirectory[str] | None = None
|
||||
if args.keep_wav:
|
||||
wav_dir = args.wav_dir.expanduser().resolve() if args.wav_dir else resolve_output_dir(args.output_dir, media_path)
|
||||
wav_dir.mkdir(parents=True, exist_ok=True)
|
||||
elif args.wav_dir is not None:
|
||||
wav_dir = args.wav_dir.expanduser().resolve()
|
||||
wav_dir.mkdir(parents=True, exist_ok=True)
|
||||
else:
|
||||
temp_dir = tempfile.TemporaryDirectory(prefix="funasr-script-")
|
||||
wav_dir = Path(temp_dir.name)
|
||||
|
||||
wav_path = wav_dir / f"{media_path.stem}_track{track_index}.wav"
|
||||
print(f"[1/2] 提取音轨 stream_index={track_index} -> {wav_path}", file=sys.stderr)
|
||||
extract_audio(media_path, wav_path, track_index)
|
||||
return wav_path, track_index, temp_dir
|
||||
|
||||
|
||||
def auto_device(device: str | None) -> str:
|
||||
if device:
|
||||
return device
|
||||
import torch
|
||||
|
||||
return "cuda:0" if torch.cuda.is_available() else "cpu"
|
||||
|
||||
|
||||
def transcribe_with_config(
|
||||
*,
|
||||
wav_path: Path,
|
||||
media_path: Path,
|
||||
track_index: int,
|
||||
output_path: Path,
|
||||
model_name: str,
|
||||
config: dict[str, Any],
|
||||
language: str | None,
|
||||
device: str | None,
|
||||
diarize: bool,
|
||||
use_itn: bool,
|
||||
sensevoice: bool,
|
||||
verbose: bool,
|
||||
) -> dict[str, Any]:
|
||||
from funasr import AutoModel
|
||||
|
||||
config = config.copy()
|
||||
if diarize and "spk_model" not in config:
|
||||
config["spk_model"] = "cam++"
|
||||
resolved_device = auto_device(device)
|
||||
|
||||
print(f"[2/2] 加载模型 {model_name} ({config['model']}) on {resolved_device}", file=sys.stderr)
|
||||
load_start = time.time()
|
||||
model = AutoModel(device=resolved_device, disable_update=True, **config)
|
||||
if verbose:
|
||||
print(f"模型加载耗时: {time.time() - load_start:.1f}s", file=sys.stderr)
|
||||
|
||||
gen_kw: dict[str, Any] = {"input": str(wav_path), "batch_size": 1}
|
||||
if language and language != "auto":
|
||||
gen_kw["language"] = language
|
||||
if use_itn:
|
||||
gen_kw["use_itn"] = True
|
||||
|
||||
start = time.time()
|
||||
raw_result = model.generate(**gen_kw)
|
||||
elapsed = time.time() - start
|
||||
output = normalize_result(
|
||||
raw_result,
|
||||
audio_path=wav_path,
|
||||
source_media=media_path,
|
||||
track_index=track_index,
|
||||
model_name=model_name,
|
||||
model_id=str(config["model"]),
|
||||
language=language,
|
||||
diarize=diarize,
|
||||
elapsed=elapsed,
|
||||
sensevoice=sensevoice,
|
||||
)
|
||||
write_json(output_path, output)
|
||||
print(f"完成: {output_path}", file=sys.stderr)
|
||||
print(f"转录耗时: {elapsed:.2f}s", file=sys.stderr)
|
||||
return output
|
||||
|
||||
|
||||
def run_pipeline(args: argparse.Namespace, *, model_name: str, config: dict[str, Any], suffix: str, use_itn: bool, sensevoice: bool) -> Path:
|
||||
media_path = args.media.expanduser().resolve()
|
||||
temp_dir: tempfile.TemporaryDirectory[str] | None = None
|
||||
try:
|
||||
wav_path, track_index, temp_dir = prepare_audio(args)
|
||||
output_path = make_output_path(resolve_output_dir(args.output_dir, media_path), media_path, track_index, suffix)
|
||||
transcribe_with_config(
|
||||
wav_path=wav_path,
|
||||
media_path=media_path,
|
||||
track_index=track_index,
|
||||
output_path=output_path,
|
||||
model_name=model_name,
|
||||
config=config,
|
||||
language=args.language,
|
||||
device=args.device,
|
||||
diarize=not args.no_diarize,
|
||||
use_itn=use_itn,
|
||||
sensevoice=sensevoice,
|
||||
verbose=args.verbose,
|
||||
)
|
||||
print(output_path)
|
||||
return output_path
|
||||
finally:
|
||||
if temp_dir is not None:
|
||||
temp_dir.cleanup()
|
||||
@@ -0,0 +1,30 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
|
||||
from funasr_common import REGULAR_MODEL_CONFIGS, add_common_args, run_pipeline
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="funasr-fast",
|
||||
description="提取指定音轨并用 SenseVoice 快速转录。",
|
||||
)
|
||||
add_common_args(parser)
|
||||
return parser
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = build_parser().parse_args()
|
||||
run_pipeline(
|
||||
args,
|
||||
model_name="sensevoice",
|
||||
config=REGULAR_MODEL_CONFIGS["sensevoice"],
|
||||
suffix="sensevoice",
|
||||
use_itn=True,
|
||||
sensevoice=True,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,30 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
|
||||
from funasr_common import NANO_MODEL_CONFIG, add_common_args, run_pipeline
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="funasr-nano",
|
||||
description="提取指定音轨并用 Fun-ASR-Nano-2512 转录。",
|
||||
)
|
||||
add_common_args(parser)
|
||||
return parser
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = build_parser().parse_args()
|
||||
run_pipeline(
|
||||
args,
|
||||
model_name="fun-asr-nano",
|
||||
config=NANO_MODEL_CONFIG,
|
||||
suffix="fun-asr-nano",
|
||||
use_itn=True,
|
||||
sensevoice=False,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,37 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
|
||||
from funasr_common import REGULAR_MODEL_CONFIGS, add_common_args, run_pipeline
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="funasr-regular",
|
||||
description="提取指定音轨并用常规 FunASR 管线转录（SenseVoice/Paraformer）。",
|
||||
)
|
||||
add_common_args(parser)
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
choices=sorted(REGULAR_MODEL_CONFIGS),
|
||||
default="sensevoice",
|
||||
help="常规 FunASR 模型，默认 sensevoice",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = build_parser().parse_args()
|
||||
config = REGULAR_MODEL_CONFIGS[args.model]
|
||||
run_pipeline(
|
||||
args,
|
||||
model_name=args.model,
|
||||
config=config,
|
||||
suffix=args.model,
|
||||
use_itn=args.model == "sensevoice",
|
||||
sensevoice=args.model == "sensevoice",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from funasr_nano import main
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,25 @@
|
||||
[project]
|
||||
name = "funasr-script"
|
||||
version = "0.1.0"
|
||||
description = "Small local FunASR transcription scripts for extracting a video audio track and writing JSON transcripts."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.13"
|
||||
dependencies = [
|
||||
"funasr>=1.3.14",
|
||||
"soundfile>=0.13.1",
|
||||
"torch>=2.13.0",
|
||||
"torchaudio>=2.11.0",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
funasr-script = "funasr_nano:main"
|
||||
funasr-nano = "funasr_nano:main"
|
||||
funasr-fast = "funasr_fast:main"
|
||||
funasr-regular = "funasr_regular:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["setuptools>=69"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[tool.setuptools]
|
||||
py-modules = ["funasr_common", "funasr_regular", "funasr_fast", "funasr_nano", "main"]
|
||||
Generated
+1906
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user