From acfbd6d72a6b03c6583a809d44134b4263da4214 Mon Sep 17 00:00:00 2001 From: nite Date: Tue, 14 Jul 2026 17:18:17 +0800 Subject: [PATCH] docs: clarify FunASR rolling context behavior --- SKILL.md | 2 +- references/funasr-nano-context-hotwords.md | 80 ---------------- references/funasr-nano-rolling-context.md | 104 +++++++++++++++++++++ 3 files changed, 105 insertions(+), 81 deletions(-) delete mode 100644 references/funasr-nano-context-hotwords.md create mode 100644 references/funasr-nano-rolling-context.md diff --git a/SKILL.md b/SKILL.md index 9a6d2a0..f075b82 100644 --- a/SKILL.md +++ b/SKILL.md @@ -132,7 +132,7 @@ uv run --directory SKILL_DIR/scripts/transcription funasr-nano VIDEO_PATH --list Where `VIDEO_DIR` is the directory containing the video file, `SKILL_DIR` is the skill installation directory, and `AUDIO_LANGUAGE_CODE` is the explicit transcription language (for example `zh` for Mandarin Chinese). -**Context and hotwords:** Fun-ASR-Nano is an LLM-based ASR model and upstream code supports prompt-style `language` and `hotwords` hints. However, do not assume the current bundled script automatically performs cross-segment semantic correction: it currently passes `language` / `use_itn`, but does not pass external hotwords or rolling `prev_text` context between VAD segments. If the video contains known game terms, names, locations, item names, or other jargon, ask the user for a hotword list or derive candidate terms and confirm them before transcription. See `references/funasr-nano-context-hotwords.md`. Hotwords are biasing hints, not a substitute for final manual checking. +**Rolling context / semantic correction:** Fun-ASR-Nano has upstream support for `prev_text`-style rolling context in streaming/incremental examples: earlier stabilized text can be fed back as context for later decoding, similar to a 2-pass-style refinement. However, do not assume the current bundled offline script automatically enables whole-file rolling semantic correction: it currently calls ordinary `AutoModel.generate(...)` on the extracted WAV and does not run a cumulative chunked pass with `prev_text`. See `references/funasr-nano-rolling-context.md`. Treat the current transcript as a strong baseline, not the final word when unclear speech or homophones matter. **Common OBS multi-track audio layouts.** When the user says "multi-track" or mentions OBS recording, these are the typical stream layouts: diff --git a/references/funasr-nano-context-hotwords.md b/references/funasr-nano-context-hotwords.md deleted file mode 100644 index 82ad4b5..0000000 --- a/references/funasr-nano-context-hotwords.md +++ /dev/null @@ -1,80 +0,0 @@ -# Fun-ASR-Nano context, hotwords, and transcript correction notes - -Use this when deciding whether Fun-ASR-Nano can use semantic context to improve recognition quality. - -## Verified from upstream code/docs - -Sources checked: - -- `modelscope/FunASR` `funasr/models/fun_asr_nano/model.py` -- `modelscope/FunASR` `funasr/models/fun_asr_nano/inference_vllm.py` -- `modelscope/FunASR` `funasr/models/fun_asr_nano/inference_vllm_pipeline.py` -- `modelscope/FunASR` `funasr/models/fun_asr_nano/inference_vllm_streaming.py` -- `modelscope/FunASR` `docs/vllm_guide_zh.md` -- `modelscope/FunASR` `docs/model_selection.md` - -The upstream model is LLM-based: a SenseVoice-style audio encoder plus an LLM decoder. The upstream model-selection guide describes Fun-ASR-Nano as stronger on hard cases, context, and proper nouns. - -The PyTorch path has a prompt builder roughly shaped as: - -```python -def get_prompt(self, hotwords: list[str], language: str = None, itn: bool = True): - if len(hotwords) > 0: - prompt = "请结合上下文信息，更加准确地完成语音转写任务。如果没有相关信息，我们会留空。\n\n\n**上下文信息：**\n\n\n" - prompt += f"热词列表：[{hotwords}]\n" - else: - prompt = "" - if language is None: - prompt += "语音转写" - else: - prompt += f"语音转写成{language}" - if not itn: - prompt += "，不进行文本规整" - return prompt + "：" -``` - -The vLLM path exposes the same idea via `hotwords` and `language`. The vLLM guide documents `language` and `hotwords` as `generate()` / API parameters. - -## What is enabled by default? - -Do **not** assume cross-segment semantic correction is automatically enabled for offline file transcription. - -In the bundled transcription script at the time of writing: - -- `funasr_common.py` calls `model.generate(input=..., batch_size=1)`. -- It passes `language` only when the user specified a non-`auto` language. -- It passes `use_itn=True` for Fun-ASR-Nano. -- It does **not** pass `hotwords`. -- It does **not** pass `prev_text` or a rolling context between VAD segments. - -Therefore, the current script uses Nano's model-internal contextual ability, language hint, and ITN, but it does not provide external domain context or previous transcript text. The transcript may still contain homophone/near-homophone mistakes such as `帐篷` -> `账本` when the source audio is unclear. - -## Practical guidance - -1. Use explicit `--language AUDIO_LANGUAGE_CODE` to avoid language ambiguity. -2. If the content has known names, game terms, locations, item names, or recurring jargon, collect them as a hotword list before transcription. -3. Prefer adding explicit script support such as `--hotword TERM` / `--hotwords-file FILE` over relying on vague “semantic correction”. -4. Treat hotwords as biasing hints, not proof. They can improve proper nouns and domain terms but cannot fully fix unclear audio. -5. For final captions, quotes, subtitles, narration, or published copy, still re-listen and manually correct the original audio segment. -6. Do not silently invent hotwords. Ask the user for domain terms or extract candidate terms from existing project notes/transcripts and confirm them. - -## Possible future script enhancement - -Expose hotwords in the bundled CLI: - -```bash -funasr-nano VIDEO_PATH \ - --track TRACK_INDEX \ - --language AUDIO_LANGUAGE_CODE \ - --hotword "帐篷" \ - --hotword "复活点" \ - --output-dir VIDEO_DIR -``` - -Then pass them to FunASR: - -```python -gen_kw["hotwords"] = args.hotwords -``` - -For longer lists, support a UTF-8 text file with one hotword per line. diff --git a/references/funasr-nano-rolling-context.md b/references/funasr-nano-rolling-context.md new file mode 100644 index 0000000..a4d15b0 --- /dev/null +++ b/references/funasr-nano-rolling-context.md @@ -0,0 +1,104 @@ +# Fun-ASR-Nano rolling context / `prev_text` notes + +Use this when deciding whether Fun-ASR-Nano can use already-transcribed context from the same audio file to improve later sentence recognition. + +## User intent + +This is **not** about manually adding fixed hotwords. The desired behavior is closer to rolling context or 2-pass-style refinement: + +- While processing one independent audio/video file, the model should use text already recognized from earlier audio as context for later audio. +- When a current sentence is ambiguous because of unclear pronunciation, it may benefit from preceding vocabulary, phrasing, topic, and dialogue context. +- This is analogous to streaming ASR systems where early partial text may be wrong, and a later stabilized/final pass revises the sentence with more context. + +## Verified upstream behavior + +Sources checked: + +- `modelscope/FunASR` `funasr/models/fun_asr_nano/model.py` +- `modelscope/FunASR` `examples/industrial_data_pretraining/fun_asr_nano/demo2.py` +- `modelscope/FunASR` `funasr/models/fun_asr_nano/inference_vllm_streaming.py` +- `modelscope/FunASR` `docs/vllm_guide.md` +- `modelscope/FunASR` `docs/vllm_guide_zh.md` + +Fun-ASR-Nano supports a `prev_text` mechanism in the model inference path. In `model.py`, `prev_text` is appended into the assistant-side prompt prefix before decoding: + +```python +if kwargs.get("prev_text", None) is not None: + source_input += kwargs["prev_text"] +``` + +The official `demo2.py` demonstrates cumulative chunk transcription: + +```python +prev_text = "" +for idx, cum_duration in enumerate(cum_durations): + audio, rate = load_audio(wav_path, 16000, duration=round(cum_duration, 3)) + prev_text = m.inference([torch.tensor(audio)], prev_text=prev_text, **kwargs)[0][0]["text"] + if idx != len(cum_durations) - 1: + prev_text = tokenizer.decode(tokenizer.encode(prev_text)[:-5]).replace("�", "") +``` + +The streaming vLLM code also uses a two-stage design: + +- Stage 1: generate early chunks without `prev_text` to find stable output. +- Stage 2: use stable output as `prev_text` for later cumulative chunks. +- It keeps a rollback/unfixed tail, so the last few characters may change as more audio arrives. + +The vLLM guide explicitly describes this as: + +- first chunks: no `prev_text` +- subsequent chunks: use stable output as `prev_text` + +## What the current bundled script does + +The current `scripts/transcription/funasr_common.py` path calls ordinary offline `AutoModel.generate(input=..., batch_size=1)` on the extracted WAV file. + +It currently passes: + +- `input` +- `batch_size=1` +- explicit `language` when non-`auto` +- `use_itn=True` for Fun-ASR-Nano + +It does **not** currently implement a chunked/cumulative pass that feeds prior recognized text back through `prev_text`. + +Therefore: + +- Fun-ASR-Nano has model-level support for rolling text context. +- Upstream examples show this mechanism in streaming/incremental usage. +- But the current bundled offline script should **not** be described as automatically enabling whole-file rolling context correction. + +## Practical implication for this skill + +When accuracy matters, especially for unclear speech, homophones, game terms, and dialogue continuity, do not assume the default offline transcript is the best possible use of Nano's context mechanism. + +The skill should describe this honestly: + +1. Current default offline transcription is a strong baseline. +2. It may still produce wrong words such as `帐篷` -> `账本` when source speech is unclear. +3. Fun-ASR-Nano has `prev_text`/rolling-context mechanisms upstream, but the bundled script does not yet expose a stable offline mode for that. +4. A future enhancement can evaluate a chunked/cumulative or 2-pass-like transcription mode that feeds stabilized earlier text as `prev_text`. +5. Such a mode should be benchmarked before becoming default, because cumulative re-encoding may be slower and may have different timestamp/segmentation behavior. + +## Future enhancement idea + +A possible experimental mode: + +```bash +funasr-nano VIDEO_PATH \ + --track TRACK_INDEX \ + --language AUDIO_LANGUAGE_CODE \ + --context-mode rolling \ + --rollback-chars 5 \ + --output-dir VIDEO_DIR +``` + +Implementation direction: + +- Split or cumulatively extend audio in chunks. +- Keep a stable prefix and an unfixed tail, similar to upstream `demo2.py` / streaming vLLM logic. +- Feed the stable prefix into `prev_text` for the next pass. +- Preserve a normal offline mode because it is simpler, faster, and already works. +- Compare quality on real user samples before enabling by default. + +This is a transcription-mode experiment, not a hotword-list feature.