105 lines
4.5 KiB
Markdown
105 lines
4.5 KiB
Markdown
# Fun-ASR-Nano rolling context / `prev_text` notes
|
|
|
|
Use this when deciding whether Fun-ASR-Nano can use already-transcribed context from the same audio file to improve later sentence recognition.
|
|
|
|
## User intent
|
|
|
|
This is **not** about manually adding fixed hotwords. The desired behavior is closer to rolling context or 2-pass-style refinement:
|
|
|
|
- While processing one independent audio/video file, the model should use text already recognized from earlier audio as context for later audio.
|
|
- When a current sentence is ambiguous because of unclear pronunciation, it may benefit from preceding vocabulary, phrasing, topic, and dialogue context.
|
|
- This is analogous to streaming ASR systems where early partial text may be wrong, and a later stabilized/final pass revises the sentence with more context.
|
|
|
|
## Verified upstream behavior
|
|
|
|
Sources checked:
|
|
|
|
- `modelscope/FunASR` `funasr/models/fun_asr_nano/model.py`
|
|
- `modelscope/FunASR` `examples/industrial_data_pretraining/fun_asr_nano/demo2.py`
|
|
- `modelscope/FunASR` `funasr/models/fun_asr_nano/inference_vllm_streaming.py`
|
|
- `modelscope/FunASR` `docs/vllm_guide.md`
|
|
- `modelscope/FunASR` `docs/vllm_guide_zh.md`
|
|
|
|
Fun-ASR-Nano supports a `prev_text` mechanism in the model inference path. In `model.py`, `prev_text` is appended into the assistant-side prompt prefix before decoding:
|
|
|
|
```python
|
|
if kwargs.get("prev_text", None) is not None:
|
|
source_input += kwargs["prev_text"]
|
|
```
|
|
|
|
The official `demo2.py` demonstrates cumulative chunk transcription:
|
|
|
|
```python
|
|
prev_text = ""
|
|
for idx, cum_duration in enumerate(cum_durations):
|
|
audio, rate = load_audio(wav_path, 16000, duration=round(cum_duration, 3))
|
|
prev_text = m.inference([torch.tensor(audio)], prev_text=prev_text, **kwargs)[0][0]["text"]
|
|
if idx != len(cum_durations) - 1:
|
|
prev_text = tokenizer.decode(tokenizer.encode(prev_text)[:-5]).replace("�", "")
|
|
```
|
|
|
|
The streaming vLLM code also uses a two-stage design:
|
|
|
|
- Stage 1: generate early chunks without `prev_text` to find stable output.
|
|
- Stage 2: use stable output as `prev_text` for later cumulative chunks.
|
|
- It keeps a rollback/unfixed tail, so the last few characters may change as more audio arrives.
|
|
|
|
The vLLM guide explicitly describes this as:
|
|
|
|
- first chunks: no `prev_text`
|
|
- subsequent chunks: use stable output as `prev_text`
|
|
|
|
## What the current bundled script does
|
|
|
|
The current `scripts/transcription/funasr_common.py` path calls ordinary offline `AutoModel.generate(input=..., batch_size=1)` on the extracted WAV file.
|
|
|
|
It currently passes:
|
|
|
|
- `input`
|
|
- `batch_size=1`
|
|
- explicit `language` when non-`auto`
|
|
- `use_itn=True` for Fun-ASR-Nano
|
|
|
|
It does **not** currently implement a chunked/cumulative pass that feeds prior recognized text back through `prev_text`.
|
|
|
|
Therefore:
|
|
|
|
- Fun-ASR-Nano has model-level support for rolling text context.
|
|
- Upstream examples show this mechanism in streaming/incremental usage.
|
|
- But the current bundled offline script should **not** be described as automatically enabling whole-file rolling context correction.
|
|
|
|
## Practical implication for this skill
|
|
|
|
When accuracy matters, especially for unclear speech, homophones, game terms, and dialogue continuity, do not assume the default offline transcript is the best possible use of Nano's context mechanism.
|
|
|
|
The skill should describe this honestly:
|
|
|
|
1. Current default offline transcription is a strong baseline.
|
|
2. It may still produce wrong words such as `帐篷` -> `账本` when source speech is unclear.
|
|
3. Fun-ASR-Nano has `prev_text`/rolling-context mechanisms upstream, but the bundled script does not yet expose a stable offline mode for that.
|
|
4. A future enhancement can evaluate a chunked/cumulative or 2-pass-like transcription mode that feeds stabilized earlier text as `prev_text`.
|
|
5. Such a mode should be benchmarked before becoming default, because cumulative re-encoding may be slower and may have different timestamp/segmentation behavior.
|
|
|
|
## Future enhancement idea
|
|
|
|
A possible experimental mode:
|
|
|
|
```bash
|
|
funasr-nano VIDEO_PATH \
|
|
--track TRACK_INDEX \
|
|
--language AUDIO_LANGUAGE_CODE \
|
|
--context-mode rolling \
|
|
--rollback-chars 5 \
|
|
--output-dir VIDEO_DIR
|
|
```
|
|
|
|
Implementation direction:
|
|
|
|
- Split or cumulatively extend audio in chunks.
|
|
- Keep a stable prefix and an unfixed tail, similar to upstream `demo2.py` / streaming vLLM logic.
|
|
- Feed the stable prefix into `prev_text` for the next pass.
|
|
- Preserve a normal offline mode because it is simpler, faster, and already works.
|
|
- Compare quality on real user samples before enabling by default.
|
|
|
|
This is a transcription-mode experiment, not a hotword-list feature.
|