feat: reconcile speaker identities through OpenRouter audio

Short chunk overlaps often contain only one participant, so returning
speakers receive new transcript IDs. MAI can also split one voice into
several IDs within a single request, which overlap stitching cannot fix.

Add an opt-in audio comparison pass: --speakers N selects Gemini 3.8 Flash
with a known count, and --speaker-model allows another compatible model.
Keep request-local identities separate, select up to three clean excerpts
per ID, and compare all excerpts in one OpenRouter request. Gemini uses
Google AI Studio because the Vertex backend limits prompts to one audio
file. No local model or extra Python dependency is required.

Only reconcile identities whose sample assignments agree. Preserve
uncertain identities separately, reject incomplete or malformed replies,
and retain the original words and timestamps. Bound sample count and
request size, clip sampling to the real audio duration, and extend worker
cleanup and buffered output to the comparison step. Document the sampled
method's limitations and additional provider request.

Test Plan:
- python3 -B -m unittest -v: all 82 tests passed, including real MP3
  encoding with fake transcription/comparison responses and cancellation.
- bash -n transcribe: passed.
- git diff --cached --check: passed.
- Live API and voice accuracy unverified: this workspace could not access
  the configured 1Password credential. Mocked responses prove pipeline
  behavior, not model accuracy on the supplied recording.
This commit is contained in:
ddidderr committed 2026-09-12 10:04:56 +02:00
1 parent 1d03dcd587
commit 138c796399
9 files changed
+988 -19

No files matched your search

+52 -4
View File
@@ -1,8 +1,9 @@
# Transcribe
Requires Bash 5.1 or newer, FFmpeg, curl, jq, Python 3, and standard coreutils. Keep
`chunks.py`, `subtitles.py`, and `transcript.py` beside the script. The API key is read using
the 1Password CLI (`op`) from `op://API Keys/openrouter-mai-transcribe-2/credential`.
`chunks.py`, `subtitles.py`, `transcript.py`, and `diarization.py` beside the script.
The API key is read using the 1Password CLI (`op`) from
`op://API Keys/openrouter-mai-transcribe-2/credential`.
```sh
./transcribe recording.m4a
@@ -13,6 +14,8 @@ the 1Password CLI (`op`) from `op://API Keys/openrouter-mai-transcribe-2/credent
./transcribe --chunk-minutes 10 too_long.webm > transcript.txt
./transcribe --jobs 4 too_long.webm > transcript.txt
./transcribe --jobs 1 --srt too_long.webm > subtitles.srt
./transcribe --speakers 2 too_long.webm > transcript.txt
./transcribe --speaker-model google/gemini-3.8-flash recording.m4a > transcript.txt
```
Accepts m4a, mp4, aac, flac, opus, mp3, ogg, wav, and other formats your
@@ -43,7 +46,7 @@ without punctuation. The formatter preserves the words and their order.
Unlabelled passages use `[Speaker unknown]` when other words have labels; a
response without any labels is printed without speaker markers. A warning on
stderr reports missing or incomplete labels when diarization was requested.
Speaker labels use the provider's diarization; they are not names inferred from text.
Speaker labels use audio evidence; they are not names inferred from text.
SRT uses labels to break cues at speaker changes but does not print the IDs.
Long recordings are automatically split into uploads of at most **10 minutes**
@@ -88,6 +91,50 @@ recording: the same person can receive another ID when absent from an overlap
or when diarization disagrees. Numeric provider IDs alone are never used to
identify a speaker across requests.
For interviews and other recordings with fragmented speaker IDs, use
`--speakers N` (1 to 64) to add a voice comparison pass through OpenRouter.
This selects `google/gemini-3.8-flash` and tells it the known number of people.
Use `--speaker-model MODEL` to select a different audio chat model supporting
multiple audio inputs and JSON schema output; without `--speakers`, the model
estimates the number of voices. Both options require
diarization and cannot be combined with `--no-diarization`. The normal command
without either option retains the overlap-based behavior described above.
The comparison keeps MAI's words and timestamps, initially treating IDs from
different requests as separate. It takes up to three nonoverlapping excerpts
per provisional speaker (1.5 to 8 seconds each) from uninterrupted labelled
speech and sends them together in **one additional audio request**. Comparing
voices in a shared context can reunite a person absent from a chunk overlap
and merge IDs that MAI split within a single request. The model receives the
voice excerpts and sample numbers, without the written transcript or names.
This uses [OpenRouter audio chat](https://openrouter.ai/docs/guides/overview/multimodal/audio)
and [Gemini's documented audio diarization capability](https://ai.google.dev/gemini-api/docs/audio),
with no local model, new Python package, or separate API key.
Gemini requests use OpenRouter's Google AI Studio provider, whose API supports
[multiple audio inputs](https://ai.google.dev/gemini-api/docs/generate-content/audio#technical-details-about-audio),
with provider fallback disabled. The Vertex backend
[documents a one-audio-file limit](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/capabilities/audio-understanding).
Every returned sample assignment must be present exactly once. Excerpts for
the same provisional speaker must agree on one non-null model ID before it
can be merged with others. Speakers without suitable excerpts, uncertain
assignments, or conflicting voices retain separate IDs and produce warnings;
a known count does **not** force uncertain speakers into that many output labels.
If no suitable excerpts
exist, comparison is skipped with a warning. Malformed, incomplete, truncated,
or failed API responses fail the command without partial stdout.
This is sampled voice comparison, not a fresh segmentation of every word.
It cannot repair a MAI label that incorrectly combines several people into one
turn, and consistent model mistakes can still misidentify a voice. A single
usable excerpt has weaker evidence than several. Review brief interjections,
crosstalk, and edited intros in particular. Word timing and spoken text are
never generated by the comparison model. Requests are bounded to 256 excerpts
and 20 MB of encoded JSON; exceeding a bound fails rather than silently omitting
voices. The extra request adds processing time and provider charges. Its actual
voice-matching accuracy must be checked on real audio; fake-provider tests only
verify the pipeline and its validation rules.
Audio is first decoded to temporary mono 16 kHz PCM (about 115 MB per hour),
so all cuts share the same sample clock even when container duration metadata
is missing. Up to **eight chunks** are encoded and transcribed concurrently
@@ -103,7 +150,8 @@ cancels the remaining workers. Interruptions also stop workers and their child
processes before temporary files are removed. Requests already accepted by the
provider may still be processed and billed. There is no automatic retry or
resume, and rerunning sends earlier chunks again. No transcript is emitted
unless every worker and the final conversion succeed.
unless every worker, any requested voice comparison, and the final conversion
succeed.
`--srt` requests word timestamps using OpenRouter's `verbose_json` response.
Words are grouped into cues of at most two lines (42 characters per line,