ddidderr/transcribe
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat!: use whole-recording Pyannote speaker diarization

Browse Source 
Sampled Gemini voice matching could leave one person under separate IDs
across transcription chunks. Use hosted Precision-2 on the complete
recording and align its exclusive speaker timeline with MAI word timing.
Preserve transcription text, order, and timestamps; leave ambiguous word
assignments unknown instead of merging identities to satisfy a count.

Upload lossless FLAC, bound polling, retry transient status reads against
the same job, and validate results before emitting stdout. Keep optional
speaker-count and hosted model overrides. Disable MAI diarization and use
30-minute transcription uploads in both modes. Respect environment keys,
keep authentication out of process arguments, and save optional diagnostics
without audio, credentials, or signed upload URLs.

Test Plan:
- python3 -B -m unittest -v: all 101 regression tests passed, including
  mocked provider upload, polling, alignment, failure, and cleanup paths.
- bash -n transcribe and shellcheck transcribe: passed.
- git diff --cached --check: passed.
- Prepared the 74-minute input locally and verified identical decoded PCM
  hashes for the normalized WAV and full-recording lossless FLAC.
- Live API testing was explicitly deferred by the user; mocked results do
  not establish speaker accuracy on the recording.

BREAKING CHANGE: Default diarization requires PYANNOTE_API_KEY or its
1Password reference. --speaker-model accepts hosted precision-2 or
community-1 instead of OpenRouter audio-chat model IDs.
This commit is contained in:
ddidderr committed 2026-09-12 11:18:30 +02:00
1 parent d7827d39f5
commit 3042b7743b
7 files changed
+1336 -864
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+91 -68
View File
@@ -10,8 +10,16 @@ Use `--srt` for subtitles or `--no-diarization` for text without speaker labels.
Requires Bash 5.1 or newer, FFmpeg, curl, jq, Python 3, and standard coreutils. Keep
`chunks.py`, `subtitles.py`, `transcript.py`, and `diarization.py` beside the script.
The API key is read using the 1Password CLI (`op`) from
`op://API Keys/openrouter-mai-transcribe-2/credential`.
No local model or additional Python package is required.
Set `OPENROUTER_API_KEY` for transcription and `PYANNOTE_API_KEY` for speaker
diarization. Existing environment values are respected. When `OPENROUTER_API_KEY`
is unset, the script reads its key with the 1Password CLI (`op`) using
`OPENROUTER_API_KEY_REF`, which defaults to
`op://API Keys/openrouter-mai-transcribe-2/credential`. For Pyannote, either set
`PYANNOTE_API_KEY` directly or set `PYANNOTE_API_KEY_REF` to your own 1Password
reference. There is no default Pyannote reference. `op` is required only when
resolving a reference. `--no-diarization` needs only the OpenRouter key.
Accepts m4a, mp4, aac, flac, opus, mp3, ogg, wav, and other formats your
FFmpeg supports. Uses the first audio track, converts it to mono 16 kHz MP3
@@ -20,14 +28,15 @@ and sends it to OpenRouter's `microsoft/mai-transcribe-2` model. Temporary
files are removed on exit. The transcript goes to stdout; progress and errors
go to stderr. Output is emitted only after all requests and conversion succeed.
Speaker diarization and voice comparison are **enabled by default**.
MAI first labels speech through
`provider.options.azure.diarization.enabled`, following
[OpenRouter's speaker diarization API](https://openrouter.ai/docs/guides/overview/multimodal/stt#verbose-transcripts-timestamps-and-speakers).
Gemini then compares voice excerpts through OpenRouter to match speakers across
the recording, automatically estimating their number. `--no-diarization`
disables both steps, with either plain text or `--srt` output. Text output prints
speaker IDs above each paragraph:
Speaker diarization is **enabled by default**. MAI supplies the transcription
text and word timestamps through OpenRouter, with its own diarization disabled.
The script separately uploads the **whole recording** as lossless mono 16 kHz
FLAC to [Pyannote Precision-2](https://docs.pyannote.ai/models). Pyannote detects
the speakers across the recording and returns a timeline; the script assigns
those speaker labels to MAI's words without rewriting their text, timestamps,
or order. `--no-diarization` skips Pyannote and removes any unsolicited speaker
labels in the transcription response. It works with plain text and `--srt`.
Text output prints speaker IDs above each paragraph:
```text
[Speaker 0]
@@ -47,14 +56,9 @@ stderr reports missing or incomplete labels when diarization was requested.
Speaker labels use audio evidence; they are not names inferred from text.
SRT uses labels to break cues at speaker changes but does not print the IDs.
Long recordings are automatically split into uploads of at most **10 minutes**
with diarization, or **30 minutes** with `--no-diarization`, including overlap
and MP3 padding. These defaults account for
[Microsoft's documented diarization failures around 15 minutes and longer](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/mai-transcribe).
Use `--chunk-minutes N` (an integer from 1 to 30) to override the limit in either
mode. Larger chunks with diarization may fail with a timeout or
`diarization_unavailable`; reduce the chunk size or use `--no-diarization`.
At 64 kbps, a 10-minute upload is about 4.8 MB of MP3 or 6.4 MB of base64;
Long recordings are automatically split into MAI uploads of at most
**30 minutes** in either mode, including overlap and MP3 padding. Use
`--chunk-minutes N` (an integer from 1 to 30) to override the limit. At 64 kbps,
a 30-minute upload is about 14.4 MB of MP3 or 19.2 MB of base64.
These are local defaults, not guaranteed provider limits:
[OpenRouter documents a 60-second processing timeout](https://openrouter.ai/docs/guides/overview/multimodal/stt),
@@ -67,9 +71,10 @@ no pause. Each upload includes three seconds beyond the join, so neighbouring
uploads share six seconds of audio even during continuous speech. The duration
cap includes this overlap; the next chunk starts before the previous one ends.
Requests with diarization and all chunked requests ask for word timestamps,
including for plain text. Short requests with `--no-diarization` retain the
provider's existing text line breaks and add wrapping to long lines.
With speaker output enabled, and for all chunked requests, MAI is asked for
word timestamps, including for plain text. Short requests with
`--no-diarization` retain the provider's existing text line breaks and add
wrapping to long lines.
The script matches words and their times in the overlap, switches transcripts
at a shared word, and keeps the repeated passage once. It prefers a matching
phrase to distinguish intentional repetitions. If no words match, it uses
@@ -80,53 +85,45 @@ are an error. Plain text for a chunked recording is assembled from the words.
SRT timestamps are shifted back to the original recording's timeline before
cue generation, with continuous cue numbering. Text uses `[Speaker 0]`,
`[Speaker 1]`, etc., without exposing upload boundaries to the reader.
Provider IDs are request-local. The automatic voice comparison maps them to
transcript-wide IDs using audio evidence, including for people absent from a
chunk overlap. Numeric provider IDs alone never establish identity across
requests.
`[Speaker 1]`, etc., without exposing upload boundaries to the reader. Speaker
numbering follows first appearance in the transcript. One Pyannote job supplies
the identities for the entire recording, including speakers absent from a MAI
chunk overlap. MAI's request-local speaker IDs are not used.
Voice comparison uses `google/gemini-3.8-flash` by default and estimates the
number of speakers. If you already know the count, `--speakers N` (1 to 64)
supplies it as an optional constraint. `--speaker-model MODEL` overrides the
default with another audio chat model supporting multiple audio inputs and JSON
schema output. Neither flag is needed for the normal command. Both options
require diarization and cannot be combined with `--no-diarization`.
If you already know the count, `--speakers N` (1 to 64) supplies it to Pyannote
as an optional constraint. For example:
The comparison keeps MAI's words and timestamps, initially treating IDs from
different requests as separate. It takes up to three nonoverlapping excerpts
per provisional speaker (1.5 to 8 seconds each) from uninterrupted labelled
speech and sends them together in **one additional audio request**. Comparing
voices in a shared context can reunite a person absent from a chunk overlap
and merge IDs that MAI split within a single request. The model receives the
voice excerpts and sample numbers, without the written transcript or names.
This uses [OpenRouter audio chat](https://openrouter.ai/docs/guides/overview/multimodal/audio)
and [Gemini's documented audio diarization capability](https://ai.google.dev/gemini-api/docs/audio),
with no local model, new Python package, or separate API key.
Gemini requests use OpenRouter's Google AI Studio provider, whose API supports
[multiple audio inputs](https://ai.google.dev/gemini-api/docs/generate-content/audio#technical-details-about-audio),
with provider fallback disabled. The Vertex backend
[documents a one-audio-file limit](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/capabilities/audio-understanding).
```sh
./transcribe --speakers 2 too_long.webm > transcript.txt
```
Every returned sample assignment must be present exactly once. Excerpts for
the same provisional speaker must agree on one non-null model ID before it
can be merged with others. Speakers without suitable excerpts, uncertain
assignments, or conflicting voices retain separate IDs and produce warnings;
a known count does **not** force uncertain speakers into that many output labels.
If no suitable excerpts
exist, comparison is skipped with a warning. Malformed, incomplete, truncated,
or failed API responses fail the command without partial stdout.
`--speaker-model MODEL` accepts `precision-2` (the default) or `community-1`.
Both run through the Pyannote API; `community-1` does not load a local model.
Neither flag is needed for the normal command. Both require diarization and
cannot be combined with `--no-diarization`.
This is sampled voice comparison, not a fresh segmentation of every word.
It cannot repair a MAI label that incorrectly combines several people into one
turn, and consistent model mistakes can still misidentify a voice. A single
usable excerpt has weaker evidence than several. Review brief interjections,
crosstalk, and edited intros in particular. Word timing and spoken text are
never generated by the comparison model. Requests are bounded to 256 excerpts
and 20 MB of encoded JSON; exceeding a bound fails rather than silently omitting
voices. The extra request adds processing time and provider charges. Its actual
voice-matching accuracy must be checked on real audio; fake-provider tests only
verify the pipeline and its validation rules.
The script requests Pyannote's
[exclusive diarization timeline](https://docs.pyannote.ai/api-reference/diarize),
where at most one speaker is active at a time, and aligns it with MAI word
timestamps. A word goes to the unique speaker whose turns cover the largest
share of its duration, provided that share is at least half. If no turn overlaps
the word, the nearest turn is usable only if the entire word is within 0.2
seconds of it. Ambiguous or tied assignments remain unknown and produce warnings.
Malformed timelines, failed jobs, or a completely empty timeline for nonempty
speech fail the command without partial stdout.
The two models may disagree about timing, and even a known speaker count does
not guarantee correct assignments. Review brief interjections, crosstalk, and
changes in recording conditions such as host-read advertisements. Local tests
verify the integration and validation rules; actual speaker accuracy requires
checking the output against the recording.
Pyannote accepts a recording up to **24 hours** and a FLAC upload up to
**1 GiB**, according to its [API limits](https://docs.pyannote.ai/support/faqs).
The script checks these bounds and uses bounded polling for its job.
The whole-recording job adds processing time and Pyannote charges. Uploaded
audio is automatically deleted within 48 hours, and job results after 24 hours,
under [Pyannote's retention policy](https://docs.pyannote.ai/data-retention).
Temporary local audio is removed on exit.
Audio is first decoded to temporary mono 16 kHz PCM (about 115 MB per hour),
so all cuts share the same sample clock even when container duration metadata
@@ -141,10 +138,36 @@ Responses may finish in any order; merging and speaker numbering always follow
the recording's original order. A failed worker stops further scheduling and
cancels the remaining workers. Interruptions also stop workers and their child
processes before temporary files are removed. Requests already accepted by the
provider may still be processed and billed. There is no automatic retry or
resume, and rerunning sends earlier chunks again. No transcript is emitted
unless every worker, the enabled voice comparison, and the final conversion
succeed.
provider may still be processed and billed. Transcription and job submission
are not automatically retried; transient polling failures retry the same job.
There is no automatic resume, and rerunning sends earlier chunks again. No
transcript is emitted unless every worker, the enabled Pyannote job, and the
final conversion succeed.
Use `--diagnostics DIR` to save intermediate JSON for reviewing an individual
run:
```sh
./transcribe --diagnostics review-run too_long.webm > transcript.txt
```
The script creates a new private directory; it does not reuse an existing one.
It saves `chunks.json` (the upload plan), `transcript.json` (the assembled MAI
transcript before speaker assignment), `diarization.json` (the successful
Pyannote job ID, status, and speaker timelines), and `labeled.json` (the transcript after speaker
assignment), as each stage completes. With `--no-diarization`, only the chunk
plan and unlabelled transcript are saved. Diagnostics contain transcription
content and may be incomplete after a failed run. They remain available for
review, including if the final text or subtitle conversion fails. They contain no raw audio,
credentials, or signed upload URLs. Keep diagnostics and generated transcripts
out of commits.
Re-render saved results without another provider request:
```sh
python3 -B transcript.py review-run/labeled.json > transcript.txt
python3 -B subtitles.py review-run/labeled.json > transcript.srt
```
`--srt` requests word timestamps using OpenRouter's `verbose_json` response.
Words are grouped into cues of at most two lines (42 characters per line,