CAIMAN-ASR benchmark

The benchmark streams audio to a CAIMAN-ASR server in real time, as a user would, and measures the word error rate (WER) and the user-perceived latency. It’s the caiman-asr-benchmark command in the caiman-asr-core package, which uv installs and runs:

git clone https://github.com/MyrtleSoftware/caiman-asr
cd caiman-asr/training/caiman-asr-core

Run all the commands below from that directory.

Note on the datasets

The benchmark reads a JSONL manifest, passed with --manifest, and --data_dir, the directory that its relative audio paths are relative to. Please refer to benchmarking with custom datasets for instructions on how to use a custom dataset.

LibriSpeech dev-clean

To benchmark on LibriSpeech dev-clean, first download it, with a manifest of its word alignments:

uv run --extra bench caiman-asr-benchmark download-librispeech

This saves both under ~/.cache/myrtle/librispeech/LibriSpeech. Then pass --manifest ~/.cache/myrtle/librispeech/LibriSpeech/librispeech-dev-clean-flac.cuts.jsonl.gz --data_dir ~/.cache/myrtle/librispeech/LibriSpeech to the commands below.

CAIMAN-ASR

caiman-asr-benchmark run is composed of 2 steps which are both performed in sequence. There are however flags to control the flow and skip individual steps.

  1. Transcription
  2. Evaluation

Running the command will perform both steps.

uv run --extra bench caiman-asr-benchmark run --manifest <manifest> --data_dir <audio_dir> --address <ADDRESS> --port <PORT> --run_name <unique name> --append_results caiman-base --lang_code <lang-code>

where --address is the websocket address of transcription service and --port is the port. If --prefix is omitted, the latest server uses the model’s declared default prefix. See Prefix to check or override it.

The above command assumes you’re using the CAIMAN-ASR base model. If you want to use the large model, replace --append_results caiman-base with --append_results caiman-large.

Note on client speed

The client simulates a real user by feeding the audio to the server at real time. Since there are several hours of audio, the client will take several hours to run.

To instead do a trial run on just five audio files, pass the flag --limit_to 5.

Transcription

This step runs the transcription step and generates *.caiman-asr.trans transcription files. These are saved to ~/.cache/myrtle/benchmark/<run_name>/, mirroring the audio folder layout. For each audio file, the script generates one transcription file.

By default, the transcription is skipped for already done audio, which will happen if you run the script another time with the same run_name. This behavior can be changed by passing the --force_transcription flag, and all files will be transcribed from scratch and existing transcription files will be overwritten.

uv run --extra bench caiman-asr-benchmark run --manifest <manifest> --data_dir <audio_dir> --address <ADDRESS> --port <PORT> --force_transcription --run_name <name of previous run> --append_results caiman-base --lang_code <lang-code>

The transcription step can be skipped by passing the --skip_transcription flag.

uv run --extra bench caiman-asr-benchmark run --manifest <manifest> --data_dir <audio_dir> --address <ADDRESS> --port <PORT> --skip_transcription --run_name <name of previous run> --append_results caiman-base --lang_code <lang-code>

Evaluation

This step aggregates the generated trans files into a hypothesis JSONL and computes word error rate (WER) and latency. The evaluation step can be skipped by passing the --skip_evaluation flag.

uv run --extra bench caiman-asr-benchmark run --manifest <manifest> --data_dir <audio_dir> --address <ADDRESS> --port <PORT> --skip_evaluation --run_name <unique name> --append_results caiman-base --lang_code <lang-code>

Results will also be saved in ~/.cache/myrtle/benchmark/results/[append_results].csv.

Prefix (punctuation/casing and language)

Use --prefix to condition the RNN-T prediction network:

  • <pnc>: punctuated and cased output
  • <nopnc>: lowercase only (slightly more accurate)

Choose the prefix supported by your model:

  • Models trained with punctuation tags only: use --prefix='<pnc>' or --prefix='<nopnc>', without a language tag.
  • Models trained with punctuation and language tags: use both, with punctuation first, e.g. --prefix='<pnc><lang_en>' for English.

For English with a model that accepts punctuation and language tags:

uv run --extra bench caiman-asr-benchmark run --manifest <manifest> --data_dir <audio_dir> --address <ADDRESS> --port <PORT> --run_name <unique name> --append_results caiman-base --prefix='<pnc><lang_en>' --lang_code en

The accepted prefixes depend on the model’s training. See Prefix for details. --lang_code controls evaluation normalization, not the model language.

Keyword Boosting

CAIMAN-ASR supports keyword boosting of up to 100 keywords. Keyword boosting is a technique to improve the recognition of domain-specific words/phrases and proper nouns, by boosting the probability of specific tokens at inference time.

To provide keywords to the server, use the --keywords argument with a space-separated list of keywords e.g.

uv run --extra bench caiman-asr-benchmark run --manifest <manifest> --data_dir <audio_dir> --address localhost --run_name keyword-test \
  --append_results caiman-base --lang_code en \
  --keywords caiman asr myrtle --keyword-c0 0.3 --keyword-beta 0.9 \
  --keyword-context-score 1.5

The three boost controls (i.e. the --keyword-* flags) take numeric values and require a non-empty --keywords list. Omitted controls use server defaults. --keyword-context-score is an overall boost multiplier. The same settings apply to each file, on a new connection. These controls override the server-wide values per connection. See Keyword boosting hyperparameters for what c0, beta, and context_score do and tuning recommendations. Use a new --run_name when comparing settings to avoid reusing transcripts.

Flush

Add --manual-flush to enable Enter-to-flush in an interactive terminal:

uv run --extra bench caiman-asr-benchmark run --manifest <manifest> --data_dir <audio_dir> --address localhost --run_name flush-test \
  --append_results caiman-base --lang_code en \
  --manual-flush

Press Enter while audio is streaming to finalize a segment and continue on the same connection. Acknowledgements are saved as flush_received events; word timestamps remain relative to the file start. Flush is disabled by default. Each file still ends with an end-of-input message.

With --pause_after_each, Enter at the prompt after a file finishes advances to the next file. Flushing can affect WER and latency, so compare against a run without flushing and record when you pressed Enter.

List of arguments

The arguments for caiman-asr-benchmark run are as follows:

--address <STRING_VALUE>: Server address the transcription service is running on, required.
--port <INT_VALUE>: Port number the transcription service is running on, default=3030.
--manifest <PATH>: Path to the JSONL manifest (.jsonl or .jsonl.gz), required.
--data_dir <PATH>: Directory that the manifest's relative audio paths are relative to, required.
--force_transcription: Re-transcribe all audio files from scratch, default=False.
--skip_transcription: Skip transcription step and go directly to the evaluation step, default=False.
--skip_evaluation: Skip evaluation step, default=False.
--limit_to <INT_VALUE>: Limit transcription to a number of files. Suitable for testing purposes, default=None.
--run_name <STRING_VALUE>: Name used to identify results.
  Individual transcription files will be saved to `$HOME/.cache/myrtle/benchmark/<run name>`.
--append_results <STRING_VALUE>: Results will be appended to `~/.cache/myrtle/benchmark/results/[append_results].csv`
  Must be one of caiman-base, caiman-large.
--prefix <STRING_VALUE>: Model-specific prefix; omitted to use the model's default on the latest server.
--keywords <STRING_VALUE>: Space-separated list of keywords to be boosted.
--keyword-c0 <FLOAT_VALUE>: Keyword boost override; requires non-empty --keywords.
--keyword-beta <FLOAT_VALUE>: Keyword boost override; requires non-empty --keywords.
--keyword-context-score <FLOAT_VALUE>: Overall boost multiplier; requires non-empty --keywords.
--manual-flush: Press Enter to flush while streaming; disabled by default.
--pause_after_each: Wait for Enter after each file before continuing.
--lang_code <STRING_VALUE>: ISO language code to use for normalization of transcripts.
--transcript_key <KEY>: Supervision custom field to use as the WER reference instead of text; falls back to text if missing.