JSONL format

The JSONL format is the default in this repository and if you are training on your own data it is recommended to manipulate it into this format. Note that the data preparation steps are slightly different given the model you have decided to train so please refer to the model configuration page first.

The JSONL format stores a Lhotse CutSet, each cut must have a recording and a single supervision.

Prepare LibriSpeech in JSONL format

This page takes LibriSpeech as it is distributed from the https://www.openslr.org website and prepares it into a JSONL manifest format.

Quick Start

To run the data preparation steps for LibriSpeech and the base model run the following in the container, from caiman_train/, where it starts:

# Download data to /datasets/LibriSpeech: requires 120GB of disk
./scripts/prepare_librispeech.sh

For another configuration, pass its spm size and config name from the model configurations table, for example for large:

SPM_SIZE=17407 CONFIG_NAME=large-17407sp ./scripts/prepare_librispeech.sh

Note

If ~/datasets on the host is mounted to /datasets, the downloaded data will be accessible outside the container at ~/datasets/LibriSpeech.

Further detail: prepare_librispeech.sh

The script downloads LibriSpeech to /datasets/LibriSpeech, writes a librispeech-<subset>-flac.cuts.jsonl.gz manifest beside each subset and adds <EOS> tokens to their transcripts. From the train-960h subsets it then makes the missing fields of the config: the sentencepiece tokenizer in /datasets/sentencepieces/, the log-mel stats in /datasets/stats/, the n-gram language model in /datasets/ngrams/, and the populated configs/<config-name>_run.yaml. _run indicates that this is a complete config, not just a template.

Selecting transcripts

Each supervision must include a “text” field, which contains the transcript of the corresponding audio file.

If multiple transcripts exist for the same audio file, the model can be trained using a specific transcript by specifying it in the use_transcripts field of the YAML configuration:

use_transcripts:
  to_use: ["<transcript_key_1>", "<transcript_key_2>", ...]
  on_missing: "raise_error" # or "use_default"

These extra transcripts are stored as custom data inside each supervision.

The first transcript key that is found in the JSONL manifest will be used.

If no key is found in the JSONL manifest, the behaviour will depend on the value of the on_missing field:

  • “raise_error”: an error will be raised.
  • “use_default”: the default transcript will be used.

Standardizing transcripts

Training on multiple datasets can negatively impact performance, as identical words may be transcribed differently across datasets - for example, “colour” (British English) vs “color” (American English). This inconsistency splits the model’s probability distribution across multiple spellings, reducing prediction confidence.

To address this, we standardize the training transcripts using Whisper EnglishTextNormalizer. This process follows the same approach as WER standardization (see WER calculation docs for more details).

On-the-fly standardization

standardize_text: true in the YAML config, the templates’ default, standardizes the training transcripts on the fly during training. With the SHAR data format this is overlapped with GPU compute, so it costs little training speed.

On-the-fly standardization expands all contractions (e.g “don’t” -> “do not”). As a result, a model trained on standardized transcripts will always predict the expanded form e.g. it will output “do not” whether the speaker says “don’t” or “do not”.

Alternative: pre-standardizing manifests

Alternatively, standardize your manifests once, before training:

python -m caiman_train.data.text.standardize_manifest \
  --manifests /path/to/shar/manifest1 /path/to/shar/manifest2 \
  --output-dirs /path/to/output/for/manifest1 /path/to/output/for/manifest2 \
  --model-config /path/to/model_config.yaml \
  --lang_code en \
  --preserve-case \
  --preserve-punctuation

This stores the standardized text of each supervision in a transcript-standardized custom field, or, with --transcript-fields, each named field <field> in <field>-standardized. --help lists the other options. Unlike on-the-fly standardization, the script expands contractions only with --expand-contractions; not expanding them worsens WERs by ~1-2% relative.

Train on the standardized field by selecting it in your model’s YAML, and turn off on-the-fly standardization with standardize_text: false:

use_transcripts:
  to_use: ["transcript-standardized"]
  on_missing: "raise_error" # or "use_default"

Preserving casing and punctuation

The preserve_case and preserve_punctuation fields of the YAML config decide whether training standardization keeps the transcripts’ casing and punctuation. This is irrelevant to the WER calculation, which always lowercases and standardizes the transcripts before calculating WER.

Next steps

Having run the data preparation steps, go to the training docs to start training.

See also