JSONL format
The JSONL format is the default in this repository and if you are training on your own data it is recommended to manipulate it into this format. Note that the data preparation steps are slightly different given the model you have decided to train so please refer to the model configuration page first.
The JSONL format stores a Lhotse
CutSet, each cut
must have a recording and a single supervision.
Prepare LibriSpeech in JSONL format
This page takes LibriSpeech as it is distributed from the https://www.openslr.org website and prepares it into a JSONL manifest format.
Quick Start
To run the data preparation steps for LibriSpeech and the base model run the following in the container, from caiman_train/, where it starts:
# Download data to /datasets/LibriSpeech: requires 120GB of disk
./scripts/prepare_librispeech.sh
For another configuration, pass its spm size and config name from the
model configurations table, for example for large:
SPM_SIZE=17407 CONFIG_NAME=large-17407sp ./scripts/prepare_librispeech.sh
If ~/datasets on the host is mounted to /datasets, the downloaded data will be accessible outside the container at ~/datasets/LibriSpeech.
Further detail: prepare_librispeech.sh
The script downloads LibriSpeech to /datasets/LibriSpeech, writes a
librispeech-<subset>-flac.cuts.jsonl.gz manifest beside each subset and adds
<EOS> tokens to their transcripts. From the train-960h
subsets it then makes the
missing fields of the
config: the sentencepiece tokenizer in /datasets/sentencepieces/, the
log-mel stats in /datasets/stats/, the
n-gram language model in /datasets/ngrams/, and the populated
configs/<config-name>_run.yaml. _run indicates that this is a complete
config, not just a template.
Selecting transcripts
Each supervision must include a “text” field, which contains the transcript of the corresponding audio file.
If multiple transcripts exist for the same audio file, the model can be trained using a specific transcript by specifying it in the use_transcripts field of the YAML configuration:
use_transcripts:
to_use: ["<transcript_key_1>", "<transcript_key_2>", ...]
on_missing: "raise_error" # or "use_default"
These extra transcripts are stored as custom data inside each supervision.
The first transcript key that is found in the JSONL manifest will be used.
If no key is found in the JSONL manifest, the behaviour will depend on the value of the on_missing field:
- “raise_error”: an error will be raised.
- “use_default”: the default transcript will be used.
Standardizing transcripts
Training on multiple datasets can negatively impact performance, as identical words may be transcribed differently across datasets - for example, “colour” (British English) vs “color” (American English). This inconsistency splits the model’s probability distribution across multiple spellings, reducing prediction confidence.
To address this, we standardize the training transcripts using Whisper EnglishTextNormalizer. This process follows the same approach as WER standardization (see WER calculation docs for more details).
On-the-fly standardization
standardize_text: true in the YAML config, the templates’ default,
standardizes the training transcripts on the fly during training. With the
SHAR data format this is overlapped with GPU compute, so it costs little
training speed.
On-the-fly standardization expands all contractions (e.g “don’t” -> “do not”). As a result, a model trained on standardized transcripts will always predict the expanded form e.g. it will output “do not” whether the speaker says “don’t” or “do not”.
Alternative: pre-standardizing manifests
Alternatively, standardize your manifests once, before training:
python -m caiman_train.data.text.standardize_manifest \
--manifests /path/to/shar/manifest1 /path/to/shar/manifest2 \
--output-dirs /path/to/output/for/manifest1 /path/to/output/for/manifest2 \
--model-config /path/to/model_config.yaml \
--lang_code en \
--preserve-case \
--preserve-punctuation
This stores the standardized text of each supervision in a
transcript-standardized custom field, or, with --transcript-fields, each
named field <field> in <field>-standardized. --help lists the other
options. Unlike on-the-fly standardization, the script expands contractions
only with --expand-contractions; not expanding them worsens WERs by ~1-2%
relative.
Train on the standardized field by selecting it in your model’s YAML, and
turn off on-the-fly standardization with standardize_text: false:
use_transcripts:
to_use: ["transcript-standardized"]
on_missing: "raise_error" # or "use_default"
Preserving casing and punctuation
The preserve_case and preserve_punctuation fields of the YAML config
decide whether training standardization keeps the transcripts’ casing and
punctuation. This is irrelevant to the WER calculation, which always lowercases
and standardizes the transcripts before calculating WER.
Next steps
Having run the data preparation steps, go to the training docs to start training.