Data preparation
Having chosen which model configuration to train, you will need to complete the following preprocessing steps:
- Prepare your data in one of the supported training formats:
JSONLorSHAR. - Create a sentencepiece model from your training data.
- Record your training data log-mel stats for input feature normalization.
- Populate a YAML configuration file with the missing fields.
- Generate an n-gram language model from your training data.
- Optionally standardize transcripts
Text normalization
The examples train an English model (langs: [en]), whose character set Changing the character set describes.
Transcripts will be normalized on the fly during training, as configured by
the standardize_text, preserve_case, and preserve_punctuation fields
in the YAML config templates.
That page also covers configuring the character set and normalization,
including for models trained on multiple languages.
During validation, the predictions and reference transcripts will be standardized.
See also
- Prepare LibriSpeech in
JSONLformat - Supported dataset formats
- Input activation normalization
- Controlling the proportion of data from each manifest for multilingual/multi-manifest training