Data preparation

Having chosen which model configuration to train, you will need to complete the following preprocessing steps:

  1. Prepare your data in one of the supported training formats: JSONL or SHAR.
  2. Create a sentencepiece model from your training data.
  3. Record your training data log-mel stats for input feature normalization.
  4. Populate a YAML configuration file with the missing fields.
  5. Generate an n-gram language model from your training data.
  6. Optionally standardize transcripts

Text normalization

The examples assume a character set of size 73 (English, i.e. langs: [en]):

  • 26 lowercase letters,
  • 26 uppercase letters,
  • space,
  • full stop (period),
  • comma,
  • apostrophe,
  • question mark,
  • digits 0-9,
  • %, $, ¢, €, £, -.

Transcripts will be normalized on the fly during training, as configured by the standardize_text, preserve_case, and preserve_punctuation fields in the YAML config templates. See Changing the character set for how to configure the character set and normalization, including for models trained on multiple languages.

During validation, the predictions and reference transcripts will be standardized.

See also