Emission Latency
Emission latency (EL) is defined by the time difference between the end of a spoken word in an audio file and when the model outputs the final token for the corresponding word, minus the mean frame latency. After the model receives the final audio frame of a word, it might not predict the word until it has heard a few more frames of audio. EL measures this delay.
To calculate the model’s EL during validation, pass the --calculate_emission_latency flag, e.g.
./scripts/val.sh --calculate_emission_latency
Emission latencies are calculated by aligning the model’s word timestamps with the corresponding ground truth alignments. Ground truth alignments are expected to be in the manifest files for the dataset.
See Forced Alignment for details on producing ground truth alignments.
Validation reports the mean latency, as well as the 50th, 90th, and 99th percentile latencies.
Moreover, the Token Usage Rate is reported. This is the proportion of words’ timestamps that are used in the emission latency calculation.
EL logging is also compatible with val_multiple.sh.
Forced Alignment
The script forced_align.py is used to align audio recordings with their corresponding transcripts, producing ground truth timestamps for each word.
To perform forced alignment, execute the script with the required arguments e.g.
python -m caiman_train.latency.forced_align --dataset_dir /path/to/dataset --manifests data.jsonl.gz --output_dir /path/to/output/dir --lang_code lang_code
The alignments are added to each supervision segment in the cutset, and the resulting JSONL is saved to the output_dir.
--manifests takes several manifests at once, and --overwrite overwrites existing output.
Utterances are aligned in segments of --segment_len minutes. This allows forced alignment on datasets with very long utterances (e.g. Earnings21) without running out of memory.
--cpu runs the alignment on the CPU; --help lists the other options.
Alignment
The alignment uses the Lhotse Alignment format where each token has an associated AlignmentItem. Each AlignmentItem consists of the token, start time, duration and confidence score, for example:
"word": [
[
"Thank",
0,
0.32,
0.49
],
[
"you",
0.32,
0.08,
0.97
]
]
Next Steps
To improve the emission latency of your model, consider training with a Delay Penalty.