Performance

The solution has various configurations that trade off accuracy and performance. In this page:

  • Realtime streams (RTS) is the number of concurrent streams that can be serviced by a single accelerator using default settings.
  • Compute latency 99th-percentile (CL99) is the 99th-percentile compute latency, which measures how long it takes for a model to make a prediction for one audio frame. Note that CL99 increases with the number of concurrent streams.
  • User-perceived latency (UPL) is the time difference between when the user finishes saying a word and when it is returned as a transcript by the system.
  • WER is the Word Error Rate, a measure of the accuracy of the model. Lower is better.
  • HF Leaderboard WER is the WER of the model on the Huggingface Open ASR Leaderboard. WER is averaged across 8 test datasets.

The WERs in the following section are for models trained on ~150k hours of mostly open-source data.

User-perceived latency explains what the latencies are made of.

Monolingual

The solution supports decoding with a beam search with an n-gram language model for improved accuracy. The solution supports greedy decoding for higher throughput.

ModelParametersDecodingRTSCL99 at public RTSp50 UPLHF WER (<nopnc>/<pnc>)
base85Mgreedy2000/160028/45 ms203 ms9.91/9.94%
base85Mbeam1300/120080/50 ms-9.13/9.17%
large196Mgreedy800/65057/55 ms-8.76/8.81%
large196Mbeam500/40040/60 ms162 ms8.12/8.27%
extra-large404Mgreedy450/42025/- ms-8.11/8.07%
extra-large404Mbeam340/29092/- ms127 ms7.55/7.64%

Details:

  • RTS and CL99 have two values, these are without/with state resetting (see below)
  • p50 and CL99 are with the <pnc> prefix
  • UPL numbers are measured by streaming LibriSpeech dev-clean to an on-site server with an FPGA backend, with about as many streams as the RTS column gives, without state resets.

State resets is a technique that improves the accuracy on long utterances (over 60s) by resetting the model’s hidden state after a fixed duration, at the cost of real-time streams. All the datasets in the HuggingFace benchmark are sufficiently short that state resets have only a minor (i.e. <.2%) effect on WER.

Monolingual models are fine-tuned variants of the multilingual models.

Multilingual

ModelParametersDecodingEnFrEsDeItPt
base85Mgreedy12.73%16.25%12.06%15.87%16.24%20.95%
base85Mbeam, width=410.94%14.13%10.27%14.01%14.44%18.23%
large196Mgreedy11.06%13.46%9.48%12.81%13.93%16.66%
large196Mbeam, width=49.73%11.82%8.39%11.71%12.56%14.93%
extra-large404Mgreedy9.31%10.02%7.10%9.89%11.08%11.81%
extra-large404Mbeam, width=48.50%9.14%6.55%9.46%9.96%11.00%
  • The RTS/CL99 are the same as the above monolingual table but omitted for space.
  • The WER’s are against the HF leader board datasets