Performance
The solution has various configurations that trade off accuracy and performance. In this page:
- Realtime streams (RTS) is the number of concurrent streams that can be serviced by a single accelerator using default settings.
- Compute latency 99th-percentile (CL99) is the 99th-percentile compute latency, which measures how long it takes for a model to make a prediction for one audio frame. Note that CL99 increases with the number of concurrent streams.
- User-perceived latency (UPL) is the time difference between when the user finishes saying a word and when it is returned as a transcript by the system.
- WER is the Word Error Rate, a measure of the accuracy of the model. Lower is better.
- HF Leaderboard WER is the WER of the model on the Huggingface Open ASR Leaderboard. WER is averaged across 8 test datasets.
The WERs in the following section are for models trained on ~150k hours of mostly open-source data.
User-perceived latency explains what the latencies are made of.
Monolingual
The solution supports decoding with a beam search with an n-gram language model for improved accuracy. The solution supports greedy decoding for higher throughput.
| Model | Parameters | Decoding | RTS | CL99 at public RTS | p50 UPL | HF WER (<nopnc>/<pnc>) |
|---|---|---|---|---|---|---|
base | 85M | greedy | 2000/1600 | 28/45 ms | 203 ms | 9.91/9.94% |
base | 85M | beam | 1300/1200 | 80/50 ms | - | 9.13/9.17% |
large | 196M | greedy | 800/650 | 57/55 ms | - | 8.76/8.81% |
large | 196M | beam | 500/400 | 40/60 ms | 162 ms | 8.12/8.27% |
extra-large | 404M | greedy | 450/420 | 25/- ms | - | 8.11/8.07% |
extra-large | 404M | beam | 340/290 | 92/- ms | 127 ms | 7.55/7.64% |
Details:
- RTS and CL99 have two values, these are without/with state resetting (see below)
- p50 and CL99 are with the
<pnc>prefix - UPL numbers are measured by streaming LibriSpeech dev-clean to an on-site server with an FPGA backend, with about as many streams as the RTS column gives, without state resets.
State resets is a technique that improves the accuracy on long utterances (over 60s) by resetting the model’s hidden state after a fixed duration, at the cost of real-time streams. All the datasets in the HuggingFace benchmark are sufficiently short that state resets have only a minor (i.e. <.2%) effect on WER.
Monolingual models are fine-tuned variants of the multilingual models.
Multilingual
| Model | Parameters | Decoding | En | Fr | Es | De | It | Pt |
|---|---|---|---|---|---|---|---|---|
base | 85M | greedy | 12.73% | 16.25% | 12.06% | 15.87% | 16.24% | 20.95% |
base | 85M | beam, width=4 | 10.94% | 14.13% | 10.27% | 14.01% | 14.44% | 18.23% |
large | 196M | greedy | 11.06% | 13.46% | 9.48% | 12.81% | 13.93% | 16.66% |
large | 196M | beam, width=4 | 9.73% | 11.82% | 8.39% | 11.71% | 12.56% | 14.93% |
extra-large | 404M | greedy | 9.31% | 10.02% | 7.10% | 9.89% | 11.08% | 11.81% |
extra-large | 404M | beam, width=4 | 8.50% | 9.14% | 6.55% | 9.46% | 9.96% | 11.00% |
- The RTS/CL99 are the same as the above monolingual table but omitted for space.
- The WER’s are against the HF leader board datasets