Automatic batch size reduction
When validating on long utterances with the large model, the encoder may run out of memory even with a batch size of 1.
State resets are implemented by splitting one utterance into a batch
of smaller utterances, even when --val_batch_size=1.
This creates an opportunity to reduce the VRAM usage
further, by processing the ‘batch’ created from one long utterance in smaller
batches, instead of all at once.
The validation script will automatically reduce the batch size if the number
of inputs to the encoder is greater than --max_inputs_per_batch.
Note that this option can’t reduce memory usage on a long utterance if state resets is turned off, since the batch size can’t go below 1.
You may wish to reduce --max_inputs_per_batch if you have a smaller GPU or longer utterances.