Benchmarking CAIMAN_ASR on custom datasets

Data format in JSONL

The benchmark uses the same JSONL format as training, so manifests prepared for training or validation can be benchmarked directly. To select an alternative transcript as the WER reference, pass --transcript_key (see selecting transcripts).

For a custom dataset, generate a manifest with reference text in supervisions and audio paths in recording.sources. Latency also needs word alignments in each supervision (alignment.word, see forced alignment).

For example, one line has this structure:

{
  "id": "recording-001",
  "start": 0.0,
  "duration": 11.21,
  "channel": 0,
  "supervisions": [
    {
      "id": "recording-001",
      "recording_id": "recording-001",
      "start": 0.0,
      "duration": 11.21,
      "channel": 0,
      "text": "A sample reference transcript.",
      "alignment": {
        "word": [
          ["A", 0.52, 0.1, 1.0],
          ["sample", 0.66, 0.42, 1.0],
          ["reference", 1.12, 0.5, 1.0],
          ["transcript.", 1.68, 0.7, 1.0]
        ]
      }
    }
  ],
  "recording": {
    "id": "recording-001",
    "sources": [
      {
        "type": "file",
        "channels": [0],
        "source": "audio/recording-001.wav"
      }
    ],
    "sampling_rate": 16000,
    "num_samples": 179360,
    "duration": 11.21,
    "channel_ids": [0]
  },
  "type": "MonoCut"
}

Notes on the custom dataset format

Audio in a format other than 16 kHz WAV, such as FLAC, is converted before it’s streamed.
The manifest can be a .jsonl or .jsonl.gz file.
Relative audio file paths in the manifest are relative to the --data_dir directory.
Each line must be a MonoCut that covers its whole recording (start 0 and the recording’s duration) from one audio file. The client streams whole recordings, so the benchmark stops with an error on any other cut.

Running the evaluation on custom data

Run the evaluation script according to the instructions in CAIMAN-ASR benchmark with the additional flags:

--manifest <manifest> --data_dir <audio_dir>