Code Switching

Code switching trains a multilingual model on examples that change language part way through. It builds them on the fly from ordinary single-language data, so no code-switched training data is needed. It builds on Cut Concatenation and Language Detection.

Enabling

Code switching is off by default and is turned on by --codeswitch_prob. It needs concatenation, conditional language training and a <lang_detect> to also be turned on:

./scripts/train.sh \
  --conditional \
  --conditional_language \
  --lang_mask_prob <PROB> \
  --concat_max_duration <N_SECS> \
  --codeswitch_prob <PROB> \
  # ... other args ...

The model configuration must have a lang_detect user token, same as for Language Detection.

How the codeswitched cuts are built

Without code switching, concatenation keeps one bin per language and sends each utterance to the bin of its own language. With code switching, there is also a bin for every pair of languages along with the monolingual bins, and a router decides where each utterance goes: to its own language’s bin, or to a pair bin that includes its language. An utterance never goes to a bin without its language. Shuffling, packing and emitting a bin’s concatenated cut when the next utterance does not fit work as in plain concatenation.

The language prefix is added once a cut is emitted. A mixed cut gets <lang_detect> prefix. A single-language cuts gets either a<lang_detect> prefix or a prefix token matching the cuts own language, such that --codeswitch_prob of the <lang_detect> audio is code-switched and the rest is monolingual. This is done so that the <lang_detect> prefix doesn’t become a cue for the model to expect code-switching.

The language mix of the training data does not change. Each language’s audio is only divided between single-language and mixed examples.

How much is code-switched

--lang_mask_prob (\(c\)) is the share of output audio that carries <lang_detect>. --codeswitch_prob (\(p\)) is the part of that which is code-switched. So the code-switched share of output is

$$ S = c \cdot p $$

and the remaining \(c(1 - p)\) is single-language audio carrying <lang_detect>. For example, with --lang_mask_prob 0.30 and --codeswitch_prob 0.50, \(S = 0.15\). About 15% of the training audio is code-switched.

\(p\) must be between 0 and 1 (non-inclusive). At 1 every <lang_detect> cut would switch and the model would learn that the prefix means a switch is coming, so training rejects it. At 0, nothing is code-switched.

Shaping the mix

Every pair of languages is half of each language by audio, over all of its cuts. Inside a cut the two can be in any ratio, ie one {en, fr} cut can be mostly English and the next mostly French.

--codeswitch_temperature

How the code-switched audio divides between the languages, from 0 to 1 (default 1). Language \(i\)’s share of the code-switched audio is proportional to its share of the data \(d_i\) raised to the temperature \(t\):

$$ w_i = \frac{d_i^{t}}{\sum_j d_j^{t}} $$

• 1 follows the data. Common languages supply most of the switched audio, and pairs of common languages are the most common pairs.
• 0 makes every language supply the same amount. Rare languages then switch a much larger fraction of their own audio.
• any value between 0 and 1 flattens the mix partly.

Example

Three languages with English 60%, French 25% and German 15% of the data, at \(S = 0.15\). The table shows the fraction of each language’s own audio that ends up in mixed cuts, and each pair’s share of all output:

At temperature 1 there is no French–German switching. Following the data, English would supply 60% of the switched audio, but it is half of every pair it joins, so it is capped at 50% (see Limits). French and German keep their 25 : 15 ratio and are scaled up to supply the other 50%: 31.25% and 18.75%, each \(0.5 / 0.4\) times its share of the data. English’s half then needs every second of their switched audio as its partner, so none is left for French and German to pair with each other. At temperatures 0.5 and 0 no language would supply more than half, so nothing is capped.

Limits

A language can supply at most half of all the code-switched audio, because it is half of every pair it joins. A language that would supply more is capped at half, and the others make up the rest in proportion.

A language also cannot put more audio into mixed cuts than it has. Language \(i\) has to supply \(S \cdot w_i\) of the output from its own \(d_i\), so

$$ S \le \frac{d_i}{w_i} \quad \text{for every language } i $$

The language with the least audio for what it is asked to supply sets the limit. Training stops with an error if --lang_mask_prob asks for more, and the error gives the largest value that works. For example, with English 90% and French 10% at temperature 0, French has to supply half of every code-switched second from 10% of the data. That limits \(S\) to 0.2, which is --lang_mask_prob 0.4 at --codeswitch_prob 0.5. Lower --lang_mask_prob, or raise the temperature, to fit.

Routing

Each utterance draws its bin from fixed probabilities, solved for at start-up, whatever has been emitted so far. A pair bin’s cut can be emitted before its second language arrives, and then counts as single-language audio (leakage). Fixed probabilities keep each bin’s arrivals independent, so renewal theory predicts, from each language’s mean and spread of utterance lengths, how much of each pair leaks. Each pair is sent enough extra to make up for it:

$$ \text{routed} \times (1 - \text{leakage}) = \text{target} \quad \text{for every pair} $$

Leakage grows with what is routed, so this is solved by iteration. Training stops with an error when no fixed probabilities reach the target; lowering --lang_mask_prob, or raising --concat_max_duration so each cut holds more utterances, helps.

The lengths come from the manifests’ statistics, scaled for speed and tempo perturbation. Those statistics are taken before the token filters (--min_tps, --max_tps, token counts), so a manifest that loses many of its short cuts to them leaks more than predicted, and its pairs come out short. The prediction is approximate in any case, and least accurate when a concatenated cut holds only a few utterances, so the realised mix lands within a few percent of the target rather than on it.

The training log gives the probabilities, the predicted leakage, and how long they took to solve.

Checking the mix

The training loop counts the mix of the batches it trains on, from each concatenated cut’s languages and their lengths without the gaps. Each time the model is validated, and at the last training step, the training log shows that mix, summed over all of the run’s GPUs, and it is written to codeswitch_report.json in the training output directory. The file is rewritten each time, so it always holds the state so far. It gives the training step and, for every language and partner, the realised share of output, the target and the seconds trained on.

Validation

Validation sets are single-language, so they measure the cost of code switching on ordinary speech. To measure code switching itself, validate on a code-switched test set with --val_lang_prefix lang_detect, since no single language token describes its utterances.