🗣️Whisper-Small Spots Code-Switched Speech With 0.798 F1 Across 14 Language Pairs

Your speech corpus is missing half the way people talk

Code-switching, the habit of jumping between languages mid-sentence, is everywhere in real speech. Your training data mostly isn't. A new paper shows you can find those utterances directly from audio, using the encoder you probably already have. The usual route is to transcribe everything and scan the text for language switches. That only works if your multilingual ASR is accurate, which is exactly the problem when you're hunting for rare mixed speech. This method skips the transcript. It finds the hidden-state dimensions that respond to one language and watches how frame-level activations shift across an utterance. On CS-FLEURS across 14 language pairs, Whisper-small's encoder hits a mean F1 of 0.798. A CNN gets 0.776. A HuBERT linear probe trained only for monolingual language ID manages 0.751. The catch is that the probe is trained on single-language recordings, so the detector never sees mixed data during training. Spoiler: that's the point, and it still works.