How well do voice models understand Swiss German?

Swiss German is a hard test for speech models. There is no standard spelling, dialects differ a lot, and most evaluation data focuses on standard languages. So I built a small benchmark to get evidence instead of impressions.

Setup

• 200 clips from ETH SwissDial, 25 per dialect, across AG, BE, BS, GR, LU, SG, VS, and ZH
• Two tasks: translate to High German, and transcribe the dialect verbatim
• A dialect-guided prompting strategy, with paired scoring so models are only compared on clips both handled successfully

Results

• High German translation: Realtime 2.1 reached 68.6% paired word match, against 64.3% for Realtime 2
• Dialect transcription: Realtime 2.1 reached 46.6%, against 44.6% for Realtime 2
• Valais German (VS) was the hardest dialect in both tasks

Caveats

These are single-run observations on a small, balanced sample, not a universal model ranking. Text similarity also penalizes legitimate spelling and translation alternatives, which matters a lot for a language without a standard spelling.

The full workbench, including word-level diff views and per-dialect breakdowns, is on GitHub.