A multilingual benchmark for evaluating metalinguistic knowledge WALS-Bench 1.0

PID

This is a large-scale multilingual benchmark for evaluating metalinguistic knowledge (i.e. explicit knowledge about the structure of languages) in large language models using grammatical features from the World Atlas of Language Structures (WALS). The benchmark covers 192 linguistic features across 12 linguistic domains and 2,660 languages and is available in two formats (jsonl files):

  • Format 1 (192-question version): One question per feature, under which all languages with a corresponding ground truth value for that feature are listed.
  • Format 2 (76,475-question version): One question per feature-language pair with a corresponding ground truth value, fully expanded across all languages.

The original WALS data is licensed under CC BY 4.0. The data has been adapted for use in this benchmark. Source: Dryer, Matthew S. & Haspelmath, Martin (eds.). World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology. https://wals.info

Identifier
PID http://hdl.handle.net/11356/2083
Related Identifier https://arxiv.org/abs/2602.02182
Metadata Access http://www.clarin.si/repository/oai/request?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:www.clarin.si:11356/2083
Provenance
Creator Arčon, Tjaša; Klemen, Matej; Robnik-Šikonja, Marko; Dobrovoljc, Kaja; Terčon, Luka
Publisher Faculty of Computer and Information Science, University of Ljubljana
Publication Year 2026
Rights Creative Commons - Attribution 4.0 International (CC BY 4.0); PUB; https://creativecommons.org/licenses/by/4.0/
OpenAccess true
Contact info(at)clarin.si
Representation
Language English; Multiple languages
Resource Type corpus
Format text/plain; charset=utf-8; application/zip; downloadable_files_count: 1
Discipline Linguistics