This dataset accompanies the paper "Orchestrating Schema Conversions Across Heterogeneous Schema Languages: A Graph-Based, Black-Box Approach" and contains source code snapshots, evaluation inputs, and evaluation outputs for the Schema Conversion Orchestrator and its integration into the MetaConfigurator tool.
The dataset includes a snapshot of the Schema Conversion Orchestrator, a Python/Flask service that discovers and chains heterogeneous schema converters (covering JSON Schema, XSD, SHACL, LinkML, MD-Models, and further languages) and ranks the resulting conversion paths by empirical quality. It further includes a snapshot of MetaConfigurator, the end-user web application into which the orchestrator is integrated. The evaluation data was produced by running the orchestrator on three representative real-world input schemas per source language and collecting the ranked conversion outputs. Each output was then annotated by a coding agent (Claude Sonnet 4.6) and reviewed by a human, yielding a per-output quality label (G - good, L - lacking, I - invalid) and free-text notes.
The dataset is organized into four parts: (1) source code of the orchestrator and MetaConfigurator; (2) evaluation inputs: the schema files used as conversion sources, organized by language; (3) evaluation results: two run folders (20260612_protobuf_edges for Protobuf targets, 20260616_112512 for the five-language core evaluation), each containing per-path conversion outputs, per-run metadata in all_metadata.json, and annotated review CSVs (final_outputs.csv, edge_outputs.csv); and (4) plots and derived scores: the result matrix, edge robustness graph, and the machine-readable edge_robustness_scores.json and protobuf_edge_robustness_scores.json files consumed by the orchestrator at runtime.
The review CSVs are the primary evaluation artifact: each row records one conversion attempt with its source and target language, input file, path signature (the chain of converter libraries used), rank, automatic validity flag, and human annotation. The edge_robustness_scores.json files summarize per-converter robustness (fraction of syntactically valid outputs) and conditional quality ((0.9·G + 0.5·L) / reviewed outputs) and are directly loadable by the orchestrator to reproduce the reported ranking behavior. The included evaluation code and input schemas are designed for reuse: new converters or target languages can be added to the orchestrator registry and evaluated by re-running evaluate.py, and the set of input schemas can be extended with additional or more complex examples for a more thorough assessment. The annotated outputs can further serve as a reference for benchmarking alternative converters or ranking strategies.
