research-bundle-reconciliation

Reconciliation of the semantic-naming research bundle

Type: Synthesis
Status: Active
Canonical for: Reconciled naming claims, evidence boundaries, corrections and experiment precedence
Depends on: Evidence ledger, primary studies and the preserved source archive

This document does not recommend PDS naming, a component contract, or a production architecture.

Bundle reviewed

The six supplied attachments contain overlapping research stages rather than six independent studies. Their evidence-changing conclusions are preserved in this reconciliation and the evidence ledger. Textual snapshots of the original synthesis and attachments are retained in the source archive, but remain inputs rather than canonical conclusions:

1. Detailed experimental design
2. Audit and deepened report
3. Focused report on semantic naming and contracts
4. Design-system practitioner leads on X
5. Tool/schema practitioner leads on X
6. Three-brief deep-research synthesis

The longer semantic-naming and machine-readable-contracts synthesis is preserved beside them. It is interpreted through this reconciliation rather than used as an unqualified source of truth.

The same papers recur across several attachments. Repetition is not independent corroboration. Each primary study is counted once.

Evidence precedence

Where the attachments disagree, use this order:

1. Verified primary paper or benchmark.
2. The audit, because it explicitly corrects identifier-class and causal overreach.
3. The reconciled synthesis whose corrected findings are preserved in this file and the evidence ledger.
4. Focused/deep reports as discovery and interpretation aids.
5. Practitioner reports with direct, verifiable post or artifact URLs.
6. Unlinked X summaries as leads only.

This precedence preserves the central distinctions:

• tool/component name is not an enum-value name;
• parameter/property name is not an enum value;
• a legal enum is not the correct enum;
• an expressive standard is not evidence of LLM performance;
• a proposed architecture is not a demonstrated optimum.

Claims retained

Demonstrated in adjacent tool/schema tasks

• Tool names affect tool selection when descriptions and tasks are held fixed.
• Parameter names affect parameter identification in controlled tool-use settings.
• Noise, opacity and semantic mismatch in names can materially change performance.
• Name–description conflict can produce severe, highly confident errors in the cited method-selection report.
• Schema-key wording can act as an instruction channel, with model-family-dependent effects.
• Hard structured constraints improve structural validity.
• Structural validity does not guarantee semantic or executable correctness; hard constraints can increase wrong-but-valid output in some small-model settings.
• Concise standardized documentation and examples can improve related tool retrieval/use while reducing context cost.

These findings justify a design-system experiment. They do not establish its outcome.

Design-system practice and standards precedent

• Mature systems expose names, types, states, examples and implementation bindings in different forms.
• Carbon, Storybook and Figma provide inspectable retrieval/mapping/validation mechanisms.
• Atlassian and SAP describe intent, contextual fit and composition semantics more completely than they expose the underlying public schema.
• WAI-ARIA, DTCG, SKOS, MCP and JSON Schema demonstrate separation of semantic kinds, identifiers/labels, annotations and constraints.

This is evidence that the information can be represented, not that every item belongs in an AI-ready contract.

Claims downgraded or rejected

Claim found in the bundle	Reconciled status	Reason
Tool/parameter naming evidence demonstrates enum-value naming effects	Rejected	The manipulated identifier class is different.
Meaningful enum names outperform opaque identifiers	Unestablished	No verified fixed-schema, fixed-description enum-name ablation was found.
Familiar-but-misleading enum values are worse than opaque IDs	Unestablished	This is the critical proposed comparison, not a result.
Descriptions override misleading enum names	Unestablished	Per-value name × description override has not been isolated.
PA-Tool and RoTBench directly map to variant-value choice	Adjacent only	They support the mechanism at tool/parameter level.
“By extension enum values” is demonstrated	Rejected wording	It collapses inference into direct evidence.
Guidance to the model plus post-validation is the proven optimum	Plausible architecture hypothesis	Constraint-tax and delayed-packaging work motivate it, but no matched Button/design-system bake-off establishes optimality.
JSON is more reliable than YAML for agents	Practitioner lead / unestablished	Information content, grammar, tooling and validation are confounded.
Practitioner success proves semantic naming improves accuracy	Anecdotal	System maturity, prompts, retrieval, examples and task selection are uncontrolled.
A directional “small models show no naming effect, frontier models do” hypothesis	Not justified	Hood reports a small-model null, while PA-Tool reports gains in SLMs; family, training and task differ. Test an interaction without prescribing direction.

Practitioner evidence quarantine

The X reports are useful for candidate failure modes and terminology, but most entries provide profile links rather than direct post URLs, dates, prompts, model versions, task sets or raw outcomes. They should not be cited as demonstrated findings.

Retain them in a practitioner-leads register with these possible follow-ups:

• obtain exact post/thread URL;
• preserve the complete thread and date;
• identify model, toolchain and design-system version;
• recover prompts, schemas and task cases;
• distinguish component selection, property selection, token matching and visual fidelity;
• identify whether the reported number was pre-registered, repeated or manually scored.

Until then, claims such as “14/14 token accuracy,” “JSON outperformed YAML,” or “naming fixed agent breakage” remain leads.

Experimental-design reconciliation

The supplied detailed protocol contains strong controls: a name-free concept layer, identical descriptions across naming treatments, order randomization, explicit validity/correctness separation and a prior-capture failure category.

However, it should not be run unchanged.

Required corrections

1. Use one semantic axis in the first study. The supplied C1–C7 set mixes hierarchy, consequence, dismiss, navigation, confirmation and submission. That makes enum-name effects inseparable from axis recognition. The first experiment should use Button action hierarchy only.
2. Make model interaction exploratory. Do not pre-register a directional small-versus-frontier hypothesis.
3. Use multiple label sets per treatment. One aligned pair, one opaque pair and one misleading pair confound treatment with particular strings. Lexical set/mapping must be replicated or modeled.
4. Pretest labels without contaminating evaluation. Model-prior elicitation must use separate prompts and no evaluation scenarios. Prefer a shared label pool meeting cross-model criteria, then record per-model manipulation strength.
5. Separate the property-name block. Property wording and enum wording are distinct interventions. Test property names at a fixed description level rather than fully crossing everything initially.
6. Move constraint delivery to a separate follow-up. It answers a different question and can obscure the naming effect.
7. Do not import adjacent effect sizes as expected enum effects. Perform power analysis over a smallest effect of practical interest and simulation, not PA-Tool/Hood deltas.
8. Validate the gold policy. Independent design-system experts must agree on the concept ID for each scenario before model testing; ambiguous scenarios should permit clarification or be excluded from exact-match scoring.
9. Treat confidence carefully. Prefer token probabilities where available; self-reported confidence is secondary and not comparable across providers.
10. Blind error coding. Prior-capture labels should be assigned against pre-recorded name interpretations, with coders blind to hypothesis where feasible.

The corrected protocol is in button-enum-naming-experiment-v0.2.md.

Net change to the evidence base

The bundle does not materially change the conclusion of the existing research package. It strengthens the provenance trail, makes the enum-value gap more precise and provides a nearly usable experimental draft.

The defensible current statement remains:

Names, descriptions, schema wording and structural constraints influence related LLM tool/schema tasks. Public evidence does not yet establish how enum-value names affect design-system property selection when descriptions and behavior are held fixed.

The immediate research contribution should therefore be a controlled Button hierarchy experiment, not a production rename, semantic registry or component-contract proposal.

Open questions retained for the research stream

• What are the separate contributions of property name, enum value, description, examples and negative rules?
• Can a locally exceptional definition reliably overcome a model’s learned association, and how much context does that require?
• Do more explicit properties improve selection enough to offset search-space and invalid-combination costs?
• Does retrieval quality dominate vocabulary quality when the governing rule is missing from context?
• How do model family, language and version change the result?
• What gold standard should govern genuinely ambiguous component choices?
• When do backward-compatible aliases outperform public API changes after governance and tooling costs are included?
• Can version-matched Figma properties, Code Connect mappings and code unions be compared reliably enough to audit semantic parity?

These questions are not evidence for a naming change. They define the remaining research boundary.