How We Test Whether SKU.md Helps AI Find Better-Matched Products
v1.0.0 features
A preregistered A/B benchmark design for comparing ordinary product pages with SKU.md under matched retrieval budgets, blind review, and reproducible artifacts.
Status: This article reflects the current v1.0.0-beta boundary. It reports a method and its evidence requirements, not unverified adoption, ranking, citation, recommendation, or sales outcomes.
A benchmark should answer a narrow causal question: under matched conditions, does access to SKU.md improve product matching for a defined set of tasks? It should not be designed to produce a favorable launch number. This article publishes the protocol now; it does not report a result until the complete run can be reproduced.
Preregister the hypothesis
Write the product sample, shopper questions, expected constraints, scoring rules, models, retrieval budget, prompts, and exclusion rules before observing results. The main comparison uses two conditions: ordinary public product resources, and the same resources plus SKU.md. Everything else stays fixed.
Define what would count as no improvement or harm. A benchmark that cannot disconfirm its claim is marketing material.
Build a representative question set
Include exact lookup, attribute comparison, compatibility, use-case guidance, limitation-sensitive requests, identity and variant questions, and live-state questions that should trigger a handoff. Sample both well-documented and sparse products. Keep hidden evaluation items so authors cannot tune every document to the test.
Annotators should record the hard constraints, acceptable alternatives, decisive sources, and whether refusal is expected.
Score more than retrieval
Measure whether the correct candidate was retrieved, whether decisive facts are correct, whether cited sources support them, whether limitations are respected, and whether the system refuses or asks when evidence is insufficient. A single top-choice accuracy score can hide unsafe explanations.
Report confidence intervals or per-item results when the sample is small. Do not convert a benchmark delta into a promise about rankings, citations, conversion, or every model.
Blind the review
Remove condition labels and presentation clues before human scoring. Use at least two reviewers for subjective suitability cases, define disagreement resolution, and retain reviewer notes. Automated checks can score exact identifiers and source URLs; nuanced recommendations still need a documented rubric.
Preserve a reproducibility bundle
| Artifact | Required contents |
|---|---|
| Corpus | exact pages and SKU.md versions or snapshots |
| Queries | prompts, constraints, expected action |
| Runtime | model, consumer, dependencies, configuration |
| Randomness | seeds and sampling settings |
| Outputs | raw answers, retrieved resources, citations |
| Scores | rubric, reviewer decisions, scripts |
| Failures | exclusions, timeouts, malformed cases |
Publish failures beside averages
Show cases where SKU.md made no difference, introduced stale or conflicting knowledge, or caused an incorrect match. These examples reveal which specification or authoring changes matter next. Aggregate improvement without failure analysis is hard to trust.
Reproducibility threshold
Publish a benchmark result only after a second person can run the documented procedure and reproduce the reported tables within the stated tolerance. Until then, the article describes the protocol and should not contain fabricated results.
Fix the comparison before running it
The baseline and SKU.md condition must use the same product set, shopper questions, model or retrieval system, token budget, tools, and live-state policy. The only intended difference is the availability of valid SKU.md knowledge. Freeze document snapshots and record versions, prompts, seeds where supported, timeouts, and failures. If the conditions receive different crawl budgets or manually corrected candidates, the comparison no longer isolates the file.
Questions should cover direct facts, compatibility, limitations, variant identity, ambiguous intent, missing evidence, and requests that require live commerce. Include near-duplicate products and cases with no correct recommendation. A benchmark composed only of easy positive matches rewards verbosity and does not test refusal or exclusion.
Score the path as well as the final product
Product recall or top-k accuracy shows whether the right candidate was found, but it is insufficient. Score factual accuracy, source accuracy, hard-constraint satisfaction, false-positive recommendation rate, refusal quality, and whether live questions were handed off. Record latency and resource use separately so a quality gain is not hidden behind an unbounded retrieval budget.
Define grading rubrics before seeing outputs. A cited source counts only if it supports the exact claim. A refusal is correct only when the evidence is genuinely insufficient or no candidate satisfies the constraints. A plausible explanation with a wrong variant identifier remains wrong.
Blind review and publish failures
Randomize condition labels before human review and use at least two reviewers for subjective judgments. Resolve disagreements under a written rubric and publish agreement statistics. Keep merchant and product identifiers when licensing permits, or publish stable fixtures that preserve the decision structure.
The reproducibility bundle should contain inputs, snapshots, configuration, raw outputs, scoring code, adjudication records, and summary tables. Publish category-level results and representative failures beside averages. A result should be reported only after an independent person can rerun the package and obtain results within the stated tolerance. Until then, this article describes a protocol, not a positive result.
Prevent contamination and selective reporting
Benchmark authors should not tune SKU.md documents against the held-out evaluation questions. Split questions into development and evaluation sets, document any editorial changes prompted by development errors, and keep the held-out set sealed until the protocol is frozen. If a run is repeated after inspecting results, label it as a new run instead of replacing the unfavorable one.
Report every planned metric, even when the effect is neutral or negative. Confidence intervals or repeated-run variation matter when model output is stochastic. Missing outputs, tool failures, and budget exhaustion belong in denominators under rules written before execution. Otherwise the treatment can look better simply because difficult cases disappeared.
Interpret outcomes within the tested boundary
A gain in source accuracy with unchanged product recall may still be valuable; a gain in recall paired with more incompatible recommendations may be harmful. Segment results by question class and inspect the trade-offs. No single number determines whether the contract is useful.
Even a strong benchmark supports a bounded statement about this dataset, consumer, model, and version. It does not establish platform adoption, search rank, or conversion. The public package should make that boundary obvious enough that later writers cannot quote the result as a universal guarantee.
The benchmark should also record editorial effort. If SKU.md performs better only after hours of question-specific rewriting, that cost belongs beside the quality result. Track document preparation time, number of claims added, and validation failures. A useful standard should make maintained knowledge clearer, not merely move hidden tuning labor into a new file. Publishing this cost helps merchants judge whether the measured gain is operationally realistic.
Before publication, run a leakage audit. Check whether evaluation answers appear verbatim in prompts, filenames, document summaries, or scoring metadata; verify that reviewers cannot infer the condition from formatting; and confirm that both groups receive identical failure handling. Preserve this audit with the bundle. Small leaks can produce a convincing advantage without improving product understanding, so a result review should treat them as blocking defects rather than footnotes.
The published record must include the complete run.