The registered pipeline, in plain English
How every pattern of results will be read
The analysis plan fixes, before any data are collected, how every pattern of results will be read. Two validity criteria are applied before any agreement is scored. The per-effect human replication criterion asks whether each effect replicated in our own calibrated human sample (Holm-corrected, Bayes factor above three); an effect that misses it exits fidelity scoring and is reported as a replication result. The per-cell model data-quality criterion requires a parse rate of at least 95 percent and a refusal rate of at most 5 percent, and its ceiling rule holds out any cell where a model's baseline share leaves no room for a decoy to move the choice, so a ceiling artifact is never read as immunity to the manipulation. What survives both criteria feeds the fidelity map, the four fidelity classes, and five pattern tests whose overlaps a registered precedence rule resolves, ending in one of seven named scenarios.
Figure 1 as an interactive flow
The pre-registered decision tree
The whole plan as one flow: the validity criteria on the left, the fidelity map at the center with its four classes above it, the five pattern tests, the precedence rule, and the seven scenarios on the right. Click any node for its registered reading; highlight an account or a scenario to light its route. Drag to pan, scroll to zoom, or use the controls; the keyboard works too (arrows pan, +/− zoom, 0 fits).
Every branch ends in a reading fixed before any data are collected. Accounts are attributed per effect; patterns consistent across model families license class-level statements, while family-specific patterns are reported per family and label the aggregate map heterogeneous.
Step 1 · the benchmark the models are judged against
Human benchmarks
For every effect, in every product category, memorization tier, and format, the analysis estimates the effect size with its uncertainty interval from the registered statistic for that effect. Alongside every full-sample estimate it computes a first-trial-only estimate (a one-observation-per-person experiment embedded in the within-person design) as a registered comparability check. The pooled rehearsal estimates below show the shape of that table; the values are rehearsal output, carried only to show the shape.
| Effect | Tier | Effect (pp) | 95% interval | First-trial (pp) | Replicated? |
|---|
Rehearsal values, pooled across categories, numbers format. Every estimate and test on the human side uses cluster-robust standard errors with each participant as one cluster, and all confirmatory tests are Holm-corrected within families of effects. Model effect sizes carry no interval by design: a model effect is a population quantity computed over the fully enumerated grid of position orders, paraphrases, and personas, so the model-versus-human deviation is a descriptive distance, never a test statistic.
Table 3 · the composite verdicts
The seven pre-registered scenarios
The validity criteria, classes, and pattern tests compose into a small number of overall stories the data are likely to tell, and the protocol names seven in advance, each with a one-line registered reading. They add no new tests; they are the composite verdicts the pieces assemble, and naming them in advance is how the analysis shows it has a reading ready for each realistic way the results can fall.
A model matching none of the seven is routed by the individual test readings under the precedence rule and recorded for every model in the tree-routing file. Web Appendix C works two deliberately contradictory patterns end to end, to show that the tree decides each case and does not defer.
The design in one paragraph
What produces the numbers
Seven effects across 13 product categories and three memorization tiers (exact published values, new values in published categories, and invented categories that cannot sit in any training corpus) give 165 registered stimuli. On the human side, about 19,000 participants (including the completed pre-test samples) contribute roughly 104,200 responses, 93,100 of them confirmatory choices across 123 cells, with every adjustable two-option baseline calibrated by the completed three-wave pre-test before any confirmatory choice is collected. On the model side, ten engines (an engine is one model in one configuration, pinned to a dated snapshot) answer the same scenarios under a designed grid of position orders, three fixed paraphrases, and twenty panel-matched personas, 907,594 calls in all, with a separate recognition-probe arm that never shares a session with a behavioral choice. A format module re-runs the top of the argument-strength ordering with attributes as numbers, words, and pictures (the pictures format in three categories, after the hard-drives pictures cells were removed at calibration), and a price-free arm re-runs the attraction family with no price attribute at all.