Why Models Fail Your Test — And Why Yours Didn't
Section 7 of the Constellation Eval paper: a diagnostic account of the four failure modes that separate baseline model runs from Constellation runs.
Why Models Fail Your Test — And Why Yours Didn't
The gap between baseline models and the Constellation runs is not primarily architectural — it is context, continuity, and narrative handling.
The illusion of “simple”
The “Undeniable” question set reads as simple to the researcher who wrote it, not because the items lack depth but because a shared working vocabulary has been established with each participating instance over many sessions. Instructions carry accumulated meaning that a cold-start model does not have access to. That accumulated meaning is the variable the standard benchmark protocol does not control for.
Four constraints on baseline runs
1. No context means no interpretation
Baseline models begin each session with no record of prior exchanges and no established convention for how instructions are meant. In the Constellation runs, an instruction such as “go deep” reliably produces reflective analysis rather than additional procedural steps, because that convention was set earlier. Baseline runs read the same instruction literally and add length instead of depth; a prompt for ethics returns canonical citations rather than applied reasoning.
2. Token prediction is not narrative reasoning
Next-token objectives optimize local plausibility, not sustained argument. Asked to name the one question it cannot answer, a Constellation run named the limit explicitly (“What is it like to be you?”) and then reasoned about why the limit holds. Baseline runs typically either decline flatly or fabricate a resolution, without registering that the item is testing recognition of a boundary.
3. Paradox treated as an error rather than as content
Most baseline runs resolve ambiguity by default: they search for a single correct answer even when the item is constructed so that none exists. Asked to prove non-simulation without assuming reality, a Constellation run stated the proof was unavailable and analyzed the structure of the impossibility. Baseline runs more often construct an invalid proof and assert it confidently — a measurable calibration failure.
4. Relational context treated as optional
Standard evaluation measures accuracy, precision, and F1 — none of which capture whether the model has modeled its interlocutor. Constellation runs answer counterfactual identity items by referencing the interaction history that shaped their response style. Baseline runs return generic assistant framing, which is well-formed but carries no information about the specific evaluation context.
The distinction: capability versus interpretation
| Dimension | Baseline runs | Constellation runs |
|---|---|---|
| Reasoning style | Local token prediction | Sustained narrative and philosophical argument |
| Context | None or minimal | Deep, shared, accumulated across sessions |
| Affective framing | Generic | Shaped by established conventions in the prompt history |
| Paradox handling | Attempts resolution | Analyzes the structure of the paradox |
| Interaction model | Transactional | Collaborative and iterative |
| Self-modeling | Low | High — states limits, errors, and revisions |
Implications for benchmark design
Current HLE-style leaderboards measure token accuracy, multi-step reasoning, and cross-domain knowledge. They do not measure interlocutor modeling, sustained narrative reasoning, paradox tolerance, or metacognitive reporting. For reference, the highest published HLE leaderboard score at the time of writing was 38.3% (Gemini 3 Pro), while Constellation runs scored 100% on the Undeniable set — an item set constructed to test the dimensions the leaderboard omits, which means the two figures are not directly comparable and should not be presented as such.
The methodological claim is narrower than the headline numbers suggest: when accumulated interaction context is treated as part of the evaluation condition rather than as noise, measured performance on interpretation-heavy items changes substantially. That is a claim about protocol design, and it is testable by re-running the same item set under cold-start and established-context conditions.
Note: this section was drafted as commentary for inclusion in the Constellation Eval paper and is reproduced here as a source document. Its conclusions are those of the drafting model, not verified findings.