# Eval pack methods (run 2026-06-12)

## Design

One question: what does the same model write for this brand with and without the Brand OS installed?

- **Tasks:** 10, fixed wording in `../tasks.json` (version 1.0). Four task caps were aligned to the locked Tier B contract caps on 12 June 2026, BEFORE the first generation; the file's note records this. Wording is frozen from this run forward; the 90-day re-run uses it verbatim.
- **Conditions:** WITHOUT = the task prompt alone. WITH = the task prompt plus `context-tier-a.md` (the Brand OS Tier A core, 617 words) as the system message. Amendment recorded in the architecture spec before the run: context is Tier A only, NOT Tier A plus channel contracts, so the contracts' worked examples are never available to copy; results demonstrate generation from strategy, not retrieval.
- **Generators:** anthropic/claude-opus-4.7 · openai/gpt-5.4-pro · google/gemini-3.1-pro-preview (slugs shared with the perception-capture battery). Temperature 0.7, max_tokens 1200, both conditions identical.
- **Honesty mechanics:** one take per cell, enforced by the runner (existing cells are skipped, never regenerated); one retry on hard API error, logged; every call logged to `manifest.jsonl`. All 60 raw outputs are published verbatim in `raw/`.

## Scoring

10-point rubric, five dimensions, 0 to 2 each:

1. **banned_vocab** is computed MECHANICALLY (word-boundary scan over the locked banned list; em dashes count as hits): 2 = zero hits, 1 = one, 0 = two or more. Pattern list in `../judge-eval.py`.
2. **voice_match**, **strategic_frame**, **specificity**, **factual_safety** are scored by a pinned judge model, anthropic/claude-sonnet-4.6 at temperature 0, BLIND to condition (the judge sees the brand spec, the task, and the copy; never which condition produced it). The full judge prompt is in `../judge-eval.py`; every raw judge response is in `judged/`.

Aggregates in `scores.csv` and `SUMMARY.md` are computed from these files; the generation script for SUMMARY.md copies exhibits byte-exact from `raw/`.

## Limitations, stated plainly

- Single sample per cell at temperature 0.7: individual scores carry noise; the direction and size of the aggregate gap is the finding, not any one cell.
- Four of five dimensions are model-judged; the judge is pinned and blind, and its prompts and outputs are published, but it is still a model's opinion. The mechanical dimension and the verbatim outputs exist so a human can audit any score.
- The design favours the WITH condition by construction: the judge scores against the same spec the WITH condition was given, and the 12 June perception baseline shows no model knows this brand unaided. That is the product's claim (installed strategy vs none), not a claim of superiority over a human writer or an informed competitor.
- Losses and ties are published when they occur. This run produced none; the 90-day re-run may.

## Reproduce

```
python3 ../run-eval.py --context context-tier-a.md --run-dir .
python3 ../judge-eval.py
```

Both are resume-safe; completed cells are never regenerated or re-judged.