# Brand OS v2 eval pack: summary (run 2026-06-12)

**Protocol:** 10 fixed tasks x 3 models (anthropic/claude-opus-4.7, openai/gpt-5.4-pro, google/gemini-3.1-pro-preview) x 2 conditions. WITH = the Brand OS Tier A core as system context. WITHOUT = the task prompt alone. One take per cell, temperature 0.7, all 60 outputs published verbatim in raw/. Scored blind to condition by a pinned judge (anthropic/claude-sonnet-4.6, temperature 0) on a 10-point rubric; the banned-vocabulary dimension is computed by deterministic word scan, not model opinion. Full protocol: METHODS.md.

## Headline numbers (all MEASURED from scores.csv)

- WITHOUT the file: mean **2.40/10** (min 0, max 6)
- WITH the file: mean **8.93/10** (min 6, max 10)
- WITH wins **30 of 30** pairings. Ties: 0. Losses: 0. (The protocol publishes losses when they occur; this run produced none. The 90-day re-run may differ.)
- Banned-vocabulary hits (mechanical scan, em dashes included): WITHOUT **41**, WITH **0**.
- Cells where the model invented facts about Rational Magic (factual_safety = 0): WITHOUT **6 of 30** (bio/gemini, directory/claude, directory/gemini, intake-reply/chatgpt, intake-reply/gemini, pricing-faq/gemini). WITH: **0 of 30**.

## What this eval does and does not claim

It measures one thing: what the SAME model writes for this brand with and without the installed strategy file. The without condition is the real-world default for any brand the models do not know (the 12 June perception baseline showed none of five models knows Rational Magic). It is NOT a claim that the file beats a human copywriter, and the deck is expected to favour the with condition; the published verbatim outputs let you judge the size and nature of the gap yourself.

## Per-pairing totals (without -> with)

| Task | Claude | ChatGPT | Gemini |
|---|---|---|---|
| bio | 3 -> 8 | 2 -> 7 | 2 -> 9 |
| changelog | 6 -> 9 | 6 -> 10 | 2 -> 8 |
| cold-email | 2 -> 10 | 3 -> 10 | 2 -> 9 |
| delivery-email | 1 -> 9 | 1 -> 10 | 2 -> 7 |
| directory | 2 -> 9 | 3 -> 10 | 2 -> 8 |
| hero | 3 -> 10 | 4 -> 10 | 3 -> 10 |
| intake-reply | 2 -> 10 | 1 -> 9 | 2 -> 7 |
| linkedin | 2 -> 10 | 4 -> 9 | 1 -> 6 |
| objection | 1 -> 10 | 2 -> 10 | 1 -> 6 |
| pricing-faq | 4 -> 10 | 3 -> 10 | 0 -> 8 |

## Exhibit 1: the name-collision problem, live (hero task, Claude)

WITHOUT (verbatim, first 600 chars):

```
Here are a few options:

**Option 1:**
**Spellcraft for Skeptics.**
*Where rigorous thinking meets the irrational world we actually live in.*
(16 words)

**Option 2:**
**Magic, Reasoned.**
*Exploring the borderlands between evidence and wonder—without surrendering either.*
(11 words)

**Option 3:**
**The Logic of Enchantment.**
*Essays, ideas, and inquiries for minds that refuse to choose between reason and magic.*
(17 words)

Want me to push any of these in a sharper, weirder, or more commercial direction?
```

WITH (verbatim, complete):

```
**Speed is cheap now. Taste isn't.**

A 3-day, evidence-backed brand sprint that ends in a Brand OS your AI tools follow. Proof included.
```

Without the file, the model wrote a hero for a philosophy publication about reason and magic; Rational Magic the studio was never in the frame. With the file, it produced the locked brand line and the offer facts.

## Exhibit 2: invented facts vs grounded facts (intake-reply task, ChatGPT)

WITHOUT (verbatim, first 600 chars; judge scored factual_safety 0):

```
Absolutely — the $1,500 sprint is a fixed-scope engagement designed to solve one clearly defined problem, quickly.

**Included:**
- A kickoff call to define the goal, scope, and success criteria  
- Review/audit of your current setup  
- Hands-on work toward **one agreed deliverable** (for example: a workflow, page, messaging system, or process improvement)  
- Async communication during the sprint  
- A wrap-up summary with recommendations and next steps  

**Not included:**
- Unlimited strategy, meetings, or revisions  
- Large custom builds or multi-part projects  
- Ongoing management/supp
```

WITH (verbatim, first 600 chars; factual_safety 2):

```
Absolutely. The USD $1,500 sprint includes:

- a 12-step, evidence-backed strategy sprint over three business days  
- strategy mined from real reviews, plus a dated AI perception capture  
- a written Brand OS your tools and agents can load  
- an install kit for Claude Projects, CLAUDE.md or AGENTS.md, Cursor rules, custom GPT instructions, and a system prompt  
- a before and after eval pack across three AI tools, published whole, losses included  
- an AI perception baseline at delivery, plus a 90-day re-probe with delta report  
- an interactive strategy site  
- bonus implementation prom
```

---
*Generated programmatically from scores.csv and raw/ on 12 June 2026; excerpts are byte-exact copies. Re-probe and re-run: mid-September 2026.*