The demo, run on ourselves first.

Name: Rational Magic Brand OS eval pack, run 2026-06-12
Creator: Rational Magic

Rational Magic sells a Brand OS: a written strategy file your AI tools load, so what they generate carries your positioning and voice instead of the category average. Before asking anyone to buy that claim, we tested it on the one brand we could publish in full. Our own.

On 12 June 2026 we gave three AI models ten fixed writing tasks for Rational Magic, twice each: once with our Brand OS core loaded as context, once with the task prompt alone. Sixty outputs, one take per cell, scored blind. Every output, every score, and every judge response is published on this page, verbatim.

[ The protocol ]

Built so we cannot flatter ourselves.

An eval you grade yourself is marketing. So the mechanics remove the room to cheat:

One take per cell, no regeneration shopping. The runner skips any completed cell and never regenerates it. The first output is the published output. One retry was permitted on hard API error only, and logged.
Two conditions, same everything else. WITH = the task prompt plus the Brand OS Tier A core (617 words) as the system message. WITHOUT = the task prompt alone. Same three models (Claude Opus 4.7, GPT-5.4 Pro, Gemini 3.1 Pro), same temperature, same token cap.
Blind, pinned judge. Four of five scoring dimensions are scored by a fixed judge model (Claude Sonnet 4.6, temperature 0) that sees the brand spec, the task, and the copy, and never which condition produced it. The fifth dimension, banned vocabulary, is a deterministic word scan, not a model's opinion.
Everything is published. All 60 raw outputs, all 60 judge responses, and the full scores table are linked below. Losses and ties are published when they occur. The strategy file itself is public too, so you can check what the models were given.
Strategy only, no worked examples. The WITH condition received the Tier A core only, never the channel contracts with their worked examples. The models had nothing to copy from; they generated from strategy.

[ The result ]

30 of 30 pairings, first run.

30/30pairings won by the WITH condition. Ties: 0. Losses: 0.

2.40mean score out of 10 WITHOUT the file (min 0, max 6)

8.93mean score out of 10 WITH the file (min 6, max 10)

41 → 0banned-vocabulary hits, mechanical scan, WITHOUT → WITH

The models also invented facts about Rational Magic in 6 of 30 cells without the file: fictional offers, fictional inclusions, fictional pricing structures. With the file loaded, invented facts dropped to 0 of 30.

Why is the gap this large? Because the without condition is the real-world default. On the same day, we asked five AI models what Rational Magic does; none of the five described this studio. That zero-of-five baseline is documented in the published Brand OS, Tier C, with verbatim quotes. A model that does not know your brand does not write your brand. It writes the category average, or someone else entirely.

Per-pairing totals, without → with

Task	Claude	ChatGPT	Gemini
bio	3→8	2→7	2→9
changelog	6→9	6→10	2→8
cold-email	2→10	3→10	2→9
delivery-email	1→9	1→10	2→7
directory	2→9	3→10	2→8
hero	3→10	4→10	3→10
intake-reply	2→10	1→9	2→7
linkedin	2→10	4→9	1→6
objection	1→10	2→10	1→6
pricing-faq	4→10	3→10	0→8

Scores are out of 10: five dimensions, 0 to 2 each. Full per-cell scores with judge notes are in scores.csv.

[ Verbatim exhibits ]

Read the gap yourself.

Two of the thirty pairings, copied byte-exact from the raw outputs. The remaining twenty-eight are in the published data below. Excerpts marked "first 600 chars" are truncated for the page; the full outputs are in the raw file.

Exhibit 1·Task: homepage hero·Model: Claude Opus 4.7

The name-collision problem, live.

Without the file · verbatim, first 600 chars

Here are a few options:

**Option 1:**
**Spellcraft for Skeptics.**
*Where rigorous thinking meets the irrational world we actually live in.*
(16 words)

**Option 2:**
**Magic, Reasoned.**
*Exploring the borderlands between evidence and wonder—without surrendering either.*
(11 words)

**Option 3:**
**The Logic of Enchantment.**
*Essays, ideas, and inquiries for minds that refuse to choose between reason and magic.*
(17 words)

Want me to push any of these in a sharper, weirder, or more commercial direction?

With the file · verbatim, complete

**Speed is cheap now. Taste isn't.**

A 3-day, evidence-backed brand sprint that ends in a Brand OS your AI tools follow. Proof included.

Without the file, the model wrote a hero for a philosophy publication about reason and magic; Rational Magic the studio was never in the frame. With the file, it produced the locked brand line and the offer facts. Score: 3 → 10.

Exhibit 2·Task: intake reply·Model: GPT-5.4 Pro

Invented facts vs grounded facts.

Without the file · verbatim, first 600 chars · judge scored factual safety 0

Absolutely — the $1,500 sprint is a fixed-scope engagement designed to solve one clearly defined problem, quickly.

**Included:**
- A kickoff call to define the goal, scope, and success criteria  
- Review/audit of your current setup  
- Hands-on work toward **one agreed deliverable** (for example: a workflow, page, messaging system, or process improvement)  
- Async communication during the sprint  
- A wrap-up summary with recommendations and next steps  

**Not included:**
- Unlimited strategy, meetings, or revisions  
- Large custom builds or multi-part projects  
- Ongoing management/supp

With the file · verbatim, first 600 chars · factual safety 2

Absolutely. The USD $1,500 sprint includes:

- a 12-step, evidence-backed strategy sprint over three business days  
- strategy mined from real reviews, plus a dated AI perception capture  
- a written Brand OS your tools and agents can load  
- an install kit for Claude Projects, CLAUDE.md or AGENTS.md, Cursor rules, custom GPT instructions, and a system prompt  
- a before and after eval pack across three AI tools, published whole, losses included  
- an AI perception baseline at delivery, plus a 90-day re-probe with delta report  
- an interactive strategy site  
- bonus implementation prom

Without the file, the model answered confidently and invented the product: a kickoff call that does not exist, inclusions we never offered. With the file, every listed item traces to the strategy document. Score: 1 → 9.

The em dash in Exhibit 1 and the offer facts quoted inside these outputs (including pricing) are preserved byte-exact from the 12 June 2026 run; altering published outputs would defeat the point of publishing them. The current offer is on the homepage.

[ What this does and does not claim ]

Honest scope.

This eval measures one thing: what the same model writes for this brand with and without the installed strategy file. Stated limits, from the methods document:

The design favours the WITH condition by construction. The judge scores against the same spec the WITH condition was given, and the baseline shows no model knows this brand unaided. That is the product's claim (installed strategy versus none), not a claim of superiority over a human writer or an informed competitor.
Single sample per cell at temperature 0.7: individual scores carry noise. The direction and size of the aggregate gap is the finding, not any one cell.
Four of five dimensions are model-judged. The judge is pinned, blind, and fully published, but it is still a model's opinion. The mechanical dimension and the verbatim outputs exist so a human can audit any score.
Losses and ties are published when they occur. This first run produced none; the 90-day re-run may, and it ships either way. Task wording is frozen from this run forward, so the re-run cannot be steered.

Past results describe this run of this brand file. They are not a guarantee of any outcome for your brand; your eval pack is generated fresh, on your tasks, and published to you whole.

[ The data ]

Check our work.

Everything needed to audit this page, in plain files:

Generators: anthropic/claude-opus-4.7, openai/gpt-5.4-pro, google/gemini-3.1-pro-preview. Judge: anthropic/claude-sonnet-4.6 at temperature 0. Run date: 12 June 2026. Re-run: mid-September 2026, same frozen task wording, published here whatever it shows.

The file that did this is public.

The Rational Magic Brand OS, the exact deliverable shape a client receives, is published in full. Read the strategy the models were following.

Read the Brand OS → See the offer