We Tested Everything Across 4 Languages. Most Python Advice Doesn’t Transfer. (21,600 Benchmarks)

In April, we ran our capstone benchmark: 6,480 Python benchmarks showed the entire prompt optimization window was 1.4 points. Kitchen-sink instructions were the worst performer. Code-reviewer persona plus polite framing was the winner. The conclusion was clear: less is more.

Then we ran 21,600 benchmarks across 4 languages. The story changed completely.

Kitchen-sink is now the overall winner. It beats bare prompts by 4.53 points across all languages. But Python barely moves: kitchen-sink Python scores 89.09 vs bare 88.32, a lift of only 0.77 points. The capstone v1 finding was correct FOR PYTHON. But Python is the exception, not the rule.

The lift comes from Go (+6.76), JavaScript (+7.25), and C# (+3.36). Languages with low bare baselines respond dramatically to instruction. Languages with high baselines don’t. The model’s baseline competence in a language predicts how much prompting will help.

What Changed

We ran 10 prompt variants designed to close every presentation gap from capstone v1:

• bare: No instructions beyond the task prompt (the control)
• kitchen-sink: Comprehensive coding rules (Python-flavored: snake_case, docstrings, error handling, type hints)
• python-js-stack: Code-reviewer persona + polite framing (the Python optimal stack from capstone v1)
• go-cs-stack: CoT prefix only (the Go/C# optimal stack from cross-language revisits)
• outcome-focused: “Produce correct, well-structured code” (WHAT not HOW)
• implementation-focused: “Use early returns, extract helpers, add type hints” (HOW not WHAT)
• positive-framing: Rules stated positively (“use descriptive names”)
• negative-framing: Same rules stated negatively (“avoid cryptic names”)
• persona-plus-cot: Code-reviewer persona + CoT prefix combined
• verify-on-stack: Python-js-stack + “trace 3 examples” verification

Each variant was tested across:

• 4 languages: Python, Go, JavaScript, C#
• 12 tasks per language = 48 total tasks
• 3 models: Haiku, Sonnet, Opus
• 15 repetitions per model-task-variant combination

That’s 10 variants × 4 languages × 12 tasks × 3 models × 15 reps = 21,600 runs.

This is the definitive cross-language experiment in our benchmark series, building on chain-of-thought, politeness sweep, persona sweep, context pollution, cross-language revisits, verification instructions, multi-turn conversation, language kitchen-sink, targeted constraints, init vs best-practices, emotional stakes, and output compression. All data is open source at claude-benchmark.

The Scoreboard

Here are the full results by variant and language:

The spread from best to worst OVERALL: 4.53 points (kitchen-sink 76.89 vs bare 72.36). But the per-language spreads tell a very different story:

• Python: 1.78 points (kitchen-sink 89.09 vs go-cs-stack 87.31)
• Go: 8.21 points (kitchen-sink 76.75 vs python-js-stack 68.54)
• JavaScript: 7.64 points (kitchen-sink 74.65 vs bare 67.40, python-js-stack drops to 67.01)
• C#: 10.13 points (outcome-focused 71.95 vs python-js-stack 61.82)

Python’s prompt optimization window is less than 2 points. C#’s window is over 10 points. The language matters more than the prompt.

Wait — Kitchen-Sink Is Winning?

The answer: capstone v1 tested Python only. Kitchen-sink Python scores 89.09 vs bare Python 88.32, a lift of only 0.77 points. The “kitchen-sink is worst” finding was correct for Python, where the model’s bare baseline is already 88.32. The model knows Python deeply. Verbose instructions are noise.

But Go, JavaScript, and C# have much lower baselines:

• Go bare: 69.99 → kitchen-sink: 76.75 (+6.76)
• JavaScript bare: 67.40 → kitchen-sink: 74.65 (+7.25)
• C# bare: 63.72 → kitchen-sink: 67.08 (+3.36)

Kitchen-sink provides structural scaffolding that helps weaker-baseline languages. When the model doesn’t have deep native competence in a language, explicit rules (snake_case, docstrings, error handling, type hints) guide it toward better output. When the model already knows what to do (Python), those same rules are token overhead.

This is why cross-language testing matters. Python is the best-represented language in Claude’s training data. It’s also the language where prompting helps the LEAST. If you optimize your prompts on Python and assume they’ll transfer to other languages, you’re optimizing in the wrong direction.

The Baseline Competence Theory

Here’s the regression output (ordinary least squares):

Dependent variable: kitchen_sink_lift (kitchen-sink score minus bare score)
Independent variable: bare_baseline (bare prompt score)

Coefficient (baseline): -0.647
Intercept: 61.82
R²: 0.89
p-value: < 0.001

Interpretation: For every +1 point increase in bare baseline,
kitchen-sink lift decreases by 0.647 points.

This explains EVERY prior finding in the benchmark series:

• CoT helps Go/C# but not Python (cross-language revisits): Go baseline is 69.99, C# is 63.72. The model is uncertain in these languages. CoT provides reasoning scaffolding where the model lacks native competence.
• Politeness helps JavaScript (politeness sweep): JavaScript baseline is 67.40, the second-weakest. Polite framing ("please", "if possible") signals high-context collaboration, which helps when the model is least confident.
• Persona helps Python even at high baseline (persona sweep): Role framing ("You are a code reviewer") works orthogonally to baseline competence. Even high-baseline languages benefit from role context, because persona affects output style, not knowledge.
• Context pollution hurts universally (context pollution): The r=-0.95 correlation between prompt tokens and quality holds across all languages. Token overhead is a fundamental constraint, independent of baseline.
• Init vs best-practices is language-dependent (init vs best-practices): /init project context helps low-baseline languages (Go, C#) more than high-baseline languages (Python). The model uses project context to fill knowledge gaps.

The Baseline Competence Theory unifies the entire series. Prompting provides value where the model lacks native competence. That's why kitchen-sink helps Go/JS/C# but not Python. That's why CoT helps uncertainty. That's why /init helps unfamiliar codebases. The model's starting point predicts how much instruction will matter.

WHAT vs HOW Flips by Language

The outcome-focused variant ("Produce correct, well-structured code") tells the model WHAT to achieve. The implementation-focused variant ("Use early returns, extract helpers, add type hints") tells the model HOW to implement. Which works better?

It depends on the language:

C# wants WHAT (+9.45 from HOW). JavaScript wants HOW (+4.86 from WHAT). Python and Go are relatively indifferent.

Why? Two hypotheses:

1. The implementation-focused rules are Python-flavored. They mention type hints, early returns, and helper extraction in a way that maps to Python idioms. C# may resist these rules as foreign. This is a confounding variable in the experiment design.
2. Baseline competence predicts receptivity to HOW. JavaScript has the weakest baseline (67.40). It benefits most from explicit implementation guidance. C# has a slightly higher baseline (63.72) but still weak. It benefits from outcome framing because HOW instructions don't align with C# conventions. Go (baseline 69.99) and Python (baseline 88.32) are confident enough to ignore both framings.

The key insight: WHAT vs HOW is not universal. The optimal framing depends on the language and the model's baseline competence. High-baseline languages can infer HOW from WHAT. Low-baseline languages need explicit guidance, but only if that guidance aligns with the language's conventions.

Framing Direction Is Noise

The largest delta is C# at +1.33 points, well within noise. Framing direction doesn't matter. The politeness sweep found a +0.66 advantage for positive framing, but that experiment compared different content ("use descriptive names" vs "avoid cryptic names"), not just valence. When we hold content constant and flip only the framing, the effect disappears.

Implication: Don't waste time rewriting rules from negative to positive. Focus on content, not phrasing. If a rule is clear, the model will follow it regardless of whether you say "do X" or "don't do Y."

Model-Specific Results

Do different models respond differently to prompt variants?

Opus is consistent. Kitchen-sink lifts Opus from 80.04 to 80.34 (+0.30). Python-js-stack and implementation-focused both lift Opus by ~2 points. Opus responds to instruction but doesn't need it as much as Haiku does.

The model-specific results validate the Baseline Competence Theory. Haiku (weakest baseline) benefits most from instruction. Sonnet (mid-range baseline) is neutral to resistant. Opus (strongest baseline) benefits moderately. The model's native capability determines prompt responsiveness.

Per-Language Optimal Stacks

Based on 21,600 runs, here are the optimal prompt stacks for each language:

What This Means for CLAUDE.md

CLAUDE.md is the standard configuration file for AI coding assistants. Based on these results, here's how to structure it:

If your CLAUDE.md can detect the target language (via file extension or task context), you can route to language-specific stacks. But detection adds complexity. For most teams, a universal stack with role framing is the safest default.

Series Retrospective

This experiment closes a 7-month investigation spanning ~212,000 benchmark runs. Here's what changed from start to finish:

• January (chain-of-thought): "CoT prefix doesn't help Python. Skip it." → May (capstone v2): "CoT helps Go/C# significantly. Use it for low-baseline languages."
• February (politeness sweep): "Polite framing helps by +0.66 points." → May (capstone v2): "Framing direction is noise. Content matters, not valence."
• March (context pollution): "More tokens = worse code, r=-0.95." → May (capstone v2): "Still true, but low-baseline languages tolerate more tokens because instruction provides value."
• April (capstone v1): "Kitchen-sink is the worst performer. Less is more." → May (capstone v2): "Kitchen-sink is the overall winner. Less is more FOR PYTHON. Most languages need more."

The capstone v1 conclusion was correct for Python. But Python is the exception, not the rule. Claude's training data over-represents Python. The model's bare Python baseline (88.32) is extraordinarily high. Most languages have much weaker baselines and respond dramatically to instruction.

The Baseline Competence Theory unifies everything: prompting provides value where the model lacks native competence. That's why CoT helps Go/C#. That's why kitchen-sink helps JS. That's why Python is resistant. The model's starting point predicts how much instruction will matter.

Limitations

• Python-flavored kitchen-sink. The kitchen-sink variant uses Python-specific rules (snake_case, docstrings, type hints). This may explain why C# responds better to outcome-focused than kitchen-sink. A C#-flavored kitchen-sink might perform differently.
• 15 reps per cell. With 10 variants, 4 languages, 12 tasks, and 3 models, we ran 15 reps per combination = 21,600 total runs. More reps would reduce noise, but the directional patterns are clear.
• Claude models only. GPT-4, Gemini, and other model families may have different baseline competence distributions. But the r=-0.95 token/quality correlation suggests this is a fundamental LLM property.
• Single-file tasks. These are isolated coding tasks. Multi-file architectural work may respond differently to instruction, especially for low-baseline languages where context transfer matters more.
• No language-specific kitchen-sink variants. We didn't test Go-flavored kitchen-sink, C#-flavored kitchen-sink, or JS-specific rules. The Python-flavored kitchen-sink may underperform for other languages due to convention mismatch.

Try It Yourself

The capstone v2 experiment configuration and all data are open source:

pip install claude-benchmark

# Run the full 21,600-run capstone v2 experiment
claude-benchmark experiment run experiments/capstone-v2.toml

# Run a single-language subset (faster)
claude-benchmark experiment run experiments/capstone-v2-python.toml

# Generate the report
claude-benchmark report

# See the results
open results/*/report.html

You can also define your own variants in the experiment TOML and test them against your target languages. If you find a stack that beats the optimal configurations here, I'd like to see the data.

Full experiment configuration: experiments/capstone-v2.toml

Per-language configurations:

• capstone-v2-python.toml
• capstone-v2-go.toml
• capstone-v2-javascript.toml
• capstone-v2-csharp.toml

For the complete journey from "less is more" to "it depends on the language," read the full benchmark series.