LLM Creative Writing Benchmark: How Models Score on Prose
An LLM creative writing benchmark scores how well a model writes prose, but unlike coding or math, there is no test suite that passes or fails. Evaluators substitute proxies: a model judge grading against a rubric, pairwise human votes converted to Elo, or hand counts of repetition and cliché. EQ-Bench’s Creative Writing v3, for example, has a judge grade 96 generated pieces against a rubric and then ranks by Elo (Source: EQ-Bench, 2026).
That design choice is the whole story. Creative quality is subjective, so every benchmark has to define “good” before it can measure it, and each definition smuggles in assumptions and biases. This post walks through how the leading efforts actually work: EQ-Bench’s Creative Writing leaderboard, LMArena’s creative writing category and its style-control adjustment, the LLM-as-a-judge method and its documented biases, and old-fashioned human evaluation. Because standings shift weekly, every live ranking here points to its source and access date rather than a frozen number.
What is an LLM creative writing benchmark?
An LLM creative writing benchmark is a structured test that ranks language models by the quality of fiction, poetry, or stylistic prose they produce. Because “quality” cannot be unit-tested, these benchmarks rely on graded rubrics, pairwise comparison, or human preference rather than the exact-match scoring used for math and code (Source: EQ-Bench, 2026).
The contrast with technical benchmarks is sharp. A coding benchmark like SWE-bench checks whether generated code passes tests; the answer is binary and reproducible. Creative writing has no ground truth. A model can write a story that one reader calls evocative and another calls overwrought, and both are defensible. So creative benchmarks measure agreement with a defined standard, not correctness. For how this fits the wider field, see Examples of LLM Benchmarks. The headline takeaway: a creative score is a measurement of one evaluator’s taste, formalized, not an objective fact about the prose.
Why is creative writing the hardest thing to benchmark?
The maintainers of EQ-Bench put it bluntly: “There has been a distinct lack of automated benchmarks for creative writing because it’s hard to assess writing quality without humans in the loop” (Source: EQ-Bench, 2026). Asking a model “how good is this writing, 0-10?” produces noisy, poorly calibrated answers. Good prose also resists decomposition. A scoring rubric can rate grammar, coherence, and imagery, but the parts that make writing memorable, voice, surprise, restraint, are exactly the ones that resist a checklist.
How is creative writing benchmarked?
Three methods dominate, and most serious efforts combine them. A model judge grades outputs against a rubric; humans or models vote in head-to-head matchups that feed an Elo rating; and human experts read samples directly. EQ-Bench’s Creative Writing v3 fuses the first two, pairing a rubric score with a more discriminative Elo from pairwise comparisons (Source: EQ-Bench, 2026).
Each method trades reliability against cost and scale. Understanding which one a leaderboard uses tells you more about what its numbers mean than the numbers themselves.
How does judge-based (LLM-as-a-judge) evaluation work?
In LLM-as-a-judge evaluation, a strong model reads each generated piece and scores it against a written rubric. EQ-Bench’s Creative Writing v3 has the model under test generate responses to 32 prompts across 3 iterations (96 pieces), then has a judge, Anthropic’s Claude Sonnet recommended for leaderboard parity, grade each piece against a detailed rubric (Source: EQ-Bench, 2026). It is cheap, fast, and reproducible, which is why it scales. The catch is that the judge imports its own preferences, a problem we return to below.
How does arena-style pairwise voting work?
Arena methods skip rubrics and ask which of two anonymous responses is better. LMArena (formerly Chatbot Arena) collects millions of such votes, models the results with a Bradley-Terry / Elo system, and publishes a dedicated creative writing category that “evaluates a model’s ability to craft original, imaginative, emotionally resonant content” (Source: LMArena, 2026). Pairwise comparison sidesteps the calibration problem of absolute scores, but it inherits voter biases toward length and formatting.
How does human evaluation fit in?
Human evaluation, expert readers rating or ranking outputs, remains the reference standard the automated methods try to approximate. It is also the bottleneck: slow, expensive, and hard to reproduce, since reviewers disagree and tire. EQ-Bench’s own guidance concedes the limit of automation, advising users to “always supplement scores by reading sample outputs and forming your own judgment” (Source: EQ-Bench, 2026). Most leaderboards therefore use automated scoring for scale and reserve human reads for spot checks.
Which creative writing benchmarks exist, and how do they differ?
No single benchmark owns creative writing; each picks a method and inherits its biases. The table below maps the major 2025-2026 efforts to how they score and what skews their results. Treat every standing as live: EQ-Bench and LMArena both update continuously, so read the boards directly rather than trusting a cached rank (Source: EQ-Bench; LMArena, 2026).
| Benchmark | Method | Known biases / limits | Source |
|---|---|---|---|
| EQ-Bench Creative Writing v3 | LLM judge (Claude Sonnet) on rubric + pairwise Elo over 96 pieces | Judge taste; rubric can saturate at the top; Elo depends on model pool | EQ-Bench, 2026 |
| EQ-Bench Longform Writing | LLM judge (Claude Sonnet 4.6) on planning, revision, full short stories | Judge bias; long outputs costly; few prompts | EQ-Bench, 2026 |
| EQ-Bench Slop Score | Counts overused “GPT-ism” words/phrases vs a master slop list | Measures cliché frequency, not quality; gameable | EQ-Bench, 2026 |
| LMArena creative writing category | Crowd pairwise votes, Bradley-Terry / Elo | Length and markdown bias; self-selected voters | LMArena, 2026 |
| LMArena style-control view | Same votes, regression that controls length + markdown | Reduces but does not remove style effects | LMArena (Li et al.), 2024 |
| EQ-Bench Judgemark v4 | Meta-eval: scores how well a model judges fixed samples | Tests judging skill, not writing skill | EQ-Bench, 2026 |
For a broader cross-method view of how leaderboards are built and compared, see LLM Benchmark Comparison.
How does EQ-Bench score creative writing?
EQ-Bench’s Creative Writing v3 uses a hybrid design built for discrimination at the top. The model under test writes 96 pieces (32 prompts x 3 iterations) at temperature 0.7, a judge grades each against a rubric, and the leaderboard ranks by a normalized Elo from pairwise comparisons, with the rubric score reported alongside (Source: EQ-Bench, 2026).
The two-metric split exists because each metric fails differently. The rubric score is interpretable but saturates: once strong models all score near the ceiling, it stops separating them. The Elo score, derived from head-to-head matchups, stays discriminative at the top but is “sensitive to the pool of compared models,” so a model’s rating can shift when the field changes (Source: EQ-Bench, 2026).
What bias controls does EQ-Bench apply?
EQ-Bench builds in defenses against the best-known judging biases. To curb length bias it truncates outputs to 4,000 characters before judging; to curb position bias it runs each pairwise comparison in both A/B and B/A orders and averages the results (Source: EQ-Bench, 2026). It also publishes separate Slop and Repetition columns, the Slop Score counting overused LLM phrases against a master list, so a model cannot hide cliché behind a high rubric grade. These controls reduce, but do not eliminate, the underlying problem that one model is grading another’s taste.
How does LMArena’s creative writing leaderboard work?
LMArena ranks models by human preference: users send a prompt, see two anonymous answers, and vote for the better one. Those votes feed a Bradley-Terry / Elo model, and a dedicated creative writing category isolates imaginative tasks. The overall text arena had logged more than 7 million votes across 360-plus models by mid-2026 (Source: LMArena, 2026).
Crowd voting captures real human preference at a scale no expert panel can match. But it rewards more than substance. The LMArena team found that “pairwise voting reliably rewards longer responses with bullet points and markdown formatting, even when content quality is similar” (Source: LMArena / Li et al., 2024). Some models had even drifted toward long, structured outputs partly because that style won votes, distorting the board.
What does style control change?
Style control is LMArena’s fix for that distortion. Introduced in November 2024, it uses logistic regression layered on the Bradley-Terry model to “control” four measurable style features, answer length, and the counts of markdown headers, bold elements, and list items, mathematically removing their influence to isolate content quality (Source: LMArena / Li et al., 2024). The reshuffle was real: under style control, lightweight models that leaned on formatting dropped, while substance-heavy models like Claude 3.5 Sonnet, Opus, and Llama-3.1-405B rose substantially (Source: LMArena / Li et al., 2024). When you read the creative writing board, switch on style control before drawing conclusions.
Who currently leads the creative writing board?
Frontier models from Anthropic, OpenAI, and Google trade the top creative writing positions, and the order changes week to week as new releases land and votes accumulate. Because these standings are volatile and model availability shifts with releases and regional restrictions, this post deliberately does not freeze a ranking. Check the live creative writing category with the style-control toggle enabled at lmarena.ai, accessed 2026-06-28, for the current order (Source: LMArena, 2026).
Why are LLM judges biased, and does it matter?
Every automated creative benchmark that uses a model judge inherits that judge’s biases, and the research literature has cataloged them precisely. The three best-documented are self-preference, position, and verbosity bias, each shown to shift judgments independently of actual quality (Source: arXiv:2410.21819; arXiv:2406.07791, 2024-2025).
These are not edge cases. They change rankings, which is why EQ-Bench truncates length and swaps positions, and why LMArena added style control.
What is self-preference bias?
Self-preference bias is a judge favoring outputs that resemble its own. A 2024 study measuring eight models found that “GPT-4 exhibited a significant self-preference bias,” and crucially, judges showed it “although LLM evaluators are not explicitly informed whether a given text is their own” (Source: arXiv:2410.21819, 2024). The proposed mechanism is familiarity: judges prefer low-perplexity, familiar-looking text over higher-perplexity prose that may be objectively better. For creative writing, that quietly penalizes the unusual phrasing good fiction depends on.
What are position and verbosity bias?
Position bias is the tendency to favor whichever answer appears first or second. A systematic study found it “is not due to random chance and varies significantly across judges and tasks,” and is “strongly affected by the quality gap between solutions” (Source: arXiv:2406.07791, 2024). The standard fix is to randomize or swap order and average. Verbosity bias is the preference for longer answers regardless of quality, the same skew length-control regressions target in AlpacaEval 2.0 and LMArena. Both inflate scores for reasons that have nothing to do with whether the writing is any good.
What does “good” creative writing even mean?
This is the contested core, and no benchmark resolves it. Each one encodes a definition of quality, a rubric’s criteria, a crowd’s average taste, a slop list’s idea of cliché, and then measures against that definition. EQ-Bench captures the honest limit: even with a strong judge, “it’s not immediately obvious how to formalise the assessment of creative writing objectively” (Source: EQ-Bench, 2026).
The practical consequence is that benchmarks measure different things and will disagree. A model can top a rubric board for clean, competent prose while a human reader finds it bland, exactly the failure the Slop Score tries to expose. The reasonable stance is to treat creative scores as directional signals, not verdicts. They tell you which models are in contention; reading samples in your own genre tells you which one writes the way you need. For how this compares to scoring technical skills, see Best LLM Benchmark, and for raw generation speed when you draft at length, TokenDyno tracks live output tokens/sec.
Frequently asked questions
What is the best LLM for creative writing?
There is no fixed answer; frontier models from Anthropic, OpenAI, and Google trade the lead and rankings shift weekly. Check EQ-Bench’s Creative Writing leaderboard and LMArena’s creative writing category (style control on), both accessed 2026-06-28, then read sample outputs in your genre before choosing (Source: EQ-Bench; LMArena, 2026).
How is creative writing benchmarked?
Three methods dominate: an LLM judge grading outputs against a rubric, pairwise voting converted to an Elo rating, and direct human review. EQ-Bench’s Creative Writing v3 combines a Claude Sonnet judge with pairwise Elo over 96 generated pieces, while LMArena ranks by crowd votes (Source: EQ-Bench; LMArena, 2026).
Can you objectively score AI writing?
Not fully. Creative quality is subjective, so benchmarks measure agreement with a defined standard, a rubric, a crowd, a slop list, rather than an objective truth. EQ-Bench’s maintainers note it is hard “to formalise the assessment of creative writing objectively,” and advise reading samples yourself (Source: EQ-Bench, 2026).
Why do creative writing benchmarks disagree?
Because they encode different definitions of “good” and inherit different biases. A rubric judge rewards clean structure but can saturate; crowd voting rewards length and markdown unless style-controlled; slop scoring counts cliché, not quality. Each measures a real but partial signal, so rankings legitimately diverge (Source: EQ-Bench; LMArena, 2026).
What biases affect LLM judges of creative writing?
The three best-documented are self-preference (favoring familiar, low-perplexity prose), position bias (favoring first or second placement), and verbosity bias (favoring longer answers). Studies show GPT-4 exhibits significant self-preference, and position bias varies by judge and task (Source: arXiv:2410.21819; arXiv:2406.07791, 2024).
Sources
- EQ-Bench, Creative Writing v3 and leaderboard suite (2026): https://eqbench.com/creative_writing.html
- EQ-Bench, Creative Writing Benchmark methodology and source code (2026): https://github.com/EQ-bench/creative-writing-bench
- EQ-Bench, Longform Writing Leaderboard (2026): https://eqbench.com/creative_writing_longform.html
- LMArena, creative writing category and live text leaderboard (2026): https://lmarena.ai/leaderboard/text/creative-writing
- LMArena, Chatbot Arena categories overview (2026): https://news.lmarena.ai/arena-category/
- Li et al., “Does Style Matter?” style-control methodology, LMArena (2024): https://www.lmsys.org/blog/2024-08-28-style-control/
- Chiang et al., “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference,” arXiv:2403.04132 (2024): https://arxiv.org/abs/2403.04132
- “From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline,” arXiv:2406.11939 (2024): https://arxiv.org/abs/2406.11939
- Wataoka et al., “Self-Preference Bias in LLM-as-a-Judge,” arXiv:2410.21819 (2024): https://arxiv.org/abs/2410.21819
- Shi et al., “Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge,” arXiv:2406.07791 (2024): https://arxiv.org/abs/2406.07791