POLARIS: Guiding Small Models to Write Long Stories¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/rishanthrajendhran/POLARIS
Area: Natural Language Generation
Keywords: Long-form Story Generation, Creative Writing, Reinforcement Learning, GRPO, LLM-as-a-Judge
TL;DR¶
POLARIS introduces a compute-efficient creative-writing RL recipe that integrates a structured 16-dimension LLM-as-judge Story Quality rubric with teacher-forced human-reference injection (HRI) inside GRPO groups, allowing a 9B model trained on 4 A100 GPUs for 48 hours to preserve narrative coherence and close length adherence across stories up to 12,000 words.
Background & Motivation¶
Reinforcement learning with verifiable rewards has unlocked dramatic reasoning breakthroughs in open-weight models, inspiring recent attempts to adapt policy optimization to long-form creative writing. However, the strongest existing creative writing recipes rely on immense compute and heavy infrastructureโsuch as 32B+ base parameter sizes, extensive domain continual pre-training, complex multi-stage reward model training pipelines, or vast corpora of synthetic generations. For smaller open models under 10B parameters, long-form creative writing remains an unforgiving hurdle: when tasked with authoring substantial narratives, small models either undershoot requested word counts severely by wrapping up prematurely, or experience rapid stylistic degradation characterized by formulaic filler, narrative amnesia, and debilitating repetition loops.
This failure mode stems from two fundamental obstacles inherent to open-ended creative reinforcement learning. First is the reward signal design dilemma: standard trained reward models compress multidimensional literary craft into an uninterpretable scalar score, require complex paired preference pipelines, and go stale as policy rollouts drift during training; meanwhile, generic benchmarks lack diagnostic score anchors grounded in narrative craft. Second is the stagnation problem in group relative policy optimization (GRPO): creative writing is inherently open-ended and devoid of binary verification rules, meaning policy rollouts within a group frequently converge to similar mediocre quality. This lack of within-group score variance leaves the policy without gradient pressure toward genuinely ambitious narrative execution.
To overcome these barriers without industrial-scale compute, small models need a training framework that delivers fine-grained literary critique without dedicated reward models, while injecting sustained high-quality demonstration anchors into policy updates. The paper's key insight is that frontier language models can serve directly as zero-training structured online judges when prompted with narrative-theoretic rubrics, and published literary stories can be injected directly into rollout groups as teacher-forced anchors whose statistics remain decoupled from the policy distribution. Core idea: train small language models for long-form creative writing using POLARIS, a lightweight GRPO recipe combining a structured 16-dimension LLM-as-judge Story Quality rubric with warmup-scaled Human-Reference Injection (HRI), enabling a 9B model to generalize robustly to stories up to three times its training length.
Method¶
Overall Architecture¶
POLARIS is designed as an accessible, compute-efficient reinforcement learning pipeline built on Group Relative Policy Optimization (GRPO), requiring only 4 NVIDIA A100 (80GB) GPUs and approximately 160 optimization steps (~48 hours of compute). The framework takes an open base model (Qwen3.5-9B) and aligns it directly on long-form literary story creation using ~1.4K promptโstory pairs.
Given a writing prompt, the current policy generates \(k=5\) candidate rollouts (incorporating synthetic reasoning traces inside thinking tags), while a teacher-forced human-written story from a professional anthology is inserted as the \((k+1)\)-th group member with identical formatting. All \(k+1\) trajectories are scored online by a frontier LLM judge (Gemini 3 Flash during training) following the anchored Story Quality rubric, producing positive and negative dimension scores backed by textual citations. These scores are combined with auxiliary penalties (self-repetition, length deviation, blank text) and gated into a composite reward. Crucially, group statistics (mean and standard deviation) are computed strictly from the \(k\) policy rollouts to maintain a genuine exploration baseline, whereas the injected human reference enters the policy gradient update via an advantage scaled by an explicit warmup schedule.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Prompt p & Anthology Reference y*<br/>Reverse-engineered prompt with length goal"] --> B["Teacher-Forced Human-Reference Injection<br/>5 policy rollouts + 1 injected human reference"]
B --> C["Structured Story Quality Evaluation<br/>16-dim narrative rubric with quote evidence"]
C --> D["Sublinear Length Scaling & Gated Penalties<br/>Dampens short-text inflation & halts degenerate text"]
D --> E["Decoupled Group Policy Optimization<br/>Baseline stats from rollouts only; warmup-scaled A*"]
Key Designs¶
1. Teacher-Forced Human-Reference Injection: open-ended creative RL stalls due to low within-group variance, resolved by injecting a teacher-forced human story as a high-reward anchor with decoupled statistics to provide sustained gradient pressure
In standard GRPO, candidate completions are sampled entirely from the current policy \(\pi_\theta\). In early training stages or on complex long-form narrative prompts, all policy rollouts often exhibit shallow characterization, repetitive tropes, or unearned plot turns, leading to compressed group variance and vanishing gradient updates. POLARIS resolves this exploration bottleneck through Human-Reference Injection (HRI). For each prompt \(p\), a human-written story \(y^*\) (under 4,000 words from a professional short-story anthology) is formatted into an assistant turn with a synthetic thinking trace generated via Gemini 3 Flash. It is teacher-forced through the policy to obtain its log-probabilities \(\log \pi_\theta(y^*|p)\) and added as the \((k+1)\)-th group member alongside \(k=5\) policy rollouts.
To preserve the policy's exploration dynamics and avoid skewing baseline normalization, HRI enforces three structural constraints: (i) the group mean \(\mu_\pi\) and standard deviation \(\sigma_\pi\) are calculated strictly over the \(k\) policy outputs, excluding the human reference; (ii) the human reference is scored by the exact same reward function as policy rollouts; and (iii) the reference advantage is scaled by a dynamic linear warmup schedule. Formally, policy rollouts receive normalized advantage \(A_i = (r_i - \mu_\pi)/(\sigma_\pi + \epsilon)\), while the injected reference receives: $\(A^*_t = \alpha_t \frac{r^* - \mu_\pi}{\sigma_\pi + \epsilon}, \quad \text{where} \quad \alpha_t = \alpha_{\max} \min\left(1, \frac{t}{T_w}\right)\)$ Here, \(t\) is the training step, peak weight \(\alpha_{\max} = 0.4\), warmup horizon \(T_w = 15\) steps, and \(\epsilon = 10^{-6}\). By scaling \(\alpha_t\) smoothly from 0 to 0.4 over the first 15 steps, HRI prevents the out-of-distribution demonstration gradient from overwhelming the early policy update, while acting as a persistent high-reward anchor that pulls the policy toward authentic human prose texture.
2. Structured Story Quality Evaluation: trained scalar reward models suffer from reward hacking and lack interpretability, resolved by querying a frontier LLM judge over 16 narrative dimensions with quote-grounded evidence
Traditional pairwise preference reward models compress all aspects of writing into a single scalar, masking whether an output improved in thematic depth or merely adopted sycophantic rhetoric. POLARIS dispenses with separate reward model training by querying an online frontier LLM judge with an extensive Story Quality rubric grounded in narrative theory and creative writing craft. The rubric comprises 6 positive global dimensions (totaling 100 points): Prompt Fulfillment (P1, 0โ15), Narrative Arc & Pacing (P2, 0โ20), Character Depth & Agency (P3, 0โ15), Voice & Stylistic Distinctiveness (P4, 0โ15), Concrete World & Scene Realization (P5, 0โ20), and Thematic & Emotional Richness (P6, 0โ15).
To explicitly suppress failure modes common in AI generation, the rubric adds 10 localizable negative penalty dimensions (summing to 151 maximum penalty points): Prompt Violation (N1), Coherence & Consistency Breaks (N2), Generic/Templated Language (N3), Over-Summary & Abstraction (N4), Over-Explanation & Paraphrase Loops (N5), Drift & Bloat (N6), Dialogue Problems (N7), Mechanical Errors (N8), Predictability & Clichรฉ (N9), and Overwrought Prose / Purple Language (N10). Two open-ended catch-all terms (Bonus \(B \in [0, 8]\) and Penalty \(D \in [0, 8]\)) capture unmodeled strengths or breakdowns. For every non-zero penalty, the judge must cite exact text quotes (\(\le 12\) words) from the story. The raw judge score is computed as: $\(s^{\text{raw}}_q = \sum_{i=1}^6 P_i - \sum_{j=1}^{10} N_j + B - D\)$ Dividing by \(Z = 108\) maps the raw score into a normalized quality metric \(\tilde{s}_q = s^{\text{raw}}_q / Z\).
3. Sublinear Length Scaling & Gated Penalties: short rollouts artificially score high by evading flaw detection, resolved by coupling sublinear length scaling with multi-component auxiliary penalties and hard gates
A subtle failure mode in rubric-based evaluations is reward inflation on truncated outputs: stories that end after 300 words have far fewer sentences exposed to negative penalties (such as N5 paraphrase loops or N6 structural drift), resulting in inflated normalized quality scores. POLARIS counteracts this using a sublinear length-scaling factor \(\ell(n)\) defined by generated word count \(n\) and target count \(n_{\text{target}}\):
$\(\ell(n) = \max\left(\ell_{\min}, \left(\frac{n}{n_{\text{target}}}\right)^\beta\right), \quad \text{with} \ \beta=0.5, \ \ell_{\min}=0.15\)$
A sublinear exponent \(\beta = 0.5\) avoids destabilizing training dynamics compared to aggressive linear length penalties while firmly depressing short evasive generations.
This scaling interacts with three auxiliary penalty components: self-repetition penalty \(r_{\text{rep}}\) (blending global 4-gram repetition, local 64-gram window repetition, and duplicate line ratios), asymmetric length penalty \(r_{\text{len}}\) (tolerating \(-15\%\) to \(+25\%\) bounds), and blank penalty \(r_{\text{blank}}\). A multiplicative gate \(g(r_{\text{rep}}, r_{\text{len}}, r_{\text{blank}})\) zeros out the positive reward entirely if the text is severely malformed or repetitive. The final clipped composite reward is: $\(r_{\text{qual}} = w_q \cdot \tilde{s}_q \cdot g(r_{\text{rep}}, r_{\text{len}}, r_{\text{blank}}), \quad r_{\text{pen}} = w_{\text{rep}} r_{\text{rep}} + w_{\text{len}} r_{\text{len}} + w_{\text{blank}} r_{\text{blank}}\)$ $\(r = \text{clip}\left((r_{\text{qual}} - r_{\text{pen}}) \cdot \ell(n), -c, +c\right)\)$ where \(w_q = 1.0, w_{\text{rep}} = 1.5, w_{\text{len}} = 0.8, w_{\text{blank}} = 1.0\), and clip bound \(c = 2.0\).
Training Configuration¶
- Base Model: Qwen3.5-9B.
- Dataset: 1,388 promptโstory pairs from 100 commercially purchased anthologies (431 authors, all stories \(\le 4,000\) words).
- Hardware & Efficiency: 4\(\times\) NVIDIA A100 (80GB) GPUs using PyTorch FSDP with gradient checkpointing, effective batch size of 8 GRPO groups (each with 5 policy rollouts + 1 human reference).
- Hyperparameters: Learning rate \(1 \times 10^{-6}\), PPO clip ratio \(0.20\), rollout temperature 1.0 / 0.8 (train/val), top-p 0.95, top-k 20, max output budget 8,192 tokens. Total training runs for 160 steps (~48 hours), costing ~\(400 in GPU compute and ~\)60 in judge API calls (Gemini 3 Flash), for a total under $500.
Key Experimental Results¶
Main Results¶
The primary evaluation uses 180 held-out prompts from unseen anthologies (30 prompts across each of six target length buckets). Buckets are grouped into in-distribution (ID: 1โ4k words), near-OOD transfer (Near: 4โ8k words), and far-OOD transfer (Far: 8โ12k words). All generations are evaluated by GPT-5.4 on Story Quality (SQ) and EQ-Bench Longform (EQ-LF, uniform aggregation). Length Adherence (LR) denotes generated words divided by requested words.
| Model | Parameters | Story Quality (All) | SQ (1โ4k ID) | SQ (4โ8k Near) | SQ (8โ12k Far) | EQ-Bench LF (All) | EQ-LF (8โ12k) | LR (All) | LR (8โ12k) |
|---|---|---|---|---|---|---|---|---|---|
| Reference | |||||||||
| Human | โ | 68.7 | 68.1 | 68.2 | 71.4 | 74.2 | 77.1 | 1.00 | 1.00 |
| Frontier Models | |||||||||
| GPT-5.4 | โ | 72.4 | 76.9 | 72.1 | 59.5 | 80.3 | 77.0 | 1.26 | 0.94 |
| Claude Opus 4.6 | โ | 61.8 | 66.4 | 59.5 | 52.7 | 73.4 | 70.4 | 1.15 | 1.09 |
| Gemini 3.1 Pro | โ | 57.5 | 58.7 | 55.7 | 57.1 | 67.0 | 64.2 | 0.99 | 0.57 |
| Gemini 3 Flash | โ | 48.5 | 52.9 | 39.9 | 52.6 | 63.2 | 59.9 | 0.88 | 0.37 |
| Open-Weight Models | |||||||||
| LongWriter-Zero-32B | 32B | -27.1 | -16.8 | -35.5 | -41.2 | 26.1 | 21.4 | 2.21 | 1.03 |
| Gemma 4 31B | 31B | 51.4 | 53.9 | 49.7 | 47.1 | 61.5 | 57.7 | 0.68 | 0.36 |
| Qwen3.5-27B | 27B | 42.8 | 51.5 | 38.7 | 24.6 | 57.5 | 46.7 | 0.97 | 0.82 |
| Gemma 4 26B-A4B | 26B (4B act) | 51.3 | 52.9 | 51.4 | 46.5 | 58.1 | 52.8 | 0.65 | 0.26 |
| DeepSeek-R1-Distill-14B | 14B | 0.2 | 7.3 | -4.7 | -11.1 | 34.6 | 33.7 | 0.75 | 0.23 |
| Ministral-3-8B | 8B | 26.1 | 30.2 | 22.0 | 21.8 | 39.1 | 33.3 | 0.52 | 0.23 |
| LongWriter-Llama3.1-8B | 8B | -2.0 | 7.2 | -10.5 | -12.7 | 33.0 | 29.4 | 1.22 | 0.71 |
| Qwen3.5-9B (Base) | 9B | 18.5 | 35.1 | 8.7 | -11.8 | 42.6 | 30.3 | 1.01 | 0.88 |
| POLARIS-9B (Ours) | 9B | 52.1 | 57.4 | 48.2 | 44.1 | 59.8 | 54.3 | 0.90 | 0.72 |
Ablation Study & External Benchmarks¶
The table below compares POLARIS-9B against matched training variants (SFT baseline and GRPO without HRI), as well as out-of-distribution benchmarks: WritingBench (D4 English literature subset, Gemini 3.1 Pro judge), LongBench-Write (\(\bar{S}\) overall score), and EQ-Bench Creative. In addition, blinded human evaluation results over 60 promptโgeneration pairs are summarized on the right.
| Model / Recipe | Story Quality (All / 8โ12k) | EQ-Bench Longform (All / 8โ12k) | WritingBench (D4) | LongBench-Write (\(\bar{S}\)) | EQ-Bench Creative | Human Winrate vs. Base 9B (W / T / L) | Human Winrate vs. 27B (W / T / L) |
|---|---|---|---|---|---|---|---|
| Qwen3.5-9B (Base) | 18.5 / -11.8 | 42.6 / 30.3 | 6.84 | 67.1 | 59.2 | โ | โ |
| + SFT Baseline | -60.6 / -71.2 | 26.0 / 24.2 | 2.68 | 37.5 | 26.7 | โ | โ |
| + Matched GRPO (No HRI) | 49.7 / 37.7 | 58.2 / 51.6 | 7.85 | 82.1 | 69.7 | โ | โ |
| + POLARIS (with HRI) | 52.1 / 44.1 | 59.8 / 54.3 | 7.90 | 81.2 | 70.3 | 67.5% (57.5% / 20.0% / 22.5%) | 51.2% (35.0% / 32.5% / 32.5%) |
Key Findings¶
- Three Breakdown Modes in Long Open-Weight Models: When pushed to 8โ12k requested words, baseline models fail in three distinct ways: quality collapse at target length (Qwen3.5-9B drops to -11.8 SQ), runaway repetition (LongWriter-Zero-32B averages 58.4% duplicate lines, scoring -27.1 SQ), or severe under-generation (Gemma 4 31B drops to LR 0.36 to preserve short-text quality). POLARIS-9B is the sole open-weight model that avoids all three, maintaining LR 0.72 and strong quality (44.1 SQ) at 8โ12k words.
- SFT Collapse vs. HRI Advantage at Far Transfer: SFT on the same story corpus collapses completely (-60.6 SQ), producing bloated, low-substance outputs. While matched GRPO achieves solid in-domain scores, HRI provides substantial improvements at far-transfer lengths (8โ12k SQ increases from 37.7 to 44.1, a +6.4 point margin), showing that teacher-forced human anchors prevent policy collapse when extrapolating far beyond the 4k training length.
- Blinded Human Evaluation Parity with 27B: In a rigorous blinded study with professional editors, POLARIS-9B scored a decisive 67.5% winrate over the base Qwen3.5-9B (95% CI [55.0, 80.0]) and was statistically tied with Qwen3.5-27B (51.2% winrate, 95% CI [41.2, 61.2]). Annotators commended POLARIS-9B for vivid atmospheric realization, distinct character voices, and faithful premise execution.
Highlights & Insights¶
- Decoupled Demonstration Anchoring: Injecting an expert reference into a GRPO group while computing advantage baselines strictly from policy rollouts provides a principled mechanism to prevent reward stagnation in soft-reward RL, without perturbing exploration variance.
- Multidimensional Structural Rubric as Reward: Decomposing narrative evaluation into 6 global positive achievements and 10 localized negative penalties with mandatory quote citations eliminates the need for expensive, brittle reward model training and directly penalizes purple prose and boilerplate writing.
- Sublinear Length Scaling Defense: Applying a sublinear factor \(\ell(n) = \max(\ell_{\min}, (n/n_{\text{target}})^\beta)\) seamlessly disincentivizes models from gaming rubric scores via premature completion, balancing narrative density with target length fidelity.
Limitations & Future Work¶
- Stylistic Overloading & Local Coherence Strains: Human raters observed that POLARIS-9B occasionally tries too hard, stacking jargon or strained metaphors and incurring minor local timeline contradictions across scene transitions.
- Single Literary Genre Domain: Training exclusively on published short-story anthologies means the model's gains do not transfer as strongly to non-narrative writing tasks such as biography, academic book reviews, or essayistic analysis.
- Far-Length Undershoot: While outperforming baselines at 8โ12k words, POLARIS-9B still reaches only a 0.72 length ratio, leaving headroom for explicit multi-act planning or recursive generation strategies.
Related Work & Insights¶
- vs. LongWriter-Zero-32B: LongWriter-Zero relies on a 32B model and trained scalar reward models, suffering from massive line duplication and plot drift on creative prompts; POLARIS achieves superior narrative quality with a 9B model at a fraction of the compute.
- vs. Off-Policy RL (LUFFY / BREAD): Prior off-policy injection methods focus on mathematical reasoning with partial expert prefixes; POLARIS generalizes this to full-sequence teacher-forced literary demonstrations in soft-reward policy optimization.
- vs. DPO / Direct Alignment: Pairwise preference optimization collapses holistic literary craft into binary outcomes; POLARIS's online multidimensional rubric delivers targeted credit assignment across plot, voice, and pacing.
Rating¶
- Novelty: โญโญโญโญโญ (Elegant integration of teacher-forced human anchors into GRPO and quote-grounded LLM narrative rubrics)
- Experimental Thoroughness: โญโญโญโญโญ (Evaluated across 17 models, 5 benchmarks, 160K+ pairwise judgments, and blinded human evaluation)
- Writing Quality: โญโญโญโญโญ (Exceptional clarity, rigorous mathematical framing, and thorough analysis of long-text failure modes)
- Value: โญโญโญโญโญ (Provides a highly practical, reproducible, low-compute recipe for democratizing long-form creative writing)