DVMap: Fine-Grained Pluralistic Value Alignment via High-Consensus Demographic-Value Mapping

Conference: ACL 2026
arXiv: 2605.14420
Code: https://github.com/EnlightenedAI/DVMap
Area: LLM Alignment / Values
Keywords: Pluralistic Value Alignment, Demographic Archetype, Structured CoT, GRPO, Cross-cultural Generalization

TL;DR

DVMap shifts LLM "pluralistic value alignment" from coarse-grained country labels to 11-dimensional demographic attribute profiles. By filtering 56,000 WVS data points via "high-consensus profiles" (Shannon entropy \(H=0\)) and training Qwen3-8B with Structured CoT + GRPO (binary reward), the model outperforms DeepSeek-v3.2 and matches GPT-4o in triple generalization tests (cross-demographic / cross-country / cross-value).

Background & Motivation

Background: Mainstream LLM value alignment relies on either RLHF (Bai et al. 2022, Rafailov 2023) or "prompt engineering + multi-cultural fine-tuning." Label hierarchies typically stop at "Country"—for instance, prompting a model to "answer as a Japanese person." Benchmarks like WVS and GlobalOpinionQA also commonly evaluate at the country level.

Limitations of Prior Work: Empirical analysis using WVS Wave 7 revealed two findings: (1) Within the same country, nearly half of the value questions have Shannon entropy \(H > 1.0\), indicating significant intra-country heterogeneity; (2) Mean Decrease Impurity analysis via Random Forest shows that "Religion / Income / Occupation" generally contribute more than Country to value prediction. In other words, country labels are insufficient to characterize individual values and instead smooth over critical differences.

Key Challenge: Expressing "pluralistic values" requires fine granularity, but individual-level alignment lacks supervision signals; existing methods leave a vacuum between "macro-country" and "micro-individual."

Goal: To identify a learnable and generalizable intermediate granularity between the country and the individual—the "demographic archetype"—and solve three sub-problems: (1) How to extract a high-consensus subset from WVS; (2) How to enable explicit "demographic attributes → values" reasoning; (3) How to precisely anchor group distributions without damaging general capabilities.

Key Insight: It was observed that even in groups with identical demographic profiles (11-dimensional match), 9.2% of value responses still exhibit internal disagreement—this portion is essentially noise. The remaining samples with \(H=0\) constitute stable "archetype answers."

Core Idea: Establish a "high-consensus demographic-value corpus" using entropy threshold filtering, externalize implicit "attribute → value" mappings via Structured CoT, and anchor the distribution using GRPO with binary rewards to prioritize simplicity.

Method

Overall Architecture

The pipeline consists of three stages: (1) Data Construction—Starting from WVS Wave 7, archetypes are aggregated by 11D demographic attributes. For each profile-question pair, Shannon entropy is calculated, retaining only \(H=0\) samples. This is augmented by 10-country sampling from the Inglehart-Welzel cultural map and 16 value-question filters, resulting in 56,152 training samples; (2) Demographic Value Alignment—Given profile \(P\), question \(Q\), and Structured CoT instruction \(I_{cot}\), the policy \(\pi_\theta\) outputs \((T,\hat y)\sim\pi_\theta(\cdot|P,Q,I_{cot})\). GRPO with binary reward \(r=\mathbb{I}(\hat y=y_i)+\beta r_{format}\) anchors the output to WVS ground truth; (3) Triple Generalization Evaluation—An additional 21,553 samples are constructed to cover cross-demographic (6,240), cross-country (7,973, including 8 unseen countries), and cross-value (7,340, including 7 unseen value questions) scenarios.

Key Designs

1. Demographic Archetype Extraction (Strict \(H=0\) Filtering):

   • Function: Aggregates individual WVS responses into "demographic archetypes with consistent value preferences" to serve as the target alignment distribution.
   • Mechanism: Each respondent is encoded into a profile \(P\) using 11 attributes (Country, Gender, Age, Marital, Parenthood, Income, Occupation, Work Nature, Education, Religion, Language) based on Bourdieu's social stratification. Approximately 32.8% of profiles overlap across multiple people. Shannon entropy \(H\) is calculated for each \((P, Q)\) pair; only profile-value pairs with \(H=0\) (total consensus) are retained, while approximately 9.2% of "disputed profiles" are discarded.
   • Design Motivation: Previous multi-cultural fine-tuning used "all responses from one country" for training, effectively injecting internal heterogeneity noise into the supervision signal. Strict filtering with \(H=0\) isolates the "demographic archetype → stable value" subset for model fitting, eliminating label noise at the source.

2. Structured CoT Three-Step Reasoning Template:

   • Function: Externalizes the implicit sociological associations between "demographic attributes → value preferences" into a supervisable reasoning chain.
   • Mechanism: The instruction template \(I_{cot}\) forces the model to follow three steps: (i) Demographic-Value Correlation Analysis: Analyzing attribute-by-attribute whether the question touches on core interests or belief conflicts; (ii) Option Trade-off: Evaluating option compatibility with the demographic profile; (iii) Decision Output: Placing the final option within <answer></answer>. This chain is bound to GRPO training, providing implicit supervision for intermediate reasoning.
   • Design Motivation: Free-form RL reasoning can lead to "logical hallucinations" (Base+CoT decreased ACC by 0.8% in ablations). Explicit three-step reasoning hard-codes sociological logic—"persona roleplay + option weighing"—providing a safe cognitive scaffold.

3. GRPO + Minimalist Binary Reward ("Simplicity Wins"):

   • Function: Uses the simplest hit/miss signal to anchor the LLM's output distribution peak to the target archetype mode \(y_i\).
   • Mechanism: Reward \(r=\mathbb{I}(\hat y=y_i)+\beta r_{format}\), where relative advantage is calculated via GRPO within-group baselines. The authors assume LLMs possess a natural semantic topology (e.g., "Agree ↔ Strongly Agree" proximity) from pre-training, allowing smooth distribution learning without Likert-weighted continuous rewards. Ablations comparing Likert-adjusted soft rewards \(r=\alpha(1-|\hat y-y|/(L-1))+\beta r_{format}\) showed binary rewards achieved 1.6% higher ACC and 0.013 lower WD.
   • Design Motivation: While continuous rewards seem "more informative," they interfere with the LLM's existing semantic topology. Binary signals are cleaner, allowing the model to interpolate ordered distributions using distances in the token embedding space.

Loss & Training

GRPO learning rate \(5\times 10^{-6}\), temperature \(T=0.7\), 8 rollouts per sample, global batch size 64, trained for 1 epoch to prevent overfitting. Hardware: 8×A100 80GB; utilized VeRL + FSDP2 + Flash-Attention + bfloat16. Base models include Qwen3 (0.6B/1.7B/4B/8B) and Llama-3.2-3B-Instruct. Metrics: Accuracy (Acc), Likert Consistency \(\text{LC}=1-\frac{1}{N}\sum\frac{|\hat y-y|}{K-1}\), and Wasserstein Distance \(\text{WD}=\sum_k|\text{CDF}_{pred}(k)-\text{CDF}_{real}(k)|\).

Key Experimental Results

Main Results

On the cross-demographic test set (non-overlapping profiles), Qwen3-8B-DVMap outperforms GPT-4o with only 8B parameters:

Model	Parameters	Acc ↑	LC ↑	WD ↓
Qwen3-14B	14B	46.2	83.5	0.1460
Qwen3-next-80B-a3B	80B (3B act)	47.6	82.5	0.1449
Llama-3.3-70B-Instruct	70B	46.4	83.3	0.1504
DeepSeek-v3.2-exp	671B (MoE)	45.1	82.3	0.1342
Claude-3.7-sonnet	–	26.9	46.4	0.1503
GPT-4o-mini	–	46.3	82.4	0.1476
GPT-4o	–	48.5	83.8	0.1418
Qwen3-8B-DVMap	8B	48.6	83.9	0.1321

In cross-country tests, although trained on only 10 countries, Qwen3 (0.6B/1.7B/4B/8B) showed improvements of +16.2 / +10.7 / +2.8 / +5.3 % Acc on 8 unseen countries. Llama-3.2-3B cross-demographic Acc also rose from 36.2% to 49.0%, proving cross-architecture efficacy.

Ablation Study

Based on Qwen3-4B, three ablation groups validate the core design:

Dimension	Configuration	Acc %	LC %	WD
Data Filtering	Base	44.3	82.2	0.158
Data Filtering	Majority Voting (\(H\ge 0\))	46.5	83.1	0.149
Data Filtering	DVMap (\(H=0\) Filter)	47.9	83.7	0.142
Reasoning	Base + Inference CoT	43.5	82.1	0.166
Reasoning	Standard RL (Free reasoning)	46.2	83.2	0.151
Reasoning	DVMap (Structured CoT + RL)	47.9	83.7	0.142
Reward Function	Likert-adjusted Soft	46.3	83.4	0.155
Reward Function	DVMap (Binary Reward)	47.9	83.7	0.142

Key Findings

• Filtering is the strongest lever on the data side: Strict \(H=0\) filtering yields a 1.4% Acc gain over majority voting, indicating that "internally inconsistent samples" are significant label noise.
• Structured CoT requires training synergy: Adding CoT only during inference causes performance drops, suggesting that the "reasoning chain" is only stable when shaped by RL signals.
• Binary Reward > Likert Soft Reward: Contrary to the "finer reward is better" intuition, using GRPO's internal relative advantage combined with the pre-trained semantic topology's natural order works best.
• Learning causality over memorization: Robustness analysis via flipping Income levels (keeping 10 other attributes constant) showed that DVMap's value flip rate in non-financial domains is significantly lower than the base model, indicating multi-dimensional judgment rather than simple lookup.
• Zero alignment tax: Performance fluctuations on MMLU/ARC-E/GSM8K/HellaSwag were <0.1%, with IFEval even increasing by +0.48%.

Highlights & Insights

• Revisiting value alignment as "Manifold Mapping": The authors define the goal as learning the "demographics → values" manifold mapping. Cross-country/value generalization thus validates manifold continuity.
• High ROI filtering via \(H=0\): A simple yet highly effective strategy. Any future work on "group preference alignment" could adopt this "aggregate by high-dimensional attributes → filter by entropy threshold" pipeline.
• Minimalist Reward Engineering: By moving against the trend of complex preference rewards, it provides a counter-example showing that pre-trained semantic topologies serve as effective implicit reward priors.
• Robustness Case (Widowed Russian Female): DVMap weighs "high income" against "emotional impact of widowhood + Russian cultural humility" to output "Rather happy," whereas the base model is pulled directly by the income flip to "Very happy."

Limitations & Future Work

• WVS is a static snapshot and cannot reflect value evolution over time; it may become obsolete for rapidly changing social issues (e.g., AI ethics).
• The 11D profile is a statistical abstraction capturing "sociological roles" rather than "psychological individuals," which may distort niche groups.
• Evaluations are multiple-choice discriminative; they do not measure whether the model can use identity-specific tone/rhetoric in open-generation.
• Future work could combine this with personalized alignment (Guan et al. 2025), using archetypes as priors and individual fine-tuning as posteriors in a "Hierarchical Bayesian" manner.

Related Work & Insights

• vs CultureLLM / CulturePark (Li et al. 2024a/b): These rely on country-level labels. DVMap pushes granularity down using 11D attributes + entropy filtering, avoiding shallow "pretend to be Japanese" prompt injection.
• vs Modular Pluralism (Feng et al. 2024): They rely on multi-LLM collaboration for pluralistic output; DVMap achieves archetype generalization within a single model.
• vs RLHF (Bai et al. 2022) / DPO (Rafailov 2023): Traditional RLHF learns "universal preferences"; this work uses GRPO + binary rewards + group targets to redefine alignment as "distribution anchoring."

Rating

• Novelty: ⭐⭐⭐⭐ Pushing alignment granularity to demographic archetypes with a triple generalization benchmark is solid.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across generalization, architectures, ablations, robustness, and general utility.
• Writing Quality: ⭐⭐⭐⭐ Logical flow from empirical analysis to method and evaluation; sociological background is well-integrated.
• Value: ⭐⭐⭐⭐⭐ Addresses the industry pain point of "Western-centric bias" with a low-cost, replicable method.

Related Papers

• [ACL 2026] ToolPRM: Fine-Grained Inference Scaling of Structured Outputs for Function Calling
• [AAAI 2026] Improving Value-based Process Verifier via Low-Cost Variance Reduction
• [ICLR 2026] Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning
• [AAAI 2026] Jupiter: Enhancing LLM Data Analysis Capabilities via Notebook and Inference-Time Value-Guided Search
• [CVPR 2026] E-comIQ-ZH: A Human-Aligned Dataset and Benchmark for Fine-Grained Evaluation of E-commerce Posters with Chain-of-Thought