LDC: Learning to Generate Research Ideas with Dynamic Control¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/du-nlp-lab/LDC
Area: LLM (Other)
Keywords: research idea generation, multi-dimension reward modeling, controllable reinforcement learning, dynamic decoding, novelty-feasibility trade-off
TL;DR¶
To resolve the inherent trade-offs among novelty, feasibility, and effectiveness in automated research ideation, LDC introduces a two-stage training framework combining supervised fine-tuning with multi-dimensional controllable reinforcement learning, steered at inference time by a sentence-level RNN decoder that dynamically adapts dimension weights according to context.
Background & Motivation¶
Recent advancements in large language models have highlighted their immense potential in automating scientific research, expanding from literature search and trend analysis to high-level conceptualization, including hypothesis generation, methodology formulation, and experimental planning. However, existing efforts primarily rely on prompting pre-trained models or setting up multi-agent debate loops over literature graphs (e.g., ResearchAgent, SciMON). Because their underlying generators lack task-specific end-to-end alignment against scientific review standards, the generated ideas often fail to satisfy the rigorous criteria expected by expert peer reviewers.
Domain experts and community consensus widely recognize that high-quality scientific ideas are defined across three essential subdimensions: novelty (originality distinct from existing literature), feasibility (practicality under conventional computational and temporal constraints), and effectiveness (likelihood of achieving meaningful empirical improvements). Nevertheless, these dimensions present sharp inherent trade-offs: aggressively optimizing for novelty frequently yields unrealistic, ungrounded proposals that collapse in feasibility, whereas over-indexing on feasibility restricts the scope of discovery to trivial incremental variations. Furthermore, different sections of a research proposal exhibit heterogeneous priorities: the methodology section demands high novelty, whereas the experiment plan must prioritize grounded feasibility.
Standard RLHF using a single scalarized reward suffers from gradient conflicts among competing objectives, which trapped optimization within the convex hull of the non-convex Pareto front and prevents context-adaptive balance during generation. Core idea: formulate automated research ideation as a two-stage framework combining SFT structural pre-training with three independently trained dimensional controllers under RL, guided at inference time by a sentence-level RNN dynamic decoder to traverse the non-convex Pareto trade-offs across novelty, feasibility, and effectiveness.
Method¶
Overall Architecture¶
The LDC pipeline operates across two consecutive training stagesโSupervised Fine-Tuning (SFT) and Controllable Reinforcement Learning with Multi-dimension Rewardsโfollowed by dynamic decoding at inference time.
In the SFT stage, the model is exposed to pairs of academic papers and their corresponding follow-up ideas to master the structural formatting and foundational discourse patterns of research proposals. In the RL stage, three distinct discriminator reward modelsโtrained on real-world OpenReview peer feedbackโevaluate the generated hypotheses across novelty, feasibility, and effectiveness. Rather than merging these into a single scalar reward, LDC optimizes the underlying policy with PPO while training three decoupled layer-wise dimensional controllers. During inference, a lightweight recurrent neural network (RNN) inspects the preceding sentences to dynamically predict sentence-level steering weights, emphasizing novelty in methodological derivation and feasibility in experimental planning.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Target Paper + Retrieved Literature"] --> B["Two-Stage Training Framework<br/>SFT Structural Warmup + Multi-Reward PPO"]
B --> C["Three Decoupled Dimensional Controllers<br/>Layer-wise Multiplicative Perturbation"]
C --> D["Sentence-level RNN Dynamic Decoder<br/>Context-Aware Steering Weight Prediction"]
D --> E["Balanced High-Quality Research Idea<br/>Novel Method + Feasible Plan + High Impact"]
Key Designs¶
1. Two-Stage Training Framework: SFT Structural Grounding and Multi-Reward PPO
Directly applying reinforcement learning to pre-trained base models induces severe optimization instability and structural degeneration. To ground the model in standard academic conventions, LDC begins with a supervised fine-tuning stage using 1,000 paired instances mined from ICLR 2023โ2024 (a target paper alongside its most influential supporting paper retrieved via citation graph analysis and LLaMA-3 prompting): $\(\mathcal{L}_{\text{sup}} = \text{CE}(Y, \hat{Y})\)$ This ensures the model reliably adheres to a coherent proposal structure covering problem statements, methodology, and experiment plans.
In the reinforcement learning stage, the model interacts with three dedicated reward models (\(R_n, R_f, R_e\)) trained on OpenReview review comments and normalized scores. Crucially, each objective is optimized with PPO using generalized advantage estimation (GAE) per dimension rather than summing rewards into a single scalar value. This prevents conflicting gradients from canceling out meaningful updates and avoids collapsing the optimization into sub-optimal regions.
2. Three Decoupled Dimensional Controllers: Multiplicative Latent Subspace Steering
To enable fine-grained and tunable steering without re-training the base model for every desired trade-off, LDC incorporates three learnable parameter matrices \(W_n, W_f, W_e\) into the \(l\)-th layer of the LLM: $\(M_l^n = M_l + \epsilon_n W_n M_l, \quad M_l^f = M_l + \epsilon_f W_f M_l, \quad M_l^e = M_l + \epsilon_e W_e M_l\)$ During training, \(\epsilon_n, \epsilon_f, \epsilon_e\) are fixed to 1.0, and each controller is optimized independently against its corresponding reward function.
By training each controller separately, each parameter set freely explores the subspace specific to its target dimension. Setting \(\epsilon = 0\) reduces the network exactly to the underlying RL policy, while scaling \(\epsilon\) projects the hidden states into the learned dimension-specific subspace. Because the objectives are trained independently, they are not constrained to the convex hull of scalarized multi-objective optimization, permitting arbitrary compositional control at test time.
3. Sentence-level RNN Dynamic Decoder: Context-Aware Steering Across Sections
In naive static decoding, fixing all controller weights to a constant value (e.g., all set to 1.0) leads to severe contextual mismatches: demanding high novelty throughout the generation forces the model to invent bizarre, ungrounded benchmark setups in the experiment plan, whereas excessive feasibility in early stages stifles algorithmic creativity.
To resolve this, LDC deploys a recurrent neural network with a hidden size of 256 that operates at sentence granularity. Conditioned on all preceding sentences \(S_{<t}\), the RNN dynamically predicts the triple of control weights for the upcoming sentence \(\hat{\epsilon}_t = (\epsilon_t^n, \epsilon_t^f, \epsilon_t^e)\): $\(\mathcal{L}_{\text{rnn}} = \text{CE}(\text{RNN}(S_{<t}), \hat{\epsilon}_t)\)$ At each generation step, the composite hidden representation in the final layer is steered according to: $\(M_l^{\text{steer}} = M_l + \epsilon_t^n W_n M_l + \epsilon_t^f W_f M_l + \epsilon_t^e W_e M_l\)$ As verified in Figure 4 of the paper, this decoder naturally exhibits a sharp contextual transition: it allocates high novelty weights across sentences 1โ5 (the core methodology), and smoothly shifts toward high feasibility and effectiveness around sentence 6 (the onset of dataset selection, baselines, and evaluation protocols).
Loss & Training¶
- SFT Stage: Learning rate \(1\times 10^{-5}\), batch size 32, trained for 3 epochs.
- PPO Reinforcement Learning: Policy learning rate \(1\times 10^{-6}\), PPO clipping ratio \(\epsilon_{\text{clip}} = 0.2\), discount factor \(\gamma = 1.0\), GAE parameter \(\lambda = 0.95\), and 4 PPO epochs. Controller matrices \(W_n, W_f, W_e\) are initialized with Xavier uniform.
- RNN Controller Weight Generator: Trained on 1,000 high-quality generated proposals with normalized sentence-level reward targets scaled between \(0.0\) and \(5.0\) (\(\epsilon_{\max} = 5.0\)).
Key Experimental Results¶
Main Results¶
Evaluations were conducted on a held-out test split of 500 papers using a retrieval-augmented GPT-4 reviewing agent (grounded with the latest literature from Semantic Scholar), alongside a rigorous manual evaluation by 15 domain experts across 30 randomly selected papers (rated on a 1โ10 scale).
| Model | Novelty (N) | Feasibility (F) | Effectiveness (E) | Overall |
|---|---|---|---|---|
| ResearchAgent (Baek et al., 2025) | 5.2 | 6.0 | 5.3 | 5.3 |
| MHABTO (Su et al., 2025) | 5.4 | 5.9 | 5.3 | 5.4 |
| T5-SFT | 3.3 | 5.1 | 4.2 | 4.0 |
| LLaMA-2-7B-SFT | 4.5 | 5.6 | 5.2 | 5.1 |
| LLaMA-2-7B-RLHF (no controllers) | 5.3 | 6.0 | 5.5 | 5.4 |
| LLaMA-3-8B-RLHF (no controllers) | 5.4 | 5.7 | 5.6 | 5.7 |
| LLaMA-3 + Single Novelty Ctrl | 6.3* | 5.3 | 5.4 | 5.8 |
| LLaMA-3 + Single Feasibility Ctrl | 5.1 | 6.8* | 5.3 | 5.3 |
| LLaMA-3 + Single Effectiveness Ctrl | 5.4 | 5.8 | 6.3* | 5.7 |
| LLaMA-3-8B + All Ctrls (Static Decoding) | 5.9 | 6.3 | 6.0 | 6.1 |
| Qwen-2.5-3B + All Ctrls (Dynamic Decoding) | 5.6 | 5.9 | 5.8 | 5.9* |
| LLaMA-2-7B + All Ctrls (Dynamic Decoding) | 5.7 | 6.1 | 5.8 | 5.8* |
| LLaMA-3-8B + All Ctrls (Dynamic Decoding, Ours) | 6.2 | 6.4 | 6.1 | 6.3* |
Note: * indicates statistical significance against all competing baselines under paired t-tests (\(p < 0.05\)).
Ablation Study: Novelty vs. Feasibility Trade-Off¶
Under the single-controller setting, sweeping the novelty steering weight \(\epsilon_n\) demonstrates the direct empirical trade-off between conceptual novelty and execution feasibility:
| Novelty Steer Weight (\(\epsilon_n\)) | Novelty Score (N) | Feasibility Score (F) | Note |
|---|---|---|---|
| 1.0 | 6.4 | 6.1 | Balanced baseline setting |
| 2.0 | 6.7 | 5.8 | Moderate novelty gain with slight feasibility loss |
| 3.0 | 7.0 | 5.3 | Distinctly novel, but implementation complexity rises |
| 4.0 | 7.3 | 4.9 | Highly novel, but feasibility drops below viable threshold (< 5.0) |
In decoding strategy ablations: - Replacing Dynamic Decoding with Static Decoding (all weights fixed to 1.0) on LLaMA-3 drops Overall score from 6.3 to 6.1 and Novelty from 6.2 to 5.9, confirming the value of contextual adaptation. - Removing all dimensional controllers entirely (plain SFT+RLHF) reduces LLaMA-3 Overall performance to 5.7, proving that explicit latent space steering provides essential gain over monolithic RLHF.
Key Findings¶
- The innovation-feasibility trade-off is quantitative and monotonic: Scaling the novelty weight from 1.0 to 4.0 pushes Novelty from 6.4 to 7.3, but forces Feasibility down from 6.1 to 4.9. This verifies that unconstrained optimization for originality naturally damages real-world viability.
- Dynamic steering resolves section-level tension: The sentence-level RNN dynamically pivots control signals at sentence 6, shifting priorities from novelty during methodological reasoning to feasibility and effectiveness during experiment planning.
- High alignment between automatic evaluator and expert judges: Pearson correlation coefficients between the retrieval-augmented GPT-4 judge and human experts reach 0.982 (Novelty), 0.948 (Feasibility), 0.716 (Effectiveness), and 0.871 (Overall). Furthermore, reward model predictions achieve a Spearman rank correlation of 0.80 on feasibility with human consensus.
Highlights & Insights¶
- Decoupled multi-dimensional reward optimization: Instead of collapsing multiple alignment objectives into a scalar reward that causes conflicting gradient updates, training independent layer-wise controllers allows full exploration of non-convex Pareto fronts.
- Sentence-level dynamic steering: Context-aware routing with an RNN allows an LLM to alter its generative posture mid-generation, effectively acting like a scientist who thinks radically during brainstorming and conservatively during experimental verification.
- Broad transferability for multi-objective text generation: The paradigm of "independent subspace controllers + sequential dynamic routing" can be directly transferred to other trade-off-heavy generation tasks, such as creative writing (humor vs. narrative coherence) or customer dialogue (empathy vs. conciseness).
Limitations & Future Work¶
- Lack of long-term scientific impact modeling: Current evaluation rubrics rely on conventional peer review criteria; integrating forward citation graph forecasting could better identify high-impact, paradigm-shifting hypotheses.
- Interpretability of latent controller weights: While the multiplicative hidden state perturbations reliably guide generation, deeper mechanistic interpretability is needed to understand which specific attention heads or neuron representations are being altered.
- Potential bias inheritance from automated review labels: Because some training samples lack explicit human review scores for feasibility and effectiveness, relying on LLM-extracted labels may propagate inherent inductive biases from the extractor model.
Related Work & Insights¶
- vs ResearchAgent (Baek et al., 2025): ResearchAgent relies heavily on knowledge graph traversal over scientific literature and prompt-based multi-agent critique, but leaves the underlying base model parameters untouched. LDC optimizes intrinsic generative capabilities via SFT and controllable RL, achieving superior idea quality (Overall 6.3 vs 5.3) without needing computationally intensive graph retrieval during generation.
- vs The AI Scientist (Lu et al., 2024): The AI Scientist focuses on automated code execution, running experiments, and compiling LaTeX manuscripts within narrow, code-constrained problem templates. LDC addresses open-ended, literature-grounded scientific ideation, providing higher-level conceptual innovation that complements downstream execution engines.
Rating¶
- Novelty: โญโญโญโญโญ [Pioneering framework unifying controllable RL and dynamic sentence-level decoding for scientific hypothesis generation]
- Experimental Thoroughness: โญโญโญโญโญ [Extensive validation across 500 papers with retrieval-augmented LLM evaluation, 15-expert human study, and detailed trade-off analyses]
- Writing Quality: โญโญโญโญโญ [Clear motivation, mathematically rigorous formulation of multi-objective Pareto optimization, and intuitive empirical diagrams]
- Value: โญโญโญโญโญ [Provides an actionable and extensible technical blueprint for automated scientific discovery and controllable language modeling]