💬 ACL2025 Accepted Papers¶
1853 ACL2025 paper notes covering LLM (Other) (442), Multimodal VLM (111), LLM Evaluation (89), Multilingual & Translation (86), Information Retrieval & RAG (84), Alignment & RLHF (82), Model Compression (78), Safety (69) and other 44 areas. Each note has TL;DR, motivation, method, experiments, highlights, and limitations — 5-minute reads of core ideas.
💡 LLM Reasoning (54)¶
- Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory
-
Through systematic experiments across 6 LLMs \(\times\) 8 prompting strategies \(\times\) 6 benchmarks, this work discovers that as the number of majority voting samples increases, simple CoT consistently outperforms complex prompting strategies. This phenomenon is theoretically proven from a probability perspective, and an \(O(1)\) complexity scaling performance prediction method along with two improvement strategies are proposed.
- LogicPro: Improving Complex Logical Reasoning via Program-Guided Learning
-
This paper proposes LogicPro, a data synthesis method that leverages LeetCode algorithmic problems and Python code solutions as logic sources. Through a three-step pipeline ("problem generation \(\rightarrow\) code intermediate variable extraction \(\rightarrow\) program-guided reasoning generation"), it synthesizes 540K high-quality textual reasoning data from 2,360 algorithmic problems, significantly outperforming existing reasoning datasets on multiple OOD benchmarks such as BBH27, LogicBench, and DROP.
- Beyond the Answer: Advancing Multi-Hop QA with Fine-Grained Graph Reasoning and Evaluation
-
To address the issues of opaque reasoning processes and coarse evaluation granularity in multi-hop question answering (Multi-hop QA), this paper proposes a fine-grained graph reasoning framework. By constructing a reasoning graph to explicitly model evidence chains, and introducing fine-grained evaluation metrics, the framework measures the quality of the reasoning process rather than solely focusing on the correctness of the final answer.
- Enhancing Retrieval Systems with Inference-Time Logical Reasoning
-
Proposes an Inference-Time Logical Reasoning (ITLR) framework, which leverages LLMs to decompose natural language queries into logical expressions (AND/OR/NOT), and then composes the cosine similarity scores of individual sub-terms based on fuzzy logic. It consistently outperforms traditional dense retrieval and the BRIGHT baseline on synthetic data and three real-world datasets (NFCorpus/SciFact/ArguAna), showing significant improvements particularly on complex queries containing negations.
- Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks
-
Proposes Self-Correction Learning (SCL), which categorizes self-correction data generated by the VLM itself (both successful and failed correction samples) into preference/dispreference pairs to perform preference fine-tuning via DPO. This fundamentally enhances the model's ability to directly generate correct answers, rather than relying solely on iterative refinement during inference.
- Towards Better Chain-of-Thought: A Reflection on Effectiveness and Faithfulness
-
This paper systematically analyzes the representation patterns of CoT from the dual perspectives of effectiveness and faithfulness. It finds that problem difficulty, information gain, and the monotonicity of information flow govern the effectiveness of CoT. It also reveals the mechanism of unfaithful CoT: the model recalls correct information from the question that was omitted by the CoT when predicting answers. Based on these insights, the QUIRE algorithm is proposed, which simultaneously improves both the effectiveness (+2.4%) and faithfulness (+5.6%) of CoT.
- Critic-CoT: Boosting the Reasoning Abilities of Large Language Model via Chain-of-Thoughts Critic
-
This paper proposes the Critic-CoT framework, which transitions LLM self-criticisms from System-1 intuitive judgments to System-2 deliberate step-by-step analyses through a step-by-step Chain-of-Thought critic paradigm and automated weakly supervised data construction without human annotation. Two-stage training (GPT-4 distillation + self-criticism) improves Llama-3-70B-Instruct performance on GSM8K from 89.6% to 95.4% and on MATH500 from 50.4% to 68.4%. Additionally, it is discovered that criticism capabilities and task-solving capabilities can mutually reinforce each other.
- DRT: Deep Reasoning Translation via Long Chain-of-Thought
-
This work introduces long CoT reasoning into machine translation by establishing a multi-agent framework (Translator \(\to\) Advisor \(\to\) Evaluator) to iteratively refine literary translations containing metaphors and similes. It synthesizes a 22K long-thought translation training dataset, and the resulting DRT-14B model outperforms large models such as QwQ-32B and DeepSeek-R1-Distill-32B in literary translation.
- Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractions
-
This paper proposes QuaSAR (Quasi-Symbolic Abstract Reasoning), a Chain-of-Thought (CoT) variant that guides LLMs to first abstract the problem symbolically (extracting variables/predicates), reconstruct it using a semi-formal representation, and finally solve it based on a quasi-symbolic reasoning chain. QuaSAR achieves up to an 8% accuracy improvement over standard CoT on GPT-4o, while significantly enhancing robustness against adversarial variants (e.g., option shuffling, numerical substitution).
- Large Language and Reasoning Models are Shallow Disjunctive Reasoners
-
This paper evaluates the systematic generalization capabilities of LLMs and LRMs on disjunctive rule reasoning tasks that require composing multiple reasoning paths, using a synthetic spatial and temporal reasoning benchmark (STaR). It finds that even reasoning models like o3-mini can only handle single-path reasoning, with performance degrading drastically in multi-path disjunctive reasoning scenarios.
Browse all 54 LLM Reasoning papers →
🦾 LLM Agent (63)¶
- OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use
-
A comprehensive survey of Operating System Agents (OS Agents) based on Multimodal Large Language Models (MLLMs), systematically analyzing their fundamental concepts (environment/observation/action space), core capabilities (understanding/planning/grounding), construction methodology (foundation models + agent frameworks), and evaluation systems. It covers a categorical comparison of 30+ foundation models and 20+ Agent frameworks.
- MultiAgentBench: Evaluating the Collaboration and Competition of LLM Agents
-
This paper proposes the MultiAgentBench benchmark and the MARBLE framework to systematically evaluate the performance of LLM multi-agent systems in collaborative and competitive scenarios. Covering 6 interactive environments (Research, Minecraft, Database, Coding, Bargaining, and Werewolf), the study introduces milestone-based KPI metrics and coordination scores. The evaluation reveals that GPT-4o-mini achieves the highest overall task score, graph-structured coordination protocols perform best in research scenarios, and cognitive planning improves milestone completion rates by 3%.
- BookWorld: From Novels to Interactive Agent Societies for Story Creation
-
BookWorld is the first multi-agent social simulation system based on novels. It constructs interactive virtual worlds by extracting character data and worldview specifications from source books, allowing novel characters to act and interact autonomously to generate creative stories, outperforming previous story generation methods in 75.36% of pairwise comparisons.
- CortexDebate: Debating Sparsely and Equally for Multi-Agent Debate
-
This paper proposes CortexDebate, a multi-agent debate method inspired by the mechanism of the human cerebral cortex. By constructing a sparse dynamic debate graph and an evaluation module based on the McKinsey Trust Formula (MDM), it simultaneously addresses two core challenges of existing Multi-Agent Debate (MAD) methods: "excessively long input context" and "unequal debate caused by overconfidence."
- Play2Prompt: Zero-shot Tool Instruction Optimization for LLM Agents via Tool Play
-
Proposes Play2Prompt, which enables LLMs to autonomously "play" with tools (exploring input-output behaviors) to generate tool-use examples and optimize tool documentation in a zero-shot manner, significantly enhancing the tool-calling capabilities of LLM agents without requiring any annotated data.
- Voting or Consensus? Decision-Making in Multi-Agent Debate
-
This work systematically compares 7 decision protocols (voting vs. consensus) in multi-agent debate (MAD). It is found that consensus protocols improve performance by 2.8% on knowledge tasks, while voting protocols improve performance by 13.2% on reasoning tasks. Two new methods, AAD and CI, are proposed to enhance answer diversity, yielding performance gains of 3.3% and 7.4%, respectively.
- CoMet: Metaphor-Driven Covert Communication for Multi-Agent Language Games
-
This paper proposes the CoMet framework. By integrating a hypothesis-testing-based metaphor reasoner and a self-improving metaphor generator, CoMet enables LLM agents to utilize metaphors for covert communication and semantic evasion in multi-agent language games. It significantly enhances the strategic communication capabilities of agents in Undercover and Adversarial Taboo games (improving the win rate from 0.20 to 0.70).
- MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation
-
This paper proposes MetaSynth, a meta-prompting-driven multi-agent collaborative framework that generates highly diverse synthetic data. Using only 25M tokens of synthetic data (without mixing real data), it successfully adapts Mistral-7B to financial and biomedical domains, achieving performance gains of \(4.08\%\) and \(13.75\%\) respectively, without compromising general capabilities.
- Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Tools
-
Agentic Reasoning proposes a framework that integrates three agent tools—Web search, code execution, and knowledge-graph-based memory (Mind-Map)—into the LLM reasoning process. It improves the accuracy of DeepSeek-R1 on Humanity's Last Exam from 9.4% to 23.8% (+14.4%) and GPQA from 71.5% to 81.2%, approaching the performance level of OpenAI Deep Research.
- EMULATE: A Multi-Agent Framework for Determining the Veracity of Atomic Claims by Emulating Human Actions
-
The EMULATE multi-agent fact-checking framework is proposed, which simulates the complete human action chain for verifying claims (search → ranking → content evaluation → evidence sufficiency judgment → classification) through 7 specialized LLM agents, outperforming existing methods in both Macro-F1 and Weighted-F1 on three fact-checking benchmarks.
Browse all 63 LLM Agent papers →
⚖️ Alignment & RLHF (82)¶
- MTSA: Multi-Turn Safety Alignment for LLMs through Multi-Round Red-Teaming
-
The proposed MTSA framework simultaneously enhances the attack capabilities of the red-team model and the safety defense performance of the target model through thought-guided multi-round red-teaming and reinforcement learning optimization with future rewards within an adversarial iterative process. It achieves state-of-the-art (SOTA) performance across multiple safety benchmarks without degrading general model capabilities.
- Understanding Impact of Human Feedback via Influence Functions
-
Applying influence functions to feedback data auditing for RLHF reward models for the first time, combined with OPORP vector compression to achieve a 2.5x speedup. It outperforms GPT-4o in bias detection (AUC 0.8 vs. 0.747) and identifies 47% of mislabeled samples from the Anthropic-HH dataset.
- JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs
-
The first unified comprehensive assessment framework covering both automatic and non-automatic jailbreak attacks: compiling 17 representative jailbreak attacks, establishing a taxonomy of six attack categories, and performing large-scale systematic evaluation across 9 aligned LLMs and 8 defense strategies, revealing the key insight that heuristic-based attacks exhibit "high ASR but low utility."
- M2S: Multi-turn to Single-turn jailbreak in Red Teaming for LLMs
-
This paper proposes the M2S framework, which compresses multi-turn human jailbreak conversations into single-turn prompts using three simple format conversion methods (Hyphenize/Numberize/Pythonize). This approach not only maintains but even exceeds the original multi-turn attack effectiveness (achieving an ASR up to 95.9%, improving by up to 17.5% over multi-turn attacks) while reducing token usage by more than half.
- Towards Reward Fairness in RLHF: From a Resource Allocation Perspective
-
This paper unifies various reward biases in RLHF (such as length bias, category bias, and social bias) under the concept of "reward unfairness". Drawing on resource allocation theory, the authors propose two bias-agnostic methods, Fairness Regularization and Fairness Coefficient, applied to reward model training and policy model training respectively. These methods simultaneously mitigate multiple biases and improve alignment quality without being tailored to any specific bias.
- Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
-
It is discovered that the traditional DPO preference data construction strategy (max-min) suffers from performance degradation as the sample size increases. Through a systematic exploration based on reward distribution, it is found that the rejected response should be selected at \(\mu-2\sigma\) instead of the minimum. Based on this finding, a preference data construction method is proposed that consistently scales with the sample size.
- FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings
-
This paper proposes FocalPO, a DPO variant that introduces a focal loss-inspired modulation factor to down-weight incorrectly ranked pairs, prioritizing the reinforcement of the model's understanding of already correctly ranked preference pairs, outperforming DPO and its variants on benchmarks such as AlpacaEval 2.0.
- Constitutional Classifiers: Defending Against Universal Jailbreaks Across Thousands of Hours of Red Teaming
-
Anthropic proposes "Constitutional Classifiers", which train input/output safety classifiers by generating synthetic training data from natural language safety principles (constitutions). In over 3000 hours of professional red teaming, no universal jailbreak attacks were discovered, while only incurring a 0.38% increase in over-refusal rate and a 23.7% inference overhead.
- Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization
-
This paper proposes Segment Supervised Preference Optimization (SSPO), which models the duration alignment problem between translated text and source speech in video dubbing as segment-level preference optimization. By using sentence-by-sentence sampling and fine-grained DPO loss, it achieves duration consistency for each dialogue line while maintaining translation quality and output format.
- DiffPO: Diffusion Alignment with Direct Preference Optimization
-
DiffPO is proposed to reformulate LLM alignment as a sentence-level diffusion denoising process. Through parallel decoding, it achieves efficient inference-time alignment and serves as a plug-and-play module that enhances the alignment quality of any base model.
Browse all 82 Alignment & RLHF papers →
🔒 Safety (69)¶
- PrivaCI-Bench: Evaluating Privacy with Contextual Integrity and Legal Compliance
-
Proposes PrivaCI-Bench, the largest contextual privacy evaluation benchmark to date (154K instances) built upon Contextual Integrity theory. It covers real court cases, privacy policies, and synthetic data from EU AI Act compliance checkers to evaluate the legal compliance capabilities of LLMs under HIPAA, GDPR, and the AI Act.
- SpeechFake: A Large-Scale Multilingual Speech Deepfake Dataset Incorporating Cutting-Edge Generation Methods
-
The SpeechFake dataset, a large-scale speech deepfake dataset, is constructed. It contains over 3 million deepfake samples, encompasses more than 3,000 hours of audio, covers 40 generation tools, and spans 46 languages. Through baseline experiments, the systematic impacts of generation methods, linguistic diversity, and speaker variations on detection performance are analyzed.
- Language Models Can Subtly Deceive Without Lying: A Case Study on Strategic Phrasing
-
This work constructs a legislative environment testing platform (LobbyLens) to study whether LLMs can employ strategic phrasing—specifically, coloring expressions without outright lying—to obscure the corporate benefits embedded within bill amendments. The authors find that LLMs optimized via iterative re-planning can boost their deception success rate by up to 40 percentage points.
- CENTAUR: Bridging the Impossible Trinity of Privacy, Efficiency, and Performance in Privacy-Preserving Transformer Inference
-
This paper proposes the Centaur framework, which integrates random permutation matrices and Secure Multi-Party Computation (SMPC) to break the "impossible trinity" in Privacy-Preserving Transformer Inference (PPTI)—simultaneously achieving strong privacy protection, 5-30x speedup, and plaintext-level inference accuracy.
- FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypes
-
This paper propose Indic-Bias, the first large-scale LLM fairness benchmark tailored to the diverse Indian society. Testing 14 LLMs across three evaluation tasks using 20,000 human-verified scenario templates, it reveals that models possess severe negative biases against marginalized groups such as Dalits and reinforce stereotypes in over 70% of the cases.
- Private Memorization Editing: Turning Memorization into a Defense to Strengthen Data Privacy in Large Language Models
-
This paper proposes PME (Private Memorization Editing), which transforms the memorization tendency of LLMs from a security vulnerability into a defense mechanism. By editing the parameters of Feed Forward layers, it removes memorized personally identifiable information (PII), achieving privacy protection without retraining.
- WET: Overcoming Paraphrasing Vulnerabilities in Embeddings-as-a-Service with Linear Transformation Watermark
-
This work reveals that existing EaaS embedding watermarking methods (EmbMarker/WARDEN) can be bypassed by paraphrasing attacks. It proposes WET (Watermark via Linear Transformation), which injects watermarks by applying linear transformations to embeddings using a secret circulant matrix. Theoretical analysis and empirical results demonstrate its robustness against paraphrasing attacks, achieving a verification AUC near 100%.
- Efficiently Identifying Watermarked Segments in Mixed-Source Texts
-
Proposes two efficient methods (Geometric Cover Detector and Adaptive Online Locator) to detect and precisely locate watermarked segments in long mixed-source texts, reducing the time complexity from \(O(n^2)\) to \(O(n \log n)\), significantly outperforming baselines across three mainstream watermarking techniques.
- Robust and Minimally Invasive Watermarking for EaaS
-
Proposed ESpeW (Embedding-Specific Watermark), an embedding-specific watermarking method that injects unique watermarks at different positions of each embedding vector, achieving robust copyright protection for Embeddings as a Service (EaaS). It resists various watermark removal attacks while affecting the embedding quality by less than 1%.
- Gender Inclusivity Fairness Index (GIFI): A Multilevel Framework
-
This work proposes GIFI (Gender Inclusivity Fairness Index), a multi-level evaluation framework covering seven dimensions: pronoun recognition, sentiment neutrality, toxicity, counterfactual fairness, stereotype association, occupational fairness, and mathematical reasoning consistency. It systematically quantifies binary and non-binary gender fairness across 22 mainstream LLMs, revealing deep bias patterns such as the complete absence of neopronouns without prompting and the over-correction of "she".
👻 Hallucination Detection (27)¶
- Removal of Hallucination on Hallucination: Debate-Augmented RAG
-
DRAG (Debate-Augmented RAG) proposes introducing a Multi-Agent Debate (MAD) mechanism in both the retrieval and generation stages of RAG systems. Through a structured process of proponent-opponent debate and judge arbitration, it eliminates the "hallucination on hallucination" problem caused by erroneous retrieval, significantly improving factual accuracy across six QA benchmarks.
- Monitoring Decoding: Mitigating Hallucination via Evaluating the Factuality of Partial Response during Generation
-
A Monitoring Decoding (MD) framework is proposed to dynamically monitor the factuality of partial responses during generation. It identifies hallucination-prone tokens via a monitor function and selectively revises these key tokens using a tree-search strategy, significantly improving factual accuracy while maintaining efficiency.
- Aligning Large Language Models to Follow Instructions and Hallucinate Less via Effective Data Filtering
-
This paper proposes the NOVA framework, which filters knowledge-aligned, high-quality instruction data by measuring the LLM's familiarity with instructions via Internal Consistency Probing (ICP) and familiarity with target responses via Semantic Equivalence Identification (SEI). Fine-tuning LLaMA-3-8B with only 5% of selected data achieves an 8.6-point improvement on BioGEN and a 7.2-point improvement on FollowRAG, while preserving instruction-following capability.
- Alleviating Hallucinations from Knowledge Misalignment in Large Language Models via Selective Abstention Learning
-
To address the hallucination issue in LLMs caused by knowledge misalignment (inconsistency between model parametric knowledge and reality), this paper proposes a Selective Abstention Learning method. This approach enables the model to actively refuse to answer when encountering questions outside its knowledge boundary instead of fabricating content, thereby reducing hallucinations.
- HalluLens: LLM Hallucination Benchmark
-
Proposes HalluLens, a hallucination benchmark that clearly distinguishes hallucination from factuality, establishes a clear taxonomy of extrinsic hallucination (inconsistency with training data) and intrinsic hallucination (inconsistency with input context), introduces three dynamically regenerable extrinsic hallucination evaluation tasks, and comprehensively analyzes the limitations of existing benchmarks.
- Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention
-
This paper proposes ASD (Activation Steering Decoding), a training-free, inference-time hallucination mitigation method. By identifying hallucination direction patterns within the intermediate hidden states of LVLMs, it leverages bidirectional steering and contrastive decoding to suppress hallucinated outputs while preserving the model's performance on general visual understanding tasks.
- Visual Evidence Prompting Mitigates Hallucinations in Large Vision-Language Models
-
This paper proposes Visual Evidence Prompting (VEP), which uses the outputs of small vision expert models (such as object detectors and scene graph generators) as textualized "visual evidence" input for LVLMs. This training-free approach significantly reduces hallucinations across 11 LVLMs—improving LLaVA-1.5 by 7.2% and Claude 3 by 12.1% on the POPE benchmark.
- Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
-
Proposes the VHD metric to quantify how sensitive the output of each attention head is to visual input. It finds that only a few attention heads are highly sensitive to visual information, and the model's over-reliance on language priors is a key factor causing hallucinations. Based on this, a training-free method, VHR, is designed to adaptively reinforce the contribution of vision-sensitive heads layer-by-layer (\(\alpha=2\)), reducing the CHAIR\(_S\) of LLaVA-1.5 on CHAIR from 49.68 to 33.32, with almost no additional inference overhead.
- FIHA: Autonomous Fine-grained Hallucination Evaluation in Vision-Language Models with Davidson Scene Graphs
-
This paper proposes FIHA, an automated, fine-grained hallucination evaluation framework that requires neither LLMs nor human annotations. By extracting entities, attributes, and relations from images and descriptions to generate Q&A pairs, and introducing Davidson Scene Graphs (DSG) to model inter-question dependencies, the authors construct the FIHA-v1 benchmark to comprehensively evaluate the hallucination levels of mainstream Large Vision-Language Models.
- On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation
-
Proposes RLFH (Reinforcement Learning for Hallucination), an on-policy self-alignment method where the LLM itself acts as the judge to decompose responses into atomic facts, evaluate their truthfulness and informativeness, generate token-level dense reward signals, and optimize via online PPO to effectively mitigate hallucination.
Browse all 27 Hallucination Detection papers →
📊 LLM Evaluation (89)¶
- Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
-
This paper constructs the LM-Polygraph uncertainty quantification (UQ) benchmark, implementing 30+ SOTA methods, and systematically evaluates the performance of UQ and confidence normalization techniques across 11 text generation tasks, providing a unified evaluation framework for LLM hallucination detection.
- Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories
-
Proposes the JitVul benchmark, which pairs each function with its vulnerability-introducing and vulnerability-fixing commits. Based on 879 CVEs covering 91 vulnerability types, it systematically evaluates the capabilities of LLMs and ReAct Agents in repository-level vulnerability detection, finding that ReAct Agents outperform pure LLMs, though both still have significant room for improvement.
- Navigating Rifts in Human-LLM Grounding: Study and Benchmark
-
This paper systematically studies the grounding (establishing mutual understanding) failure problem in human-LLM dialogues. It reveals that the frequency of proactive clarification by LLMs is only 1/3 of that of humans, and the frequency of proactive follow-up questions is only 1/16. To address this, the authors propose the Rifts benchmark (~1.8K tasks) to evaluate the grounding capabilities of LLMs, and implement preliminary interventions using a grounding forecaster.
- EducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Framework
-
This work proposes EducationQ, a multi-agent dialogue framework designed to evaluate the teaching capabilities of LLMs by simulating teacher-student informal formative assessment interactions in real classrooms. The study reveals that teaching effectiveness is not linearly correlated with model size or general reasoning ability, with Llama 3.1 70B demonstrating the best teaching performance.
- FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging
-
The FinanceReasoning benchmark is proposed to improve financial numerical reasoning evaluation across three dimensions: credibility, comprehensiveness, and level of challenge. This is achieved by re-annotating public datasets, constructing a library of 3,133 Python financial functions, and introducing 908 expert-annotated hard questions.
- ChatBench: From Static Benchmarks to Human-AI Evaluation
-
Through user studies, this work converts the static MMLU benchmark into human-AI dialogues, constructing the ChatBench dataset (396 questions, 7,336 dialogues). It reveals that AI-alone accuracy cannot predict user-AI accuracy, and trains a user simulator that improves correlation by 22–26 percentage points, laying the foundation for scalable interactive evaluation.
- TripCraft: A Benchmark for Spatio-Temporally Fine Grained Travel Planning
-
Proposes TripCraft, a travel planning benchmark dataset integrating real-world spatiotemporal constraints (public transit, activity availability, user personas, etc.), accompanied by five continuous evaluation metrics to systematically assess the quality of LLM-generated itineraries. It improves the temporal meal score from 61% to 80% under the parameter-informed setting.
- HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices
-
This paper introduces HomeBench, the first evaluation benchmark for smart-home LLMs that incorporates both valid and invalid instructions across single- and multi-device scenarios. The study reveals that even GPT-4o achieves a success rate of only 0.0% in multi-device scenarios involving invalid instructions.
- KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding
-
KITAB-Bench is a comprehensive Arabic OCR benchmark covering 8,809 samples across 9 major domains and 36 sub-domains. Evaluation results indicate that modern vision-language models (such as GPT-4o and Gemini) outperform traditional OCR methods by an average of 60% in terms of character error rate (CER), yet the best model only achieves a 65% accuracy in PDF-to-Markdown conversion, highlighting the massive challenges of Arabic document understanding.
- Towards Dynamic Theory of Mind: Evaluating LLM Adaptation to Temporal Evolution of Human States
-
This paper proposes the DynToM benchmark, evaluating LLM capabilities in tracking the temporal evolution of human mental states across 5,500 temporally linked scenes within 1,100 social contexts and 78,100 questions, revealing that models lag behind humans by an average of 44.7%.
Browse all 89 LLM Evaluation papers →
⚡ LLM Efficiency (42)¶
- Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
-
ModernBERT is proposed, systematically introducing modern LLM architectural optimizations (RoPE, GeGLU, alternating local/global attention, and unpadding) into encoder-only models. Trained on 2T tokens and natively supporting a context length of 8192, it outperforms BERT/RoBERTa/DeBERTaV3 across classification and retrieval tasks while achieving significantly faster inference speeds and superior memory efficiency.
- Tetris: Optimal Draft Token Selection for Batch Speculative Decoding
-
Tetris proposes a method to dynamically select the optimal draft tokens across requests in batch speculative decoding scenarios, maximizing inference throughput under limited computational resources by greedily choosing tokens with the highest cumulative acceptance probability.
- Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attention
-
Proposes Dynamic Block-Sparse Attention (DBSA), a training-free inference framework that achieves near-fine-tuning inference latency in many-shot in-context learning through structured block-sparse attention encoding and dynamic Retrieval-based KV cache while maintaining >95% of the accuracy of the best-performing methods.
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
-
DeepSeek proposes NSA—a natively trainable hierarchical sparse attention mechanism that achieves efficient long-context modeling through three parallel attention paths: token compression, token selection, and sliding window. After pre-training on a 27B parameter model, its performance matched or even surpassed Full Attention across all metrics, while delivering significant acceleration on 64k sequences.
- Scaling Context, Not Parameters: Training a Compact 7B Language Model for Efficient Long-Context Processing
-
Proposes MegaBeam-Mistral-7B, a 7B language model supporting a 512K token context length. Through engineering practices such as four-stage progressive training, RoPE theta tuning, bfloat16 precision correction, and XLA compiler memory optimization, this compact model achieves and even surpasses the performance of larger parameter models (such as Llama-3.1-70B, GPT-4) on long-context tasks.
- A Drop-In Solution for On-the-Fly Adaptation of Speculative Decoding in Large Language Models
-
This paper proposes a drop-in adaptive solution for speculative decoding that dynamically adjusts the speculative window size \(\gamma\) (and potentially the choice of draft models) during inference, thereby maximizing the end-to-end speedup of speculative decoding under diverse input distributions.
- LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
-
LongBench v2 is a challenging long-context evaluation benchmark consisting of 503 hard multiple-choice questions, with context lengths ranging from 8k to 2M tokens, covering six major task types. Under a 15-minute time limit, human experts only achieve an accuracy of 53.7%, while the strongest direct-generation model (GPT-4o 2024-08) achieves only 50.1%, and the reasoning model o1-preview reaches 57.7%, highlighting the critical importance of test-time compute scaling for deep long-context understanding.
- FocusLLM: Precise Understanding of Long Context by Dynamic Condensing
-
This paper proposes FocusLLM, a framework that dynamically condenses key information from long text by partitioning it into chunks and injecting dynamic prompts into each chunk. These chunks are condensed into candidate tokens using a trainable mechanism, which are then aggregated into the local context via a parallel decoding mechanism to generate the next token. Only requiring an 8K training length and a 0.5B token training budget, it successfully extends LLaMA-2 to a 400K context, outperforming all baselines on LongBench and ∞-Bench.
- How to Train Long-Context Language Models (Effectively)
-
This paper systematically studies how to effectively train long-context language models through continual pre-training and SFT. It proposes a series of key designs, including data mixing ratios, training length scaling, and evaluation protocols. The resulting ProLong-8B achieves the same-scale SOTA on 128K length using only 5% of Llama-3.1's long-context training data.
- LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training
-
Ours proposes LaMPE (Length-aware Multi-grained Positional Encoding), which adaptively determines the optimal mapping length using a parameterized scaled sigmoid function, and designs a three-region multi-grained attention mechanism (fine-grained local head + linearly normalized and compressed middle + tail that restores long-range dependencies) to achieve training-free plug-and-play context window extrapolation for LLMs, comprehensively outperforming existing methods on five major long-context benchmarks.
Browse all 42 LLM Efficiency papers →
📚 Pretraining (40)¶
- Byte Latent Transformer: Patches Scale Better Than Tokens
-
Proposes Byte Latent Transformer (BLT), a tokenizer-free byte-level LLM architecture that aggregates bytes into variable-length patches via entropy-based dynamic grouping. It matches the performance of token-based models at the 8B scale for the first time, while unlocking a new scaling dimension of improving inference efficiency by simultaneously scaling both patch and model sizes.
- Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models
-
Proposes the Meta-rater multi-dimensional data selection framework, defining four quality dimensions under PRRC (Professionalism, Readability, Reasoning, and Cleanliness). By using a proxy model regression to learn the optimal weighted combination of multiple quality scores, it doubles the training convergence speed of a 1.3B model and improves downstream task performance by 3.23%.
- Tokenization is Sensitive to Language Variation
-
This paper systematically investigates the contrasting impacts of three key design choices of BPE tokenizers (fitting corpus, pre-tokenizer, and vocabulary size) on downstream performance across tasks requiring language variation robustness versus sensitivity. It proposes a task-aware tokenizer evaluation metric based on logistic regression, which significantly outperforms task-agnostic metrics such as Rényi efficiency.
- Splintering Nonconcatenative Languages for Better Tokenization
-
This paper proposes Splinter—a "pre-tokenization" step added prior to BPE/UnigramLM. By iteratively deleting characters, it rearranges words from languages with "root-in-template" structures like Hebrew, Arabic, and Malay into linearly splittable sequences. This enables standard tokenizers to segment roots into contiguous tokens, leading to improvements in both intrinsic metrics and downstream Hebrew tasks.
- CritiQ: Mining Data Quality Criteria from Human Preferences
-
CritiQ proposes an automatic data quality criteria mining method based on agent collaboration. With only about 30 human preference annotation pairs, it can automatically discover interpretable data quality criteria and train a scorer for efficient data selection, significantly improving the downstream performance of Llama 3.1 in code, math, and logic domains.
- Towards Effective and Efficient Continual Pre-training of Large Language Models
-
This work systematically investigates data strategies for the continual pre-training of Llama-3 (8B). By employing three primary strategies—topic-based data mixture, perplexity-based curriculum learning, and high-quality synthetic scientific QA data—the proposed approach significantly enhances Chinese capabilities (C-Eval +8.81) and scientific reasoning (MATH +12.00) using only 100B tokens, while effectively maintaining the original English capabilities.
- Diversity Explains Inference Scaling Laws: Through a Case Study of Minimum Bayes Risk Decoding
-
Reinterprets MBR decoding from the theoretical perspective of bias-diversity decomposition: estimation error \(MSE = Bias - Diversity\), indicating that increasing diversity (the diversity of pseudo-references) is the key to improving MBR performance. It further extends this to general inference methods through information theory, revealing that diversity is the theoretical root of the inference scaling law (improving performance by increasing samples yields diminishing returns), and empirically validates this on machine translation, summarization, and image captioning tasks.
- Large Vocabulary Size Improves Large Language Models
-
Experiments demonstrate that larger subword vocabulary sizes consistently improve LLM performance on downstream tasks. This work also proposes a simple vocabulary replacement method (Swap & Insert) for switching to a more appropriate vocabulary in continual training scenarios.
- LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models
-
This paper proposes LeanCode, a context-aware attention-score-based code simplification method. By leveraging CLS attention (for classification tasks) and encoder-decoder attention (for generation tasks) to measure token importance, LeanCode outperforms SOTA methods DietCode/SlimCode by up to 60% and 29% in code search and code summarization tasks respectively while reducing inference time by up to 40.9%.
- Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion
-
Inspired by human second language acquisition, this paper proposes Progressive Vocabulary Expansion (PVE), a method that incrementally, exponentially introduces Arabic subwords into the LLaMA2 vocabulary across stages. This approach achieves efficient Arabic language adaptation while preserving the original English knowledge of the model, culminating in the AraLLaMA 7B/13B models.
Browse all 40 Pretraining papers →
✏️ Knowledge Editing (19)¶
- ChainEdit: Propagating Ripple Effects in LLM Knowledge Editing through Logical Rule-Guided Chains
-
The ChainEdit framework is proposed, which aligns logical rules mined from knowledge graphs with the intrinsic logical reasoning capabilities of LLMs to achieve chain-based updates during knowledge editing, improving logical generalization accuracy from ~20% to 58-65%.
- BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning
-
Proposes BMIKE-53, a cross-lingual benchmark covering 53 languages and integrating three knowledge editing datasets (zsRE, CounterFact, and WikiFactDiff). It systematically evaluates in-context knowledge editing methods from zero-shot to 8-shot settings, revealing that writing systems (Latin vs. non-Latin) are more decisive than language families for cross-lingual editing performance, and that metric-specific exemplar strategies significantly outperform hybrid configurations.
- The Mirage of Model Editing: Revisiting Evaluation in the Wild
-
This paper reveals systematic flaws in the evaluation practices of the model editing field—the near-perfect success rates (~96.8%) reported by prior methods plunge to 38.5% in real-world application scenarios. The root cause is the leakage of ground-truth information through teacher forcing during testing. The authors propose the QAEdit benchmark and the WILD evaluation framework to foster more reliable evaluations.
- A General Knowledge Injection Framework for ICD Coding
-
This paper proposes GKI-ICD, a general knowledge injection framework. By employing guideline synthesis and multi-task learning mechanisms, it simultaneously integrates three types of ICD knowledge—Description, Synonym, and Hierarchy—without requiring extra network modules, achieving SOTA performance on the MIMIC-III benchmark.
- CKnowEdit: A New Chinese Knowledge Editing Dataset for Linguistics, Facts, and Logic Error Correction in LLMs
-
Constructs CKnowEdit, the first knowledge editing dataset oriented towards Chinese linguistic characteristics. It covers three major categories (linguistics (pinyin/ancient poetry/classical Chinese/idioms/proverbs), facts (history and geography), and logical traps (homophones/reasoning/wordplay)) with a total of 1,854 samples. It systematically evaluates the performance of five mainstream knowledge editing methods on four Chinese LLMs, revealing unique editing challenges in Chinese.
- CompKe: Complex Question Answering under Knowledge Editing
-
Proposes the CompKe benchmark—containing 11,924 complex questions—to evaluate the performance of knowledge editing methods in complex reasoning scenarios involving one-to-many relations, logical operations (intersection/union), and condition confirmation, revealing the significant deficiencies of existing methods in complex question answering.
- Context-Robust Knowledge Editing for Language Models
-
This work identifies that existing knowledge editing methods significantly fail when prefix contexts are present (with editing success rates dropping from 90.9% to 69.1%). It introduces the CHED benchmark to evaluate context robustness and designs CoRE, a method that enhances the context robustness of editing through diversified prefix contexts and cross-prefix hidden state variance regularization, significantly narrowing the performance gap between settings with and without context while maintaining general model capabilities.
- DocMEdit: Towards Document-Level Model Editing
-
This paper proposes the document-level model editing task for the first time and constructs the DocMEdit benchmark containing 37,990 data items and 105,652 editing facts, revealing the severe shortcomings of existing editing methods in long-context, multi-fact parallel editing scenarios.
- Efficient Knowledge Editing via Minimal Precomputation
-
Demonstrates that the precomputation step (caching 44 million hidden vectors) for knowledge editing methods like MEMIT/ROME/EMMET can be reduced to 2-10 times the theoretical minimum (less than 0.3% of the original size), reducing precomputation time from dozens of hours to minutes with virtually no loss in editing performance.
- Memorizing is Not Enough: Deep Knowledge Injection Through Reasoning
-
Proposes a four-level knowledge injection framework (Memorization → Retrieval → Reasoning → Association) and builds the DeepKnowledge synthetic evaluation platform. It systematically reveals the key factors for each level of knowledge injection: repetitive learning for memorization, diverse expressions for retrieval, and explicit reasoning patterns for deep reasoning and association, providing a complete method-level mapping for LLM knowledge updates.
Browse all 19 Knowledge Editing papers →
💬 LLM (Other) (442)¶
- LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
-
Establishes Judge-Bench with 20 datasets (70k+ instances) to systematically evaluate 11 LLM judges against human annotations. Findings reveal colossal performance variance across tasks, attributes, and user expertise, indicating that task-specific human validation remains critical before deploying LLM judges.
- DeAL: Decoding-time Alignment for Large Language Models
-
DeAL reformulates the LLM alignment problem as a heuristic search problem during decoding. It utilizes customizable reward functions (including programmatic constraints and parameterized reward models) to guide token selection during the inference phase, achieving flexible multi-objective alignment that can complement and stack with RLHF.
- Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions
-
This paper systematically investigates whether LLMs can predict individual annotators' subjective text perceptions using sociodemographic attributes (age, gender, education, race/ethnicity). The authors find that performance improvements after fine-tuning primarily result from learning individual annotator behaviors rather than sociodemographic patterns, raising doubts about the feasibility of using LLMs to simulate sociodemographic variations.
- Understanding the Repeat Curse in Large Language Models from a Feature Perspective
-
Investigates the repeat curse in LLMs from the perspective of mechanistic interpretability. Specifically, Sparse Autoencoders (SAEs) are used to extract monosemantic features to locate "repeat features" in the middle and final layers. Activating these features induces repetition, whereas turning them off mitigates repetition without compromising model performance.
- Theory of Mind in Large Language Models: Assessment and Enhancement
-
This paper presents a systematic survey of evaluation benchmarks (10+ story-based benchmarks) and enhancement strategies (prompt-only and fine-tuning methods) for the Theory of Mind (ToM) capabilities of LLMs, highlighting that current LLMs still fall significantly short in ToM reasoning and outlining future directions.
- Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference
-
This paper proposes the FormalBench benchmark to systematically evaluate the program semantics reasoning capabilities of LLMs through formal specification inference tasks. It finds that while LLMs perform well on simple control flows, they struggle with complex structures like loops. Additionally, self-repair prompts are designed to improve the success rate by 25%.
- Representation Bending for Large Language Model Safety
-
Proposes RepBend, which integrates the core concept of activation steering (the vector difference between safe and unsafe representations) into the loss function design of LoRA fine-tuning. By "bending" the representation space of the model, it separates safe and unsafe states in the latent space, achieving up to a 95% reduction in Attack Success Rate (ASR) across various jailbreak benchmarks while maintaining minimal impact on general capabilities.
- Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers
-
Proposes the LimitGen benchmark to systematically evaluate the capability of LLMs in identifying limitations of scientific research papers. It includes a synthetic dataset (created via controlled perturbations) and a human-annotated dataset (from ICLR 2025 reviews), and enhances LLMs' ability to generate more specific and constructive feedback through RAG-augmented literature retrieval.
- SConU: Selective Conformal Uncertainty in Large Language Models
-
SConU introduces significance testing into the conformal uncertainty framework of LLMs for the first time. By constructing two types of conformal p-values, it identifies and filters out uncertainty data outliers that violate the exchangeability assumption, thereby achieving strict control over the miscoverage rate in both single-domain and cross-domain QA scenarios.
- The AI Gap: How Socioeconomic Status Affects Language Technology Interactions
-
Through a large-scale survey of 1,000 users of different socioeconomic status (SES) and an analysis of 6,482 real LLM prompts, this study reveals significant, systematic differences between high- and low-SES groups in terms of language technology usage frequency, interaction styles, and topic choices, calling for the development of more inclusive NLP technologies to narrow the AI gap.
Browse all 442 LLM (Other) papers →
📖 NLP Understanding (30)¶
- iQUEST: An Iterative Question-Guided Framework for Knowledge Base Question Answering
-
iQUEST proposes an iterative sub-question guided framework that dynamically generates answerable sub-questions at each reasoning step to maintain reasoning direction. Combined with GNNs to aggregate semantic information from two-hop neighbors for "look-ahead" entity exploration, it achieves SOTA or near-SOTA performance on four benchmarks (CWQ, WebQSP, WebQuestions, and GrailQA) without the need to fine-tune the LLM.
- Can LLMs Reliably Simulate Real Students' Abilities in Mathematics and Reading Comprehension?
-
Using Item Response Theory (IRT) to evaluate 11 LLMs on the same capability scale as real students, this work finds that strong models without styling far outperform average students. While persona prompting to "act as a student of a certain grade" can alter performance, no single model-prompt combination can reliably simulate an average student across all subjects and grades.
- A Comprehensive Graph Framework for Question Answering with Mode-Seeking Preference Alignment
-
This paper proposes the GraphMPA framework, which achieves global document understanding by constructing a hierarchical document graph based on general similarity metrics, and introduces mode-seeking preference optimization to replace traditional DPO for more precise human preference alignment, comprehensively outperforming existing RAG methods across six QA datasets.
- ReSCORE: Label-free Iterative Retriever Training for Multi-hop Question Answering with Relevance-Consistency Supervision
-
This paper proposes ReSCORE, which leverages the joint probability of document-query relevance and document-answer consistency generated by an LLM as pseudo-labels to train a dense retriever in an unsupervised manner within an iterative RAG framework, achieving SOTA performance on three multi-hop QA datasets.
- Dynamic Order Template Prediction for Generative Aspect-Based Sentiment Analysis
-
This paper proposes the Dynamic Order Template (DOT) method, which decomposes ABSA sentiment quadruple generation into two stages: first predicting the template size (number of quadruples) and generating the initial template, and then generating specific sentiment quadruples based on the dynamic templates. This approach achieves SOTA performance across 9 ABSA datasets while reducing inference time by 7 times compared to MvP.
- Sentiment Reasoning for Healthcare
-
This work introduces a new task termed "Sentiment Reasoning," which requires models to generate explanatory rationales while predicting sentiment labels for healthcare conversations. A multimodal sentiment analysis dataset comprising 30K samples across five languages is constructed. Rationale-augmented training improves classification accuracy and macro-F1 by approximately 2%.
- Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classification
-
Proposes a political bias analysis framework for LLMs based on Target-Oriented Sentiment Classification (TSC). By substituting the names of 1,319 politicians into 450 political sentences and predicting sentiments using 7 models across 6 languages, this study defines an entropy-based inconsistency metric to quantify bias. The findings reveal that LLMs exhibit a positive bias toward left-wing and centrist politicians and a negative bias toward the far-right, with larger models demonstrating stronger and more consistent biases.
- A Variational Approach for Mitigating Entity Bias in Relation Extraction
-
Proposes applying Variational Information Bottleneck (VIB) to entity debiasing in relation extraction. By mapping entities to a probability distribution \(\mathcal{N}(\mu, \sigma)\) to compress entity-specific information while retaining task-relevant features, the variance \(\sigma^2\) can quantify the model's level of reliance on entities vs. context. It achieves SOTA on both ID and OOD settings across three domains: TACRED, REFinD, and BioRED.
- Adapting Psycholinguistic Research for LLMs: Gender-Inclusive Language in a Coreference Context
-
By adapting the psycholinguistic experiment of Tibblin et al. (2023) from French to English and German LLMs, this work measures coreferent word probabilities and analyzes generated content. The findings show that: English LLMs generally maintain antecedent-coreference gender consistency, but singular they is rarely used and an underlying masculine bias persists. The German Leo Mistral 7B model exhibits a stronger masculine bias that dominates all 8 gender-inclusive strategies; nevertheless, these inclusive strategies still increase the probability of feminine/neutral gender occurrences, aligning with the results of human psycholinguistic experiments.
- Automatic Generation of Inference Making Questions for Reading Comprehension Assessments
-
A reading comprehension inference question taxonomy (pronominal bridging / text-connecting / gap-filling) is developed to automatically generate multiple-choice questions for specific inference types using GPT-4o few-shot prompting; while 93.8% of the questions are of acceptable quality, only 42.6% accurately match the target inference type, indicating LLMs still lack precise control over their reasoning abilities.
Browse all 30 NLP Understanding papers →
✍️ Text Generation (27)¶
- Unveiling Attractor Cycles in Large Language Models: A Dynamical Systems View of Successive Paraphrasing
-
Starting from dynamical systems theory, this paper reveals that during successive paraphrasing, the outputs of LLMs converge to stable 2-period attractor cycles instead of exploring a broad paraphrase space, uncovering inherent limitations in the generation capabilities of LLMs.
- What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentations
-
This paper proposes the VISTA dataset consisting of 18,599 AI conference presentation videos paired with paper abstracts, and introduces a plan-based summarization framework that guides structured summary generation for scientific videos by generating intermediate question sequences, significantly improving factual consistency.
- CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generation
-
Proposes CoCoLex, a training-free decoding strategy that constructs a copy distribution using the Euclidean distance between decoding hidden states and context token hidden states. By using a prediction entropy-based confidence score to dynamically balance the ratio of "copying from context" and "free generation", it consistently improves faithfulness and correctness across five legal benchmarks, showcasing particularly outstanding performance in long-text generation tasks.
- DTCRS: Dynamic Tree Construction for Recursive Summarization
-
DTCRS is proposed to dynamically construct summary trees based on document structure and query semantics. By using question decomposition and sub-question guided clustering, it reduces redundant summary nodes and significantly outperforms the static summary tree method RAPTOR on three QA datasets.
- Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems
-
Through a literature review and crowdsourcing study, this work systematically compiles 21 categories of interventions to mitigate anthropomorphism in text generation system outputs. It proposes a four-dimensional conceptual framework encompassing intervention type, target behavior, operationalization, and negative impact, providing the most comprehensive infrastructure for deanthropomorphization research.
- Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generation
-
This paper proposes a systematic evaluation framework based on Context-Preserving Prefix Trees (CP-Trie) to evaluate the intrinsic adaptability of truncation sampling methods between diversity and risk using probability-free and tuning-free metrics, providing practical guidance for parameter selection in real-world applications.
- Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework
-
To address the trade-off among multiple metrics (coherence/diversity/perplexity) in open-ended text generation, this paper proposes three complementary multi-criteria evaluation methods: the Extended Bradley-Terry model (ordinal ranking), Union-Free Generic Depth (partial ordering allowing incomparability), and Q*Text (cardinal comprehensive evaluation metric). Validated on over 1.8 million generated texts across 6 LLMs, 59 decoding strategies, and 3 datasets, the results show that moderate hyperparameter configurations generally outperform extreme ones, and smaller models with appropriate decoding strategies can match the performance of larger models.
- A Representation Level Analysis of NMT Model Robustness to Grammatical Errors
-
A systematic representation-level analysis of how NMT encoders process grammatical errors reveals that encoders first "detect" errors in shallow layers (indicated by rising GED probing \(F1\)), and then "correct" them in deep layers (indicated by falling CKA distance). It proposes the concept of "Robustness Heads" to identify the specific attention heads involved in error correction, validating this two-stage "detection-then-correction" mechanism across 4 models \(\times\) 5 language directions.
- An Empirical Study of Many-to-Many Summarization with Large Language Models
-
This work presents the first systematic study of Large Language Model (LLM) performance on the Many-to-Many Summarization (M2MS) task. By integrating 8 datasets, the authors construct a benchmark containing 47.8K samples across 5 domains and 6 languages. Evaluating 18 LLMs reveals that zero-shot LLMs perform comparably to fine-tuned traditional models, and significantly outperform them after instruction tuning. However, factual consistency remains a critical bottleneck.
- ATGen: A Framework for Active Text Generation
-
The authors propose ATGen, the first systematic active learning (AL) framework for NLG. It integrates state-of-the-art (SOTA) AL strategies, human/LLM annotation interfaces, parameter-efficient fine-tuning (PEFT), and vLLM inference optimization. Evaluation on four NLG tasks (including TriviaQA and GSM8K) demonstrates that active learning can reduce annotation costs by 2 to 4 times.
Browse all 27 Text Generation papers →
🗣️ Dialogue Systems (18)¶
- Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profiles
-
This paper proposes the USP (User Simulator with Implicit Profiles) framework. By extracting implicit user profiles from human-machine dialogues and combining conditional supervised fine-tuning with cycle-consistency-based reinforcement learning, USP significantly outperforms baseline methods across three dimensions: authenticity, consistency, and diversity, improving semantic similarity and style similarity by approximately 34% and 43%, respectively.
- UniConv: Unifying Retrieval and Response Generation for Large Language Models in Conversations
-
This work explores how to unify dense retrieval and response generation in conversational scenarios into a single LLM. Through three joint training objectives (conversational retrieval, response generation, and context identification instruction) and a data discrepancy mitigation mechanism, it achieves mutual reinforcement of retrieval and generation across five conversational search datasets, outperforming pipeline-based baselines.
- DEMO: Reframing Dialogue Interaction with Fine-grained Element Modeling
-
This paper proposes dialogue element modeling (DEMO), a novel task that systematically defines a comprehensive element taxonomy across the dialogue lifecycle from "prelude" to "epilogue." Based on this, the authors construct the DEMO benchmark covering both element awareness and dialogue agent interaction capabilities, and train DEMO agents using imitation learning, achieving superior performance on both in-domain and out-of-domain tasks.
- Dialogue Systems for Emotional Support via Value Reinforcement
-
This paper proposes ES-VR, the first method that integrates human value reinforcement into emotional support dialogue systems. By leveraging a target value detector and a reference generator (both trained on Reddit data), combined with a two-stage SFT + DPO training scheme, the supporter model not only alleviates the seeker's negative emotions but also explores and reinforces their positive values, achieving a deeper, internal transformation.
- Dynamic Label Name Refinement for Few-Shot Dialogue Intent Classification
-
Proposes a dynamic label name refinement method that utilizes LLMs to dynamically generate more distinctive intent label names (e.g., "Verify PAN" → "Verify PAN card details") based on retrieved examples in retrieval-based ICL intent classification. This effectively reduces confusion between semantically similar intents, consistently improving accuracy by 2.07%-7.51% across 6 datasets.
- Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System
-
This paper proposes an immersive multimodal conversation system that endows chatbots with "eyes and ears." It constructs the M3C dataset, a multi-session multi-party dialogue dataset integrating vision and audio, and designs a dialogue model consisting of a dialogue module and a multimodal memory retrieval module, enabling dynamic, long-term conversations where multiple speakers share audiovisual experiences.
- Enhancing Goal-oriented Proactive Dialogue Systems via Consistency Reflection and Correction
-
A model-agnostic two-stage CRC framework (Consistency Reflection & Correction) is proposed. By first prompting the model to reflect on inconsistencies between the generated response and the dialogue context and then correcting the response accordingly, it significantly improves the consistency of generated responses with the dialogue context in goal-oriented proactive dialogue systems.
- EnSToM: Enhancing Dialogue Systems with Entropy-Scaled Steering Vectors for Topic Maintenance
-
EnSToM is proposed as a lightweight method based on entropy-scaled steering vectors, which dynamically adjusts steering intensity by leveraging the differences in internal layer entropy distributions of LLMs to enhance the topic maintenance capability of task-oriented dialogue systems without modifying model parameters.
- Know Your Mistakes: Towards Preventing Overreliance on Task-Oriented Conversational AI Through Accountability Modeling
-
This paper proposes an Accountability Model for task-oriented dialogue systems, which integrates an additional accountability head as a binary classifier into LLMs to predict the probability of each slot in dialogue states. This enables the detection and self-correction of false positive and false negative errors, improving JGA from 64.34 to 70.51 (↑9.6%) on MultiWOZ and achieving SOTA.
- KokoroChat: A Japanese Psychological Counseling Dialogue Dataset Collected via Role-Playing by Trained Counselors
-
This paper proposes KokoroChat, a Japanese psychological counseling dialogue dataset collected via role-playing by trained counselors, consisting of 6,589 long sessions and detailed client feedback ratings, designed to enhance the counseling response generation and dialogue evaluation capabilities of LLMs.
Browse all 18 Dialogue Systems papers →
🌐 Multilingual & Translation (86)¶
- LACA: Improving Cross-lingual Aspect-Based Sentiment Analysis with LLM Data Augmentation
-
The LACA framework is proposed to leverage LLMs to generate high-quality pseudo-labeled data in the target language (rather than relying on machine translation). This significantly improves cross-lingual ABSA performance across six languages, outperforming the previous SOTA by an average of 1.50% and 2.62% on mBERT and XLM-R, respectively.
- Data Quality Issues in Multilingual Speech Datasets: The Need for Sociolinguistic Awareness and Proactive Language Planning
-
A systematic quality audit of three major public multilingual speech datasets (Common Voice 17.0, FLEURS, VoxPopuli) covering 40+ languages is conducted. Issues are categorized into programmable "micro-level issues" and linguistically involved "macro-level issues." It is found that macro-level issues are particularly severe for low-institutionalization languages. A 5-step dataset creation guide incorporating sociolinguistic awareness is proposed.
- Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization
-
This paper systematically evaluates the correlation of n-gram and neural network evaluation metrics with human judgments across 8 languages (representing 4 morphological typology families). The authors find that n-gram metrics negatively correlate with human judgments in highly fusional languages (Arabic, Hebrew), whereas COMET, a specially trained neural metric, consistently outperforms other methods across all language typologies.
- Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models
-
This paper proposes an automated approach based on beam search and LLM simulation to efficiently generate bilingual question pairs that expose cross-lingual performance weaknesses of multilingual LLMs in target languages. It establishes a dataset of over 6,000 samples across 16 languages, revealing that even GPT-4o suffers a cross-lingual accuracy drop exceeding 30%.
- EXECUTE: A Multilingual Benchmark for LLM Token Understanding
-
This paper extends the character understanding benchmark CUTE to 8 languages and multiple writing systems, proposing the EXECUTE framework. The study demonstrates that LLM performance varies dramatically across character, word, and sub-character levels in different languages, and counter-intuitively finds that LLMs perform better on token understanding tasks for less familiar languages.
- Trans-Zero: Self-Play Incentivizes Large Language Models for Multilingual Translation
-
The Trans-Zero self-play framework is proposed, which utilizes only monolingual data. By exploring semantically consistent candidate translations during the multilingual translation process through Genetic Monte-Carlo Tree Search (G-MCTS) and combining this with preference optimization, it achieves parallel-data-free multilingual translation training with performance comparable to large-scale supervised fine-tuning (SFT) methods.
- Translation and Fusion Improves Zero-shot Cross-lingual Information Extraction
-
TransFusion is proposed, which first translates low-resource language texts into English at inference time, performs information extraction annotation on English, and then uses a fusion model to combine English annotations with the source text to generate final predictions. It significantly outperforms baselines on zero-shot cross-lingual IE tasks across 50 languages (increasing average F1 on MasakhaNER2 from 47.9 to 62.4).
- Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learning
-
Through a systematic analysis of multilingual ICL strategies, this study reveals that mixing demonstrations of various high-resource languages (HRLs) in the prompt consistently outperforms purely English demonstrations, yielding particularly significant improvements on low-resource languages (LRLs) (e.g., an 8.9% to 12.6% average LRL accuracy gain on Llama 3.1). Intriguingly, even merely appending context-irrelevant non-English sentences to the prompt yields measurable gains, revealing the phenomenon that "multilingual exposure itself is effective."
- Language Fusion for Parameter-Efficient Cross-lingual Transfer (FLARE)
-
FLARE fuses layer-wise representations of the source (English) and target languages via lightweight linear/non-linear transformations within the low-rank bottleneck of LoRA adapters. It achieves parameter-efficient cross-lingual transfer without requiring extra parameters, improving QA exact match by 4.9% on Llama 3.1.
- NameTag 3: A Tool and a Service for Multilingual/Multitagset NER
-
This paper introduces NameTag 3, an open-source multilingual, multi-dataset, and multi-tagset named entity recognition (NER) tool and cloud service. Based on fine-tuned pre-trained language models, a single 355M parameter model achieves SOTA performance on 21 test sets across 15 languages, while being over 10,000 times faster than LLMs such as DeepSeek-R1.
Browse all 86 Multilingual & Translation papers →
🔍 Information Retrieval & RAG (84)¶
- The Distracting Effect: Understanding Irrelevant Passages in RAG
-
This paper proposes a formal metric for the Distracting Effect (DE) of passages and develops multiple techniques to acquire highly distracting passages (including answer-skewed retrieval and categorized generation). It demonstrates the robustness of this metric across different LLMs, and finally improves QA accuracy by up to 7.5% through fine-tuning LLMs with highly distracting training samples.
- PersonaBench: Evaluating AI Models on Understanding Personal Information through Accessing (Synthetic) Private User Data
-
Proposes a synthetic data generation pipeline to create the PersonaBench benchmark, which contains diverse user personas and simulated private documents (chat logs, AI interactions, purchase history). It is designed to evaluate AI models' ability to extract personal information from noisy user data. Experimental results show that current RAG formulations are far from sufficient for this task.
- Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence
-
This paper presents the first systematic study of the individual and combined effects of multiple heuristic biases (brevity, position, literal, and repetition biases) in dense retrievers. It reveals that when multiple biases are compounded, the probability of a retriever selecting the document containing the answer drops below 10%, and these biases can be exploited to manipulate RAG systems, leading to a 34% drop in performance.
- RAEmoLLM: Retrieval Augmented LLMs for Cross-Domain Misinformation Detection Using In-Context Learning Based on Emotional Information
-
This paper proposes RAEmoLLM, the first RAG framework based on emotional information retrieval. It utilizes the implicit embeddings of an emotional LLM to construct a retrieval database, providing emotion-related few-shot examples for cross-domain misinformation detection. Without the need for fine-tuning, RAEmoLLM improves performance by up to 15.64%, 31.18%, and 15.73% on three benchmarks respectively compared to other few-shot methods.
- HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generation
-
This paper proposes HoH, the first large-scale dynamic benchmark specifically designed to evaluate the impact of outdated information on RAG systems, containing 96,124 QA pairs and 219,463 documents, revealing the severe hazards of outdated information on RAG performance and safety.
- Health-LLM: Personalized Retrieval-Augmented Disease Prediction System
-
The Health-LLM framework is proposed, which integrates feature score extraction from health reports via LLM + Llama Index, RAG-augmented medical knowledge retrieval, and CAAFE automated feature engineering with an XGBoost classifier. It achieves an Accuracy of 0.833 and an F1 score of 0.762 for disease prediction on the IMCS-21 Chinese telemedicine dataset, significantly outperforming GPT-4 few-shot+RAG (Acc 0.68) and fine-tuned LLaMA-2-13B (Acc 0.73).
- InspireDebate: Multi-Dimensional Evaluation-Guided Reasoning for Debating
-
A two-component framework is proposed: InspireScore (a debate evaluation system integrating 4 subjective dimensions and 2 objective dimensions) and InspireDebate (a debate framework optimized through a three-stage process of CoT-SFT + multi-dimensional DPO + Web-RAG). This evaluation system improves correlation with expert judgment by 44%, and the debate performance surpasses the baseline by 57%.
- PRISM: A Framework for Producing Interpretable Political Bias Embeddings with Political-Aware Cross-Encoder
-
This work proposes the PRISM framework, which models political bias embedding as an interpretable task for the first time. It automatically extracts controversial topics and left/right bias indicators from a weakly labeled news corpus as embedding dimensions. Then, a political-aware cross-encoder is employed to score articles along each topic dimension, generating sparse and semantically transparent political bias embeddings. The proposed method achieves \(86.1\%\) accuracy on NewsSpectrum (outperforming POLITICS by \(34.8\%\)) while supporting diverse retrieval.
- HELIOS: Harmonizing Early Fusion, Late Fusion, and LLM Reasoning for Multi-Granular Table-Text Retrieval
-
Proposes HELIOS, a three-stage graph retrieval framework (edge-level early fusion \(\rightarrow\) node-level late fusion \(\rightarrow\) star-graph-level LLM refinement). Through multi-granular coordination, it addresses the three major challenges in table-text retrieval: retrieval unit granularity, query dependency relationship discovery, and advanced reasoning, achieving a 42.6% improvement in Answer Recall on OTT-QA.
- Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion
-
The authors question whether the performance gains of LLM-based query expansion (HyDE/Query2doc) truly stem from "hypothetical document generation." They find that performance improvements consistently occur only when the LLM-generated documents contain sentences semantically consistent with the gold evidence, revealing potential knowledge leakage issues in benchmarks.
Browse all 84 Information Retrieval & RAG papers →
💻 Code Intelligence (28)¶
- DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal
-
This paper proposes DARS (Dynamic Action Re-Sampling), an inference-time compute scaling method for coding agents. It dynamically branches and attempts alternative actions at key decision points where the agent makes suboptimal choices. Using Claude 3.5 Sonnet V2, DARS achieves a 55% pass@k and 47% pass@1 on SWE-Bench Lite, outperforming the open-source SOTA frameworks of the time.
- TeXpert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLMs
-
Proposes TeXpert, the first multi-difficulty level benchmark for systematically evaluating LLMs' capability to generate LaTeX code for scientific documents from natural language instructions. It contains 440 high-quality samples across three levels (Simple/Average/Hard). Evaluation of 9 open-source and closed-source LLMs reveals that LaTeX generation remains a significant weakness (accuracy on the Hard task is generally below 17.5%), with logical and formatting errors being the primary bottlenecks.
- ExploraCoder: Advancing Code Generation for Multiple Unseen APIs via Planning and Chained Exploration
-
This work proposes the training-free ExploraCoder framework. It decomposes complex multi-API programming problems into subtasks through task planning, and progressively conducts experiments to accumulate experiences on correct API usages via Chained API Exploration (CoAE). It achieves up to 17.28% absolute improvement in pass@10 on multi-API unseen library benchmarks.
- Beyond Sequences: Two-dimensional Representation and Dependency Encoding for Code Generation
-
This paper proposes a two-dimensional code representation that moves beyond traditional one-dimensional sequence representations. By explicitly encoding the structural dependency relationships of code (such as syntax tree structures and variable dependencies), it significantly improves the accuracy and structural correctness of code generation.
- CodeDPO: Aligning Code Models with Self Generated and Verified Source Code
-
Proposes CodeDPO, which constructs high-quality preference pairs (93K correctness + 21K efficiency) from self-generated code via a PageRank-inspired self-validation scoring mechanism. After DPO training, it achieves an average improvement of over 10 points on HumanEval across 8 code models, while accelerating code execution efficiency by 1.25-1.45×.
- FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
-
FEA-Bench is proposed as the first benchmark evaluating LLM capabilities in feature implementation within repository-level codebases. It contains 1,401 task instances from 83 GitHub repositories, with each instance equipped with unit tests. The strongest model, DeepSeek-R1, solves only about 10% of the tasks, revealing the significant challenges repository-level incremental development poses to current LLMs.
- LongCodeU: Benchmarking Long-Context Language Models on Long Code Understanding
-
The authors propose the LongCodeU benchmark, which designs 8 tasks across four dimensions—code unit perception, intra-code unit understanding, inter-code unit relation understanding, and long documentation understanding—to evaluate the comprehension capabilities of 9 long-context language models (LCLMs) on real-world, repository-level long code, revealing that 32K tokens is the practical upper limit for current LCLM long code understanding.
- SceneGenAgent: Precise Industrial Scene Generation with Coding Agent
-
This work proposes SceneGenAgent, an LLM-based code-generation agent. Through structured layout planning, layout verification, and iterative refinement, it utilizes C# code to generate industrial scenes with high precision. It achieves an 81% success rate on real industrial tasks and constructs the SceneInstruct dataset, enabling open-source LLMs to perform closely to GPT-4o.
- STaR-SQL: Self-Taught Reasoner for Text-to-SQL
-
This paper reformulates the Text-to-SQL task as a reasoning-driven process. By employing the STaR (Self-Taught Reasoner) bootstrapping approach, it enables LLMs to learn how to generate step-by-step rationales to assist in SQL generation. Integrated with an Outcome-supervised Reward Model (ORM) validator for best-of-N sampling, the framework achieves an 86.6% execution accuracy on the Spider benchmark.
- CoCo-Bench: A Comprehensive Code Benchmark for Multi-task Large Language Model Evaluation
-
This paper introduces CoCo-Bench (Comprehensive Code Benchmark), a comprehensive code benchmark covering four dimensions: code understanding, code generation, code modification, and code review. It supports multiple programming languages and difficulty levels, ensures data quality through rigorous manual review, and reveals the unbalanced performance of existing LLMs in coding capabilities.
Browse all 28 Code Intelligence papers →
🎨 Image Generation (9)¶
- Multimodal Pragmatic Jailbreak on Text-to-image Models
-
This paper proposes a new type of attack called "Multimodal Pragmatic Jailbreak" (MPJ). By generating images containing visual text through T2I models, the image content and text content are safe when evaluated individually but yield unsafe content when combined. This study reveals that all tested models, including DALL·E 3, are vulnerable to this attack.
- D-GEN: Automatic Distractor Generation and Evaluation for Reliable Assessment of Generative Models
-
Proposes D-GEN—the first open-source distractor generation model (fine-tuned LLaMA, 8B/70B) that automatically converts open-ended evaluation questions into multiple-choice formats, paired with two evaluation methods (ranking alignment and entropy analysis) to verify distractor quality, maintaining model ranking consistency with Spearman's ρ=0.99 on MMLU.
- A Unified Agentic Framework for Evaluating Conditional Image Generation
-
CIGEval is proposed as a unified evaluation framework based on Large Multimodal Model (LMM) Agents. By integrating various tools (Grounding, Highlight, Difference, Scene Graph) and adopting a divide-and-conquer evaluation strategy, it achieves correlation comparable to human annotators (0.4625 vs. human-to-human 0.47) across 7 conditional image generation tasks, and surpasses the SOTA GPT-4o baseline by fine-tuning a 7B model on only 2.3K training samples.
- FlashAudio: Rectified Flows for Fast and High-Fidelity Text-to-Audio Generation
-
This paper introduces Rectified Flow to text-to-audio generation. By leveraging bifocal samplers to optimize timestep distribution, immiscible flow to minimize total data-noise distance, and anchored optimization to correct CFG guidance errors, the proposed method achieves single-step generation with a FAD of 1.49, outperforming 100-step diffusion models while reaching a generation speed of 400x real-time.
- Generating Pedagogically Meaningful Visuals for Math Word Problems: A New Benchmark and Analysis of Text-to-Image Models
-
Math2Visual proposes a framework to automatically generate pedagogical visualizations from textual descriptions of math word problems (MWPs). It defines a visual language and design space based on teacher interviews, constructs a labeled dataset of 1,903 images, and evaluates and fine-tunes multiple TTI models, revealing key deficiencies of current models in representing mathematical relationships.
- OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matching
-
This paper proposes OZSpeech, the first zero-shot TTS system that combines Optimal Transport Conditional Flow Matching (OT-CFM) with a learned prior distribution to achieve one-step sampling. It significantly outperforms existing approaches in content accuracy (WER), inference speed, and model size.
- Planning with Diffusion Models for Target-Oriented Dialogue Systems
-
DiffTOD models dialogue planning as a trajectory generation problem, leveraging a masked diffusion language model to achieve non-sequential dialogue planning. It designs three guidance mechanisms (word-level/semantic-level/search-based) to flexibly control the dialogue toward the target, significantly outperforming baselines in negotiation, recommendation, and chitchat scenarios.
- R-VC: Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching
-
R-VC is the first zero-shot voice conversion system to achieve rhythm control. It models the target speaker's rhythm style using a Mask Transformer duration model, combined with a Shortcut Flow Matching DiT decoder to achieve efficient and high-quality speech generation in only 2 sampling steps, achieving a WER of 3.51 and speaker similarity of 0.930 on LibriSpeech.
- Synthia: Novel Concept Design with Affordance Composition
-
Synthia proposes a novel concept design framework based on affordance composition. By leveraging a hierarchical concept ontology, an affordance sampling strategy, and curriculum learning to fine-tune a T2I model, it generates innovative designs that are both visually novel and functionally coherent.
🎬 Video Generation (2)¶
- Q2E: Query-to-Event Decomposition for Zero-Shot Multilingual Text-to-Video Retrieval
-
Q2E proposes a zero-shot query-to-event decomposition method. It leverages the parameterized world knowledge of LLMs and VLMs to decompose simple queries into prequel, current, and sequel events. Combining these with dense video descriptions and speech transcriptions, it achieves SOTA multilingual text-to-video retrieval performance through inverse entropy fusion ranking.
- VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generation
-
This work proposes VidCapBench, the first video captioning evaluation benchmark designed specifically for controllable text-to-video (T2V) generation. It evaluates caption quality across four dimensions: aesthetics, content, motion, and physical laws. Comprising 643 videos and 10,644 QA pairs, experiments demonstrate that VidCapBench scores are highly positively correlated with T2V generation quality.
🧩 Multimodal VLM (111)¶
- ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation
-
Propses ChartCoder, the first dedicated chart-to-code MLLM. Using a Code LLM as the language backbone, combined with a 160K large-scale chart-to-code dataset and a "Snippet-of-Thought" step-by-step reasoning approach, the 7B model beats all open-source MLLMs across three benchmarks and approaches the performance of GPT-4o.
- HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model
-
Through CKA analysis, it is discovered that the top layer of MLLM learns task-specific information while the remaining layers learn general knowledge. This paper proposes HiDe-LLaVA: MoE-style task-specific expansion for top-layer LoRA (using dual-modal anchor matching) paired with uniform merging for LoRA in the remaining layers. On the newly constructed UCIT benchmark (free of information leakage), it achieves a 5.8% improvement over the best baseline.
- MAGIC-VQA: Multimodal and Grounded Inference with Commonsense Knowledge for Visual Question Answering
-
The paper proposes the MAGIC-VQA framework, which systematically injects external commonsense knowledge into LVLMs through a three-stage pipeline (explicit knowledge retrieval \(\rightarrow\) type-based post-processing \(\rightarrow\) GNN-based implicit enhancement). This achieves plug-and-play commonsense reasoning enhancement on benchmarks like ScienceQA, TextVQA, and MMMU, requiring only 0.33M trainable parameters.
- Can Vision-Language Models Evaluate Handwritten Math?
-
This paper proposes the FERMAT benchmark to systematically evaluate the error detection, localization, and correction capabilities of 9 VLMs on handwritten mathematical content. Using 609 manually curated Grade 7-12 math problems alongside over 2,200 handwritten erroneous solutions (covering computation, conceptual, notation, and formatting errors), the evaluation reveals that Gemini-1.5-Pro achieves the highest correction rate of 77%, though all models still face significant challenges when processing handwritten content.
- iNews: A Multimodal Dataset for Modeling Personalized Affective Responses to News
-
A personalized affective annotation dataset, iNews, is constructed, containing annotations from 291 UK annotators on 2,899 multimodal Facebook news posts. Annotator profiles (demographics, personality, media trust, etc.) explain 15.2% of the annotation variance, and combining persona information with LLM zero-shot prediction improves accuracy by up to 7%.
- A Survey on Patent Analysis: From NLP to Multimodal AI
-
A systematic survey of NLP and multimodal AI applications in four core patent analysis tasks (classification, retrieval, quality analysis, and generation), proposing a taxonomy based on the patent lifecycle, and revealing methodological evolution trends from Word2Vec+LSTM to BERT/GPT and multimodal models, along with key research gaps.
- Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
-
This paper proposes the Synchronously Self-Reviewing (SSR) paradigm. By requiring the MLLM to first generate OCR text before outputting the translation during the document image translation process, SSR leverages the "bilingual cognitive advantage" to alleviate catastrophic forgetting caused by fine-tuning, while simultaneously enhancing both OCR and Document Image Machine Translation (DIMT) performance.
- A Parameter-Efficient and Fine-Grained Prompt Learning for Vision-Language Models
-
This paper proposes the DoPL (Detail-oriented Prompt Learning) method. Based on the theory of low-entropy information concentration, it discovers shared-interest tokens between text and vision. It uses these to construct alignment weights to enhance text and visual prompts. With only 0.25M (0.12%) trainable parameters, it achieves fine-grained multimodal semantic alignment, surpassing full-parameter fine-tuning methods on six benchmarks.
- MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering
-
This work introduces MTVQA, the first multilingual text-centric visual question answering benchmark covering 9 languages. It resolves the "vision-text misalignment" issue of translation-based approaches through human expert annotation. Evaluations reveal a substantial performance gap between the best MLLM (InternVL-2.5, 32.2%) and the human baseline (79.7%), highlighting the severe challenges of multilingual text understanding.
- Can Multimodal Large Language Models Understand Spatial Relations?
-
Proposes the SpatialMQA benchmark to evaluate the spatial relation reasoning capability of MLLMs in a multiple-choice format, revealing that the state-of-the-art model only achieves 48.14% accuracy, far below the human performance of 98.40%.
Browse all 111 Multimodal VLM papers →
🧠 VLM Reasoning (18)¶
- Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoning
-
This paper identifies a severe visual forgetting phenomenon in MLLMs during long CoT reasoning—removing the image halfway through reasoning only causes a ~2% drop in accuracy, indicating that models rely excessively on self-generated text while ignoring visual evidence. To address this, the authors propose a Take-along Visual Conditioning (TVC) strategy. It injects an image review mechanism via Dynamic Visual Reaffirmation (DVR) during the training phase, and compresses and re-injects visual tokens via Periodic Visual Calibration (PVC) during the inference phase. TVC outperforms the previous SOTA by 3.4 points on average (43.4 vs 40.0) across five mathematical reasoning benchmarks.
- LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating
-
This paper proposes the LongDocURL benchmark, which covers 20 subtasks across three primary task categories: understanding, numerical reasoning, and cross-element locating. It contains 2,325 high-quality QA pairs spanning over 33,000 pages of documents. A systematic evaluation of 26 model configurations exposes key performance gaps of current LVLMs in long document understanding.
- AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large Language Models on Harmfulness
-
This work proposes AdamMeme—an adaptive evaluation framework based on multi-agent collaboration, which probes the reasoning capabilities and specific weaknesses of Multimodal Large Language Models (mLLMs) in harmful content understanding by iteratively generating more challenging meme samples.
- Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
-
This paper proposes a method to transfer the reasoning capabilities of LLMs to VLMs. Through improved chart representation pre-training, construction of large-scale synthetic reasoning datasets, and multi-task fine-tuning, the 5B-parameter PaLI-3 model outperforms models 10 times its size on ChartQA.
- FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning
-
This work constructs FCMR, a cross-modal multi-hop reasoning benchmark in the financial domain, comprising three modalities: text, tables, and charts. It is categorized into three difficulty levels: Easy, Medium, and Hard. The strongest model, Claude 3.5 Sonnet, achieves only 30.4% accuracy on the Hard level, revealing critical bottlenecks of MLLMs in the information retrieval phase.
- FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
-
This work constructs FinMME, an evaluation benchmark containing over 11,000 high-quality financial multimodal samples across 18 financial domains and 10 chart types. It proposes the FinScore evaluation framework, which integrates hallucination penalties with domain normalization. Experimental results show that even GPT-4o scores only 15.34 (with an average accuracy of 46.56%), revealing significant deficiencies of MLLMs in the financial domain.
- Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?
-
This work systematically evaluates 13 open-source small LVLMs (\(\le 9\text{B}\) parameters) serving as judges for chart comprehension and reasoning tasks. It finds that some open-source models (e.g., LLaVA-Critic-7B) can achieve evaluation capabilities close to GPT-4 (about 80% agreement rate), though issues like positional bias and length bias remain prevalent.
- MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning
-
This paper proposes leveraging code as a supervisory signal for cross-modal alignment to construct the ImgCode-8.6M dataset consisting of 8.6 million image-code pairs, and the MM-MathInstruct-3M dataset containing 3 million multimodal mathematical instruction-tuning samples. The trained MathCoder-VL achieves State-of-the-Art (SOTA) performance in multimodal mathematical reasoning among open-source models, outperforming GPT-4o and Claude 3.5 Sonnet on geometry problems.
- MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibration
-
This paper proposes the MMBoundary framework, which inserts natural language confidence statements at each step of the reasoning chain (rather than offering confidence only after the final answer). It combines textual and cross-modal self-reward signals to estimate confidence, and utilizes a two-stage training paradigm of SFT and RL to achieve step-level confidence calibration, reducing the calibration error by an average of 7.5% and improving task accuracy by 8.3%.
- Progressive Multimodal Reasoning via Active Retrieval
-
This paper proposes the AR-MCTS framework, which combines Active Retrieval with Monte Carlo Tree Search (MCTS) to dynamically retrieve key knowledge at each step of multi-step multimodal reasoning, replacing traditional beam search sampling. It automatically generates step-by-step reasoning annotations to progressively align the Process Reward Model (PRM), significantly improving the reasoning performance of multiple MLLMs on MathVista, We-Math, and GAOKAO-MM.
Browse all 18 VLM Reasoning papers →
⚡ VLM Efficiency (8)¶
- OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
-
The authors propose OMGM, a multimodal RAG system for knowledge-bound visual question answering (KB-VQA). By orchestrating the matching between query and knowledge base across various granularities and modalities via a coarse-to-fine three-step retrieval strategy, OMGM achieves state-of-the-art retrieval performance and highly competitive VQA results on InfoSeek and E-VQA datasets.
- Sharper and Faster mean Better: Towards More Efficient Vision-Language Model for Hour-scale Long Video Understanding
-
Proposes the Sophia model to handle hour-scale long videos: accurately selects query-relevant frames via Shot-adaptive Frame Pruning (a two-stage frame pruning based on shot segmentation), and replaces full attention with Hierarchical Attention of \(O(N)\) complexity. It achieves state-of-the-art (SOTA) performance on 6 out of 8 long video benchmarks, while requiring only 1/8.5 of the attention FLOPs compared to InternVL2.
- EffiVLM-Bench: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models
-
Proposes EffiVLM-Bench, a unified evaluation framework to systematically evaluate training-free acceleration methods (token compression + parameter compression) for LVLMs across four dimensions: performance, generalization, faithfulness, and efficiency. Spanning 3 cutting-edge models and 17 benchmark tasks, it reveals the Pareto-optimal trade-offs of various methods under different compression rates.
- Hierarchical Safety Realignment: Lightweight Restoration of Safety in Pruned Large Vision-Language Models
-
This paper proposes Hierarchical Safety Realignment (HSR), a method that first identifies safety-critical attention heads and then locates and restores safety-critical pruned neurons within these heads. With minimal parameter overhead (on the order of ten-thousandths), HSR significantly recovers the safety performance lost in pruned LVLMs.
- HotelMatch-LLM: Joint Multi-Task Training of Small and Large Language Models for Efficient Multimodal Hotel Retrieval
-
This paper proposes HotelMatch-LLM, an asymmetric architecture that employs an SLM to encode queries and an LLM to encode hotel documents. Combined with a tri-objective multi-task optimization (retrieval alignment + MLM geographic prediction + visual facility recognition) and patch-level mean pooling for multi-image processing, it significantly outperforms SOTA methods like MARVEL and VISTA on travel-domain multimodal retrieval tasks.
- RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMs
-
This work proposes the RedundancyLens framework to systematically reveal the extensive structured and clustered redundancy in self-attention and FFN operations for visual tokens within decoder-only MLLMs. Leveraging this finding, training-free inference acceleration is achieved, which is orthogonal to and combinable with existing token compression methods.
- Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?
-
Large-scale benchmark experiments reveal several fundamental issues with current visual token pruning methods for MLLMs: elaborately designed pruning strategies (such as FastV and SparseVLM) underperform even naive methods like random selection and pooling on most benchmarks. This is due to positional bias in attention scores, misuse of language information, imbalance between importance and redundancy, and unreliable evaluation metrics.
- MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference
-
This paper proposes MadaKV, a modality-aware KV cache eviction strategy. Through two core components—Modality Preference Adaptation (MPA) and Hierarchical Compression Compensation (HCC)—MadaKV significantly reduces KV cache memory consumption (by 80-95%) and decoding latency (1.3x to 1.5x speedup) while maintaining performance on multimodal long-context tasks.
🎵 Audio & Speech (47)¶
- TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis
-
TCSinger 2 is proposed as a multi-task multilingual zero-shot singing voice synthesis model. Through a blurred boundary encoder, a contrastive learning audio encoder, and a Flow-based customized Transformer (incorporating Cus-MOE), it achieves style transfer and multi-level style control based on singing, speech, or text prompts.
- SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Models
-
SpeechWeave proposes an end-to-end synthetic speech data generation pipeline that enhances text diversity through keyphrase sampling, performs text normalization at the generation stage (achieving \(97\%\) accuracy), and uses cross-lingual voice cloning to standardize speakers. The generated data is \(10\)-\(48\%\) more diverse than direct LLM prompting and significantly improves downstream TTS model performance.
- SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models
-
Proposes SpeechIQ, a hierarchical speech understanding evaluation framework based on Bloom's Taxonomy. It comprehensively assesses the intelligence of speech LLMs across three levels: Remember (WER), Understand (semantic similarity), and Apply (QA accuracy). The study reveals that cascaded ASR+LLM systems outperform end-to-end multimodal models at equivalent scales.
- Investigating and Enhancing Vision-Audio Capability in Omnimodal Large Language Models
-
It is discovered that current Omnimodal Large Language Models (OLLMs) perform significantly worse on vision-audio tasks than on vision-text tasks. The primary reason is the lack of direct alignment between vision and audio modalities. Consequently, this work proposes Self-KD (Self-Knowledge Distillation) to enhance vision-audio capabilities by leveraging the OLLM's own vision-text components as a teacher.
- GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages
-
GigaSpeech 2 constructs a large-scale ASR corpus of approximately 30,000 hours for low-resource languages (Thai, Indonesian, and Vietnamese). Through an automated crawling-transcription-refinement pipeline, high-quality pseudo-labels are generated from unlabeled YouTube videos. The trained model reduces WER by 25%-40% compared to Whisper large-v3 while utilizing only 10% of its parameters.
- Amplifying Trans and Nonbinary Voices: A Community-Centred Harm Taxonomy for LLMs
-
This paper adopts a community-centred research methodology, building a specialized harm taxonomy for LLM outputs affecting Trans and Nonbinary (TNB) individuals through deep collaboration with the TNB community, thereby uncovering unique harm categories unaddressed by existing LLM safety evaluations.
- Dialectal Coverage and Generalization in Arabic Speech Recognition
-
This study systematically investigates the impact of Arabic dialect coverage on ASR performance. By utilizing multi-dialectal pre-training and joint fine-tuning, the ArTST model is extended to cover speech variants from 17 Arabic countries, while multilingual optimization strategies in code-switching scenarios are additionally explored.
- MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder
-
MultiMed is introduced as the first multilingual medical ASR dataset (150 hours, 5 languages, 10 recording conditions, 16 accents), along with end-to-end Whisper model baselines spanning small to large scales. This work presents the first systematic study of multilingual medical ASR, comparing monolingual vs. multilingual fine-tuning and AED vs. Hybrid architectures. Key findings reveal that multilingual joint training benefits small models but can lead to performance degradation in larger models.
- Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
-
This paper proposes Spark-TTS, an efficient TTS system based on a novel single-stream speech codec, BiCodec, and the Qwen2.5 LLM. By decoupling speech into low-bitrate semantic tokens and fixed-length global tokens, Spark-TTS achieves zero-shot voice cloning and coarse-to-fine attribute control, reaching SOTA intelligibility on Seed-TTS-eval.
- Acoustic Individual Identification of White-Faced Capuchin Monkeys Using Joint Multi-Species Embeddings
-
This paper explores utilizing cross-species acoustic pre-trained embeddings from birds and humans to identify individual calls of white-faced capuchin monkeys. It discovers that joint multi-species representations can further enhance identification performance, providing a new transfer learning paradigm for individual identification of wild animals under extreme data scarcity.
Browse all 47 Audio & Speech papers →
🔎 AIGC Detection (15)¶
- People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text
-
Through an experiment with 1,740 annotations, it was found that human annotators who frequently use LLMs for writing tasks can detect AI-generated text with extremely high accuracy (only 1/300 errors via a 5-person majority vote). Even when facing paraphrasing and humanization evasion strategies, they perform significantly better than most automated detectors.
- Are We in the AI-Generated Text World Already? Quantifying and Monitoring AIGT on Social Media
-
This work offers the first large-scale quantification of the changing proportion of AI-Generated Text (AIGT) on social media. By collecting 2.4 million posts across Medium, Quora, and Reddit, constructing the AIGTBench dataset, and training the optimal detector OSM-Det, the study reveals that the AIGT proportion on Medium and Quora zoomed from ~2% to ~37-39% between 2022 and 2024, whereas Reddit's proportion only increased from 1.3% to 2.5%.
- A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI
-
This paper proposes using LLM-generated NLI explanations to substitute expensive human explanations for approximating Human Judgment Distributions (HJD). Experiments demonstrate that with the guidance of human label distributions, LLM-generated explanations achieve comparable performance to human explanations across metrics like KL divergence and JSD. Furthermore, the approach generalizes well to datasets without human explanations (MNLI) and out-of-domain test sets (ANLI).
- ChemActor: Enhancing Automated Extraction of Chemical Synthesis Actions with LLM-Generated Data
-
This paper proposes ChemActor, a fully fine-tuned LLM chemical executor, which addresses the data scarcity issue in chemical synthesis action extraction through a sequential LLM-generated data framework and a distribution divergence-based data selection module, outperforming baseline models by 10% on R2D and D2A tasks.
- Comparing LLM-generated and human-authored news text using formal syntactic theory
-
This study is the first to systematically investigate the grammatical differences between 6 LLMs and human NYT news writing across three levels—syntactic constructions (298 types), lexical types (1,398 types), and morphological rules (100 types)—using HPSG formal syntactic theory (via the English Resource Grammar, ERG). The findings reveal that LLMs represent an "averaged" projection of human authors in terms of grammatical features: grammatical differences among individual human authors are actually greater than any difference between humans and LLMs, while LLMs exhibit almost no differences among themselves.
- Learning to Rewrite: Generalized LLM-Generated Text Detection
-
The Learning2Rewrite (L2R) framework is proposed, which fine-tunes an LLM-based rewriting model to amplify the difference in rewrite edit distance between human-written and AI-generated text, thereby achieving highly generalized AI text detection across domains. L2R achieves an average AUROC of 0.9009 across 21 independent domains, outperforming RAIDAR by 4.67% and direct classification fine-tuning by 51.35% in out-of-distribution tests.
- MultiSocial: Multilingual Benchmark of Machine-Generated Text Detection of Social-Media Texts
-
Constructed MultiSocial (472k texts), the first large-scale machine-generated text detection benchmark covering 22 languages, 5 social media platforms, and 7 LLM generators. Experimental results show that fine-tuned detectors (Llama-3-8B/Mistral-7B, AUC ROC 0.977) perform exceptionally well on social media texts, and the choice of training platforms significantly impacts cross-platform generalization.
- HACo-Det: A Study Towards Fine-Grained Machine-Generated Text Detection under Human-AI Coauthoring
-
This study proposes HACo-Det, a fine-grained machine-generated text (MGT) detection benchmark tailored for human-AI collaborative writing. By employing a multi-round local rewriting pipeline, it automatically constructs 11,200 human-AI coauthored texts with word-level attribution labels. It adapts seven mainstream detectors into a word-level sequence labeling formulation for systematic evaluation, revealing significant room for improvement in current fine-grained detection methods.
- Iron Sharpens Iron: Defending Against Attacks in Machine-Generated Text Detection with Adversarial Training
-
This paper proposes the GREATER adversarial training framework, which simultaneously trains an adversarial attacker (Greater-A) and an MGT detector (Greater-D). The attacker identifies critical tokens through surrogate model gradients and perturbs them in the embedding space to generate adversarial samples. The detector learns generalized defense from these curriculum-style adversarial samples. Under 16 attacks, the ASR drops to 5.53% (compared to SOTA's 6.20%), while the attack efficiency is 4 times faster than SOTA.
- KatFishNet: Detecting LLM-Generated Korean Text through Linguistic Feature Analysis
-
This paper constructs the first Korean LLM-generated text detection benchmark, KatFish (covering three genres and four LLMs). By analyzing three types of Korean linguistic features—word spacing, POS diversity, and comma usage—the authors propose the KatFishNet detection method, achieving an average AUROC 19.78% higher than the best baseline under the OOD (unseen LLM) setting.
Browse all 15 AIGC Detection papers →
🎯 Object Detection (2)¶
- Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questions
-
This work provides the first mechanistic analysis of "anchored bias" (consistently choosing "A") in the GPT-2 family within multiple-choice questions (MCQs) from a failure-case perspective. It localizes specific value vectors storing the "A" preference in MLPs using Logit Lens, and achieves an average MCQ accuracy improvement of 70%+ through minimal intervention (updating the value vectors).
- Weed Out, Then Harvest: Dual Low-Rank Adaptation is an Effective Noisy Label Detector for Noise-Robust Learning
-
This paper proposes the Delora framework, which constructs a noisy label detector by introducing clean LoRA and noisy LoRA modules. By decoupling sample selection from model training, Delora breaks the vicious catch-22 cycle of mutual influence between sample selection and training in traditional "small-loss" approaches.
✂️ Segmentation (4)¶
- InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning
-
This work proposes InstructPart, the first real-world benchmark that combines task-oriented instructions with part-level segmentation, comprising 2,400 images, 48 object categories, 44 part categories, and 9,600 human-annotated task instructions. Evaluation shows that current VLMs are severely inadequate in instruction-driven part segmentation, while a baseline fine-tuned on LISA+DINOv2 achieves an approximate 100% performance gain.
- BERT-like Models for Slavic Morpheme Segmentation
-
This paper explores the use of BERT-like pretrained language models for morpheme segmentation tasks in Slavic languages. By modeling morpheme segmentation as a sequence labeling problem, the approach achieves results superior to traditional methods across multiple Slavic languages.
- DEF-DTS: Deductive Reasoning for Open-domain Dialogue Topic Segmentation
-
Proposed DEF-DTS, a dialogue topic segmentation method based on LLM multi-step deductive reasoning. Through a three-step pipeline of bidirectional context summarization \(\rightarrow\) utterance intent classification (5 classes) \(\rightarrow\) deductive topic shift judgment, it achieves unsupervised/prompt-based SOTA on three datasets: TIAGE, SuperDialseg, and Dialseg711, outperforming supervised methods on Dialseg711.
- Pixel-Level Reasoning Segmentation via Multi-turn Conversations
-
Proposes a new task of pixel-level reasoning segmentation (Pixel-level RS) to achieve fine-grained segmentation by progressively understanding user intent through multi-turn conversations. A PRIST dataset containing 24k dialogue turns is constructed, and a MIRAS framework is designed to outperform existing baselines in both segmentation accuracy and reasoning capability.
🖼️ Image Restoration (3)¶
- PreP-OCR: A Complete Pipeline for Document Image Restoration and Enhanced OCR Accuracy
-
Proposes the PreP-OCR two-stage pipeline: first restoring historical document images using a ResShift model trained on synthetically degraded data (employing multi-directional patch extraction and median fusion), and then applying ByT5 for semantic post-OCR error correction. It reduces CER by 63.9-70.3% across 13,831 pages of real-world historical documents.
- A Self-Denoising Model for Robust Few-Shot Relation Extraction
-
This paper proposes a Self-Denoising Model (SDM) to address the issue of support set label noise in few-shot relation extraction. Through the co-training of a label correction module and a relation classification module, SDM automatically corrects noisy labels and achieves more robust relation prediction, significantly outperforming baselines even in noise-free scenarios.
- DiffuseDef: Improved Robustness to Adversarial Attacks via Iterative Denoising
-
DiffuseDef inserts a diffusion denoiser layer between the encoder and the classifier. During training, it learns to predict noise in hidden states. During inference, it adds noise to the hidden representations, iteratively denoises them, and performs ensemble averaging. This plug-and-play approach significantly enhances the robustness of text classification models under both black-box and white-box adversarial attacks.
🧑 Human Understanding (2)¶
- TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments
-
This paper proposes TransBench, the first benchmark to systematically evaluate the transferability (cross-version/cross-platform/cross-app) of GUI Agents. It covers 81 Chinese Apps, 1459 screenshots, and 22K+ annotated instructions. Experiments show that fine-tuning on older versions can effectively transfer to new versions and other platforms, with Android data exhibiting the strongest generalization in cross-platform migration.
- Beyond Surface Simplicity: Revealing Hidden Reasoning Attributes for Precise Commonsense Diagnosis
-
This paper reveals that commonsense reasoning benchmarks contain issues that appear simple on the surface but actually imply complex hidden reasoning attributes, and proposes a fine-grained diagnostic framework based on hidden reasoning attributes, enabling a more precise analysis and evaluation of models' commonsense reasoning capabilities.
📹 Video Understanding (8)¶
- Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Models
-
This study presents the first systematic exploration of selection bias in multiple-choice question answering (MCQA) with Video Language Models (VLMs). By analyzing bias sources through task decomposition, it proposes BOLD, a post-processing calibration technique that reduces bias while simultaneously improving model performance.
- From Teacher to Student: Tracking Memorization Through Model Distillation
-
This work systematically investigates the impact of knowledge distillation (KD) on the memorization behavior of large language models, finding that distillation not only compresses the model but also significantly reduces the risk of verbatim memorization of training data—with reverse KL distillation (RKLD/MiniLLM) reducing the memorization ratio from 65.4% in SFT to as low as 6.0%.
- Generative Frame Sampler for Long Video Understanding
-
GenS is proposed, a generative frame sampling module based on VideoLLM. It outputs question-aware relevant frame intervals and confidence scores in natural language format. As a plug-and-play module, it consistently improves multiple VideoLLMs by 2-4 points on LongVideoBench, MLVU, and HourVideo.
- Improving Dialogue State Tracking through Combinatorial Search for In-Context Examples
-
This paper proposes CombiSearch, a method that employs combinatorial scoring to select the optimal combination of in-context examples for Dialogue State Tracking (DST). It outperforms all baselines trained on 100% of the training data using only 5% of the data. Under oracle settings, its Joint Goal Accuracy (JGA) upper bound is 12% higher than traditional methods.
- RAVEN: Robust Advertisement Video Violation Temporal Grounding via Reinforcement Reasoning
-
This paper proposes the RAVEN framework, which integrates curriculum reinforcement learning with multimodal LLMs. Through hierarchical reward mechanisms and progressive training strategies, RAVEN achieves precise temporal grounding and category prediction of advertisement video violations, unlocking emergent reasoning capabilities without requiring explicit reasoning annotation data.
- Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs
-
Based on the observation of attention score sparsity in Video-LLMs, this paper proposes the Sparse-to-Dense (StD) decoding strategy. It uses a top-K sparse attention model as a draft model to rapidly generate candidate tokens, which are then verified in parallel by a dense model, achieving up to a 1.94× lossless acceleration without requiring additional training or architectural modifications.
- A Thousand Words Paint a Picture: Multimodal Goal Tracking for Grounded Social Intelligence
-
This paper proposes a multimodal goal tracking framework that reasons about the implicit goals of participants in social situations by integrating visual and linguistic cues, thereby enhancing the model's understanding of social contexts (i.e., "grounded social intelligence").
- Attention-Seeker: Dynamic Self-Attention Scoring for Unsupervised Key-Frame Extraction
-
This paper proposes Attention-Seeker, an unsupervised method that dynamically analyzes the attention score distribution in self-attention layers of Transformer models to extract the most representative key-frames from videos without any supervision signals. It outperforms existing unsupervised methods on multiple video summarization benchmarks.
🚗 Autonomous Driving (1)¶
- Embracing Large Language Models in Traffic Flow Forecasting
-
The LEAF framework is proposed, which utilizes a dual-branch predictor comprising a graph branch (for pair-wise relations) and a hypergraph branch (for non-pair-wise relations) to generate candidate forecasts. A frozen LLM is then employed as a selector (interpreting discriminatively rather than generatively) to choose the optimal forecast, optimizing the predictor through feedback via ranking loss, achieving SOTA on PEMS datasets.
🤖 Robotics & Embodied AI (7)¶
- CHEER-Ekman: Fine-grained Embodied Emotion Classification
-
This paper proposes the CHEER-Ekman dataset, extending the binary embodied emotion annotations of the CHEER dataset into Ekman's six basic emotions. It employs an LLM-based automatic Best-Worst Scaling (BWS) technique to achieve fine-grained emotion classification without task-specific training, outperforming supervised BERT.
- Do Emotions Really Affect Argument Convincingness? A Dynamic Approach with LLM-based Manipulation Checks
-
This paper proposes a dynamic framework inspired by psychological manipulation checks, utilizing LLMs to modulate the emotional intensity of arguments and systematically investigate the causal impact of emotion on argument convincingness. The findings reveal that in more than half of the cases, human judgments of convincingness are unaffected by emotional changes; when emotion does have an effect, it is more likely to enhance rather than diminish convincingness.
- DRAE: Dynamic Retrieval-Augmented Expert Networks for Lifelong Learning and Task Adaptation in Robotics
-
This paper proposes the DRAE framework, which integrates dynamic MoE routing, parametric RAG (P-RAG), a three-layer cognitive control architecture (ReflexNet-SchemaPlanner-HyperOptima), and DPMM lifelong knowledge retention. It achieves an average success rate of 82.5% on robotic manipulation and autonomous driving tasks, effectively mitigating catastrophic forgetting.
- SELF-PERCEPT: Introspection Improves LLMs' Detection of Multi-Person Mental Manipulation in Conversations
-
This paper proposes the SELF-PERCEPT two-stage prompting framework, drawing on psychological Self-Perception Theory. It guides LLMs to first observe the behavioral cues of conversational participants before inferring their internal attitudes, significantly improving the detection of mental manipulation in multi-person, multi-turn dialogues.
- Task-aware MoILE: Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learning
-
This paper proposes a Hierarchical Embodied Continual Learning (HEC) setting, which divides agent learning into high-level instructions and low-level actions. It designs the Task-aware MoILE method—which automatically identifies tasks through cross-modal clustering, selects LoRA experts using dual routers, and retains past knowledge via SVD orthogonal training. It reduces the forgetting rate to 3.37% across 5 incremental learning scenarios (vs. 7.44% for the previous SOTA).
- Vulnerability of LLMs to Vertically Aligned Text Manipulations
-
This paper systematically reveals the severe vulnerability of LLMs to vertically aligned text inputs: vertically aligning only a small number of keywords can lead to a drop of 25-45 percentage points in text classification accuracy. While CoT reasoning fails to mitigate this issue, a well-designed few-shot learning paradigm can effectively recover performance.
- Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context
-
This work proposes the DICE dataset (2066 sentences, 402 idioms) to reveal systematic flaws in LLMs when contextual understanding is required to disambiguate idioms (literal vs. figurative meanings), achieved through highly controlled contrastive evaluation with identical idiom forms.
🎮 Reinforcement Learning (8)¶
- LLM-Enhanced Self-Evolving Reinforcement Learning for Multi-Step E-Commerce Payment Fraud Risk Detection
-
Formulates e-commerce payment fraud detection as a multi-step MDP and utilizes LLMs (Mixtral/LLaMA/Gemma) to automatically generate and optimize RL reward functions through an evolutionary algorithm, significantly improving dollar-wise precision on real eBay transaction data compared to human-designed rewards and traditional SL baselines.
- Learning to Generate Structured Output with Schema Reinforcement Learning
-
Proposes SchemaBench, a benchmark containing approximately 40,000 JSON schemas, and Schema Reinforcement Learning (SRL), a training framework. By utilizing a fine-grained schema validator to provide dense reward signals combined with a Thoughts of Structure (ToS) reasoning mechanism, SRL improves LLM accuracy in complex JSON generation by up to 16% without compromising general reasoning abilities.
- Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback
-
This paper proposes the Align-SLM framework, which applies preference optimization (DPO + RLAIF) to textless spoken language models (without text injection) for the first time. By utilizing LLMs to automatically evaluate the quality of generated speech continuations to construct preference datasets, combined with curriculum learning, the approach iteratively enhances the semantic understanding of SLMs, setting a new SOTA on benchmarks like ZeroSpeech and StoryCloze.
- An Efficient Task-Oriented Dialogue Policy: Evolutionary Reinforcement Learning Injected by Elite Individuals
-
This paper is the first to apply Evolutionary Reinforcement Learning (ERL) to the task-oriented dialogue policy task. It proposes the EIERL method, which combines the global exploration of Evolutionary Algorithms (EA) with the local optimization of Deep Reinforcement Learning (DRL). It addresses the slow evolution of EA in the large search space of natural language through an Elite Individual Injection (EII) mechanism, achieving a more efficient exploration-exploitation balance across four datasets.
- MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning
-
Proposes MAPoRL—a post-training paradigm based on multi-agent reinforcement learning. By co-training multiple LLMs within a debate framework, integrated with verifier scoring and collaborative incentive mechanisms, it significantly enhances the effectiveness of multi-LLM collaboration and demonstrates cross-task generalization capabilities.
- Prompt-based Personality Profiling: Reinforcement Learning for Relevance Filtering
-
This paper proposes RL-Profiler, which trains a post relevance filter (SelNet) using reinforcement learning to select a small subset of posts relevant to personality traits from a user's large profile. These selected posts are then passed to an LLM for zero-shot personality prediction, thereby significantly reducing context length while maintaining prediction performance close to using all posts.
- TreeRL: LLM Reinforcement Learning with On-Policy Tree Search
-
TreeRL is proposed to directly integrate Entropy-Guided Parallel Tree search (EPTree) into on-policy reinforcement learning training for LLMs. By branching at tokens with high uncertainty, it expands the diversity of reasoning paths and utilizes global and local advantages derived from the tree structure as process supervision signals, surpassing traditional multi-chain sampling RL on mathematics and code reasoning tasks.
- Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient
-
This paper proposes a policy gradient-based structural pruning method for LLMs. By learning Bernoulli pruning masks in the probability space, it directly optimizes the loss function of the pruned model without requiring any backpropagation through the LLM itself, relying solely on forward inference to complete pruning optimization.
🎁 Recommender Systems (7)¶
- LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences
-
Proposed the LOTUS leaderboard, which uniformly evaluates the detailed image captioning capabilities of Large Vision-Language Models across three dimensions: description quality (alignment, descriptiveness, language complexity), side effects (hallucinations, toxicity), and societal bias (gender, skin tone), while supporting customized evaluation based on user preferences.
- KERL: Knowledge-Enhanced Personalized Recipe Recommendation using Large Language Models
-
Proposes KERL, a unified food recommendation system that combines the FoodKG knowledge graph with Multi-LoRA fine-tuning of Phi-3-mini. It accomplishes three functions: personalized recipe recommendation (F1=0.973), recipe generation, and micronutrient estimation, performance-wise significantly outperforming baseline LLMs and traditional embedding methods.
- Beyond Single Labels: Improving Conversational Recommendation through LLM-Powered Data Augmentation
-
To address the false negative problem in conversational recommender systems (where items users might like are incorrectly labeled as negative samples), an LLM-powered data augmentation framework is proposed. It generates synthetic labels through semantic retrieval and relevance scoring, and balances semantic relevance with collaborative information via a two-stage training strategy.
- CoVE: Compressed Vocabulary Expansion Makes Better LLM-based Recommender Systems
-
The CoVE framework is proposed to expand the LLM vocabulary by assigning a unique token ID and embedding to each item, which converts sequential recommendation into a next-token prediction task. Compared to existing methods, CoVE improves recommendation accuracy by up to 62% and achieves an approximate 100x speedup in inference, while addressing memory constraints in large-scale scenarios via hashed embedding compression.
- GRAM: Generative Recommendation via Semantic-aware Multi-granular Late Fusion
-
This work proposes GRAM, a generative recommendation framework. By utilizing semantic-to-lexical translation, it encodes implicit hierarchical taxonomic and collaborative relationships of items into the LLM vocabulary space. Employing multi-granular late fusion, it independently encodes different-grained prompts and fuses them at the decoder side, yielding Recall@5 improvements of 11.5–16.0% and NDCG@5 improvements of 5.3–13.6% across four benchmarks.
- Laser: Bi-Tuning with Collaborative Information for Controllable LLM-Based Sequential Recommendation
-
This paper proposes the Laser framework, which inserts trainable virtual tokens as prefixes and suffixes to a frozen LLM (Bi-Tuning) to inject user-item collaborative information, and designs an MoE-based M-Former to capture diverse characteristics of different users, achieving parameter-efficient sequential recommendation.
- RecLM: Recommendation Instruction Tuning
-
This work proposes RecLM, a model-agnostic recommendation instruction tuning framework. It injects collaborative filtering signals into user/item profiles generated by an LLM via two-round conversational instruction tuning, and refines profile quality using RLHF (PPO). Serving as a plug-and-play component, it consistently improves performance for BiasMF, NCF, LightGCN, SGL, and SimGCL on MIND, Netflix, and industrial datasets, demonstrating significant efficacy particularly in cold-start scenarios.
🔄 Self-Supervised Learning (7)¶
- SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction
-
Proposes SHuBERT (Sign Hidden-Unit BERT), migrating the masked cluster prediction paradigm of the speech self-supervised learning model HuBERT to sign language video. By clustering hand, face, and body pose streams separately and simultaneously predicting the cluster labels of masked frames, the model is pre-trained on approximately 984 hours of ASL video, achieving state-of-the-art (SOTA) on public benchmarks across translation, isolated recognition, and fingerspelling detection tasks.
- AnalyticKWS: Towards Exemplar-Free Analytic Class Incremental Learning for Small-footprint Keyword Spotting
-
AnalyticKWS is proposed, an exemplar-free incremental learning method for keyword spotting. By freezing the feature extractor and analytically updating the classifier via recursive least squares, it outperforms all rehearsal-based methods on the GSC and SC-100 datasets with extremely low training time and memory overhead.
- Contrastive Learning on LLM Back Generation Treebank for Cross-domain Constituency Parsing
-
This paper proposes an LLM Back Generation method that takes incomplete cross-domain constituency trees as input, prompting the LLM to complete the missing words to generate a treebank. It also designs a span-level contrastive learning pre-training strategy to achieve state-of-the-art performance in cross-domain constituency parsing.
- Improving Low-Resource Morphological Inflection via Self-Supervised Objectives
-
This paper systematically explores the effectiveness of 13 self-supervised auxiliary objectives (Autoencoding, CMLM, T5-style, etc.) in extremely low-resource morphological inflection tasks. It finds that autoencoding is optimal when unlabeled data is extremely scarce, whereas character-level MLM is better when data increases. Mask sampling based on morpheme boundaries represents the most promising direction.
- Magnet: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities
-
This paper proposes Magnet, a method that augments decoder-only LLMs simultaneously into text encoders and infilling models using a hybrid attention mechanism (bidirectional + causal) and three self-supervised objectives (masked prediction + contrastive learning + missing span generation). It outperforms specialized methods like LLM2Vec on token-level and sentence-level representation learning tasks while avoiding the severe text repetition issue caused by bidirectionality.
- QAEncoder: Towards Aligned Representation Learning in Question Answering Systems
-
Proposes QAEncoder, a training-free method that estimates the expected embedding of queries corresponding to a document as a proxy for the document representation, combined with a document fingerprint to maintain discriminability. This improves bge-large from 58.5 to 61.8 NDCG@10 on BEIR with zero additional storage or latency overhead.
- WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher Learning
-
Proposes WhiSPA, which aligns the latent space of the Whisper audio encoder with SBERT semantic representations and psychological dimensions (emotion, personality) through contrastive learning, eliminating the dependency on an additional text LM in speech processing and reducing error by 73-84% on psychological evaluation tasks.
📐 Optimization & Theory (3)¶
- ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweighting
-
ScaleBiO proposes a fully first-order bilevel optimization algorithm based on penalty function reformulation, applying bilevel optimization to data source reweighting for 30B+ parameter LLMs for the first time, achieving improvements of +9% on GSM8K and +5.8% on MATH for Qwen-2.5-32B.
- Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race
-
Reveals the "race-blindness" side-effect of alignment training: Alignment prevents LLMs from representing "black/white" as racial concepts in ambiguous contexts, thus failing to activate safety guardrails and causing implicit bias to surge from 64.1% to 91.4%. Counter-intuitively, injecting race-aware activations (rather than unlearning) in early layers reduces implicit bias from 97.3% to 42.4%.
- AmbiK: Dataset of Ambiguous Tasks in Kitchen Environment
-
Proposes AmbiK, a text-only dataset dedicated to detecting ambiguous instructions in kitchen environments. It contains 1,000 pairs of ambiguous/unambiguous instructions categorized by three ambiguity types (user preference, common sense, and safety). Multiple conformal prediction-based ambiguity detection methods are evaluated, revealing that existing methods perform poorly on this benchmark.
🔗 Causal Inference (10)¶
- CausalRAG: Integrating Causal Graphs into Retrieval-Augmented Generation
-
Proposes CausalRAG, which integrates causal graphs into the retrieval process of RAG. It builds a text graph from documents and identifies causal relationships. During querying, it retrieves context through causal path discovery and causal summary generation, significantly improving context precision (92.86%) and retrieval recall in document question answering.
- On the Reliability of Large Language Models for Causal Discovery
-
Using pre-training corpora accessible via open-source LLMs (OLMo, BLOOM), this study empirically validates the "Causal Parrot" hypothesis—that an LLM's ability to identify causal relationships is highly correlated with the frequency of that relationship in the pre-training data (Spearman \(r=0.9\)), and that the presence of erroneous causal relationships and changes in context significantly affect prediction reliability.
- Causal Graph based Event Reasoning using Semantic Relation Experts
-
A causal event graph generation framework involving multi-round collaborative discussion among four types of semantic relation experts (Temporal, Discourse, Conditional, Commonsense) is proposed. Under a zero-shot setting, it achieves competitive results compared to fine-tuned models on multiple downstream tasks, such as event prediction and event forecasting, while providing explainable causal event chains.
- Counterfactual-Consistency Prompting for Relative Temporal Understanding in Large Language Models
-
This paper proposes Counterfactual-Consistency Prompting, a method that addresses the inconsistency in temporal reasoning of large language models (LLMs) by generating counterfactual questions and imposing collective constraints, achieving significant improvements across multiple temporal understanding datasets.
- FitCF: A Framework for Automatic Feature Importance-guided Counterfactual Example Generation
-
This paper proposes the FitCF framework, which leverages BERT-based feature attribution methods (such as LIME/IG/SHAP) to extract important words to guide Large Language Models (LLMs) in generating counterfactual examples under a zero-shot setting (ZeroCF). After filtering through label-flip validation, these examples are utilized as few-shot demonstrations. This approach consistently outperforms three baseline methods (Polyjuice, BAE, FIZLE) on news classification and sentiment analysis tasks.
- IRIS: An Iterative and Integrated Framework for Verifiable Causal Discovery
-
The IRIS framework is proposed. Requiring only a set of initial variable names as input, it automatically retrieves documents, extracts variable values to construct structured data, and builds causal graphs via hybrid causal discovery (GES statistical algorithm + LLM causal verification). Additionally, it iteratively expands the variable set using a missing variable proposal component, relaxing the acyclicity and causal sufficiency assumptions of traditional methods. IRIS comprehensively outperforms 0-shot, CoT, and RAG baselines in F1 score across 6 datasets: Cancer, Diabetes, Obesity, ADNI, and Insurance.
- Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning
-
This paper introduces Variation Theory into the Counterfactual Data Augmentation (CDA) framework, generating counterfactual samples using LLMs while preserving neuro-symbolic patterns, and incorporating a three-stage filtering pipeline to select high-quality data. This approach optimizes few-shot text classification in active learning, achieving significant F1 improvements across multiple datasets.
- Reasoning is All You Need for Video Generalization: A Counterfactual Benchmark with Sub-question Evaluation
-
This paper proposes COVER (COunterfactual VidEo Reasoning), a multi-dimensional video counterfactual reasoning benchmark. It classifies evaluation tasks into four quadrants comprising 13 categories across two dimensions (abstract-concrete and perception-cognition). By decomposing complex questions into sub-questions (necessary conditions), the benchmark reveals that sub-question accuracy is strongly correlated with counterfactual reasoning ability, and enhancing reasoning capacity is the key to improving robustness in video understanding.
- CoA-Reasoning: Explorations on Counterfactual Analysis in Physical Reasoning of LVLMs
-
This paper proposes the CoA-Reasoning framework to systematically evaluate and enhance the causal understanding of Large Vision-Language Models (LVLMs) in physical world reasoning by constructing counterfactual scenarios, revealing significant limitations of existing models in counterfactual physical reasoning.
- Counterfactual Explanations for Aspect-Based Sentiment Analysis
-
This paper proposes a method for generating counterfactual explanations for aspect-based sentiment analysis (ABSA). By finding the minimal text edits that flip the sentiment polarity of a specific aspect, it provides intuitive causal explanations for ABSA model predictions.
🔬 Interpretability (22)¶
- Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safety
-
Proposes the Rational framework, which employs reasoning-enhanced fine-tuning to enable LLMs to perform explicit safety reasoning (analyzing intent, ethics, and potential harms) before responding, rather than relying on rigid refusal heuristics. This significantly improves robustness against reasoning-level adversarial attacks while maintaining helpfulness.
- Bias Attribution in Filipino Language Models: Extending a Bias Interpretability Metric for Application on Agglutinative Languages
-
Extends an information-theoretic bias attribution score metric to agglutinative languages (Filipino) by averaging subword scores to handle complex morphemic structures. Analysis on four multilingual PLMs reveals that bias in Filipino models is driven by entity-type topical words (people/objects/relationships), contrasting sharply with action-type topical words (crime/sexual activity) in English.
- Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference
-
The complete circuit for syllogistic reasoning in language models is discovered using mechanistic interpretability techniques (Activation Patching + Logit Lens + Circuit Ablation). The circuit operates via a three-stage mechanism: long induction bias \(\rightarrow\) middle-term suppression (h11.10) \(\rightarrow\) transitive term movement. This circuit is both sufficient and necessary on symbolic inputs, generalizes to natural language inputs, and exhibits compatible patterns across four architectures: GPT-2, Pythia, LLaMA, and Qwen.
- Around the World in 24 Hours: Probing LLM Knowledge of Time and Place
-
This paper presents the GeoTemp dataset (320k prompts covering 289 cities and 37 time zones) to evaluate the capability of LLMs in joint temporal and spatial reasoning for the first time. The study finds that models can handle time calculation and geographic knowledge independently, but their performance drops sharply when combining both is required.
- CLEME2.0: Towards Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction
-
This paper proposes CLEME2.0, an interpretable reference-based GEC evaluation metric. By disentangling edits into four categories (correct correction TP, wrong correction FPne, under-correction FN, and over-correction FPun) and combining them with edit weighting techniques, it achieves state-of-the-art correlation with human judgments on both GJG15 and SEEDA datasets.
- Cracking Factual Knowledge: A Comprehensive Analysis of Degenerate Knowledge Neurons in Large Language Models
-
This paper redefines degenerate knowledge neurons (DKNs) in LLMs from both structural and functional perspectives, proposes a neural topological clustering (NTC) method to identify DKNs of arbitrary sizes and structures, and reveals the intrinsic relationships of DKNs with LLM robustness, evolvability, and complexity through 34 experiments.
- Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis
-
This paper proposes to locate "shortcut neurons" in contaminated models through comparative analysis and causal analysis, and suppress these neurons via activation patching to achieve more trustworthy LLM evaluation, achieving a Spearman correlation coefficient of over 0.95 with MixEval.
- EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanations
-
This paper proposes EXPERT, a reference-free image captioning evaluation metric based on VLM fine-tuning. By constructing a large-scale structured explanation dataset and designing a two-stage evaluation template, it achieves SOTA performance on multiple benchmark datasets while providing high-quality structured explanations across three dimensions: fluency, relevance, and descriptiveness.
- IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory
-
IRT-Router borrows Item Response Theory (IRT) from psychometrics, treating LLMs as "test-takers" and queries as "exam questions." It learns multi-dimensional ability vectors along with difficulty and discrimination parameters to achieve interpretable multi-LLM routing, achieving over 87% accuracy in OOD scenarios at only 1/30 of the cost of GPT-4o.
- Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs
-
This paper discovers and defines the phenomenon of "contextual entrainment" — where LLMs assign higher probabilities to any tokens that have appeared in the context. Using a differentiable masking method, the study localizes the entrainment heads responsible for this phenomenon and demonstrates that turning off these heads significantly suppresses distraction effects.
Browse all 22 Interpretability papers →
📦 Model Compression (78)¶
- BrainECHO: Semantic Brain Signal Decoding through Vector-Quantized Spectrogram Reconstruction for Whisper-Enhanced Text Generation
-
This paper proposes BrainECHO, a three-stage framework (Autoencoding-Alignment-Finetuning) that maps brain signals to the Mel-spectrogram space via vector-quantized discrete representations, enabling high-quality non-invasive brain-to-text decoding with Whisper.
- EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
-
EfficientQAT proposes a two-stage QAT framework: first performing Block-wise All-Parameter training (Block-AP) to provide a good initialization, and then executing End-to-End Quantization-Parameter fine-tuning (E2E-QP) to capture cross-block interactions. It achieves 2-bit quantization of Llama-2-70B in 41 hours on a single A100 GPU, with only a 3-point accuracy degradation.
- Explaining Puzzle Solutions in Natural Language: An Exploratory Study on 6×6 Sudoku
-
Evaluating the capabilities of five LLMs in solving and explaining \(6\times6\) Sudoku puzzles reveals that even though o1-preview can solve 65% of the puzzles, its generated explanation chains remain severely lacking in faithfulness, clarity, and educational value.
- Disentangling the Roles of Representation and Selection in Data Pruning
-
This paper systematically decomposes data pruning into two independent dimensions: "data representation" and "selection algorithms." Through theoretical analysis and large-scale experiments, it is found that representation quality (especially training gradient) plays a decisive role in pruning performance, while different selection algorithms have their own strengths and weaknesses across different scenarios and often deviate from their design goals.
- Towards the Law of Capacity Gap in Distilling Language Models
-
Unveils the "Law of Capacity Gap" in language model distillation, which states that the size of the optimal teacher model scales linearly with the student model size (approximately 2.5x). This turns the "impossible triangle" in LLM distillation into a solvable problem, leading to the successful distillation of the 3B MiniMA model.
- "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
-
This is the most comprehensive empirical study of LLM quantization to date, conducting over 500k evaluations of FP8/INT8/INT4 on the entire Llama-3.1 family (8B/70B/405B). It finds that FP8 is nearly lossless, INT8 incurs only a 1-3% drop, and INT4 is surprisingly competitive, while providing recommendations for selecting quantization formats in different deployment scenarios.
- Beyond Logits: Aligning Feature Dynamics for Effective Knowledge Distillation
-
This paper proposes a knowledge distillation method that goes beyond logit matching. By aligning the dynamics of feature changes (rather than static feature snapshots) of the teacher and student models during the training process, it achieves more effective knowledge transfer, significantly improving distillation performance on NLP tasks.
- BlockPruner: Fine-grained Pruning for Large Language Models
-
Proposes BlockPruner, which decomposes Transformer layers into two minimal residual blocks (MHA and MLP), evaluates block importance based on perplexity, and performs fine-grained pruning through iterative search, achieving superior compression performance compared to layer-level pruning.
- DoMIX: An Efficient Framework for Exploiting Domain Knowledge in Fine-Tuning
-
This paper proposes DoMIX, which stores domain-specific knowledge in independent LoRA modules and flexibly combines them during fine-tuning using a diagonally initialized bridge matrix. Under continual domain-adaptive pretraining scenarios, it reduces pretraining time by 58% and GPU memory by 87% while outperforming state-of-the-art (SOTA) methods.
- Language Models Resist Alignment: Evidence From Data Compression
-
This paper proposes the concept of LLM "elasticity" from a data compression perspective, proving that the change in compression rate under fine-tuning perturbations is inversely proportional to the dataset size. Because the pre-training data is vastly larger than the alignment data, alignment effects are preferentially "forgotten." This fundamentally explains the fragility of LLM alignment from an information-theoretic standpoint.
Browse all 78 Model Compression papers →
🕸️ Graph Learning (24)¶
- Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs
-
Proving from a graph theory perspective that Multimodal Transformers (MulTs) are essentially Hierarchical Modal-wise Heterogeneous Graphs (HMHGs), this paper proposes the GsiT model. By employing an Interlaced Mask mechanism, GsiT achieves All-Modal-In-One fusion with only 1/3 of the parameters while significantly outperforming traditional MulTs.
- M3HG: Multimodal, Multi-scale, and Multi-type Node Heterogeneous Graph for Emotion Cause Triplet Extraction in Conversations
-
This paper proposes the M3HG model, which explicitly models emotional and causal contexts in conversations by constructing a multimodal multi-type node heterogeneous graph. It fuses semantic information at both intra-utterance and inter-utterance scales to achieve end-to-end extraction of emotion-cause triplets in multimodal conversations. Additionally, it constructs MECAD, the first Chinese multi-scenario MECTEC dataset.
- FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answering
-
Proposes the FiDeLiS framework, which narrows down the search space via a Path-RAG-preselected candidate set, and progressively constructs and validates reasoning paths using Deductive-Verification Beam Search (DVBS). This improves LLM accuracy and interpretability in knowledge graph question answering without requiring training.
- Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs
-
Proposes the CAQA benchmark, which leverages knowledge graphs to automatically generate a large-scale QA attribution evaluation dataset (161K samples) containing four attribution categories (Supporting, Partially Supporting, Contradictory, Irrelevant) and four reasoning complexity levels. Systematically evaluates 25 automatic attribution evaluators, revealing that "partially supporting" identification and complex reasoning scenarios are the core bottlenecks of current evaluators.
- Knowledge Graph Retrieval-Augmented Generation for LLM-based Recommendation (K-RagRec)
-
The K-RagRec framework is proposed, which provides structured and reliable external knowledge for LLM-based recommender systems by retrieving multi-hop subgraphs from knowledge graphs. Combining a popularity-based selective retrieval strategy and a GNN encoder, it effectively mitigates hallucination and knowledge deficit issues in LLM recommendations.
- SimGRAG: Leveraging Similar Subgraphs for Knowledge Graphs Driven Retrieval-Augmented Generation
-
This paper proposes SimGRAG, an approach that employs a two-stage "query → pattern graph → subgraph" alignment strategy. By leveraging LLMs to convert queries into graph patterns and utilizing Graph Semantic Distance (GSD) to efficiently retrieve the most semantically similar subgraphs in KGs, SimGRAG achieves plug-and-play KG-driven RAG, outperforming all existing methods on question-answering and fact verification tasks.
- A Generative Adaptive Replay Continual Learning Model for Temporal Knowledge Graph Reasoning
-
This paper proposes the Deep Generative Adaptive Replay (DGAR) method, which utilizes a pre-trained diffusion model to generate historical entity distribution representations, mitigates distribution conflicts by enhancing shared features between the historical and current distributions, and designs a layer-wise adaptive replay mechanism to integrate historical and current knowledge, significantly alleviating the catastrophic forgetting problem in continual learning scenarios for temporal knowledge graph reasoning.
- A Mutual Information Perspective on Knowledge Graph Embedding
-
This paper proposes a Knowledge Graph Embedding (KGE) framework based on mutual information maximization. It enhances the semantic representation capability of entities and relations by maximizing the mutual information between different components of triples, achieving consistent performance improvements under complex relational patterns (e.g., 1-N, N-1).
- Beyond Completion: A Foundation Model for General Knowledge Graph Reasoning
-
This paper proposes MERRY, a foundation model for knowledge graphs (KGs) that unifiedly handles both in-KG (zero-shot KGC) and out-of-KG (KGQA) reasoning tasks. By fusing textual and structural information via multi-view conditional message passing (CMP), MERRY outperforms existing methods across 28 datasets.
- Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?
-
This paper proposes Morpher, a multimodal prompt learning paradigm. Under extremely weak text supervision (only a few tokens of label names), Morpher aligns a pre-trained GNN into the semantic space of an LLM by simultaneously learning graph prompts and text prompts, enabling cross-task and cross-domain graph classification transfer, as well as the first CLIP-style zero-shot GNN classification prototype.
Browse all 24 Graph Learning papers →
📈 Time Series (7)¶
- Context-Aware Sentiment Forecasting via LLM-based Multi-Perspective Role-Playing Agents
-
Proposed a multi-perspective role-playing framework (MPR) based on LLMs. By using subjective agents to simulate user posting and an objective agent (a fine-tuned "psychologist" LLM) to audit behavioral consistency, it forecasts social media users' future emotional responses to real-time events through iterative rectification, significantly outperforming traditional methods at both macro and micro levels.
- ANRE: Analogical Replay for Temporal Knowledge Graph Forecasting
-
This paper proposes the ANRE (Analogical Replay) method, which retrieves structurally similar "analogical events" from historical knowledge graph snapshots as reasoning clues to assist future event forecasting in temporal knowledge graphs, achieving significant performance improvements across multiple benchmark datasets.
- CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis
-
A CTPD framework is proposed, which utilizes Slot Attention to discover cross-modality shared temporal prototype patterns from multimodal EHR data (irregular time series and clinical notes). Temporal semantics of both modalities are aligned via a TP-NCE contrastive loss, achieving SOTA performance on mortality prediction and phenotype classification tasks on MIMIC-III.
- G2S: A General-to-Specific Learning Framework for Temporal Knowledge Graph Forecasting with Large Language Models
-
This paper proposes the G2S framework, which decouples general patterns (temporal structural regularities) from scenario-specific information (concrete entities/relations) in temporal knowledge graph (TKG) forecasting. By first learning general patterns on anonymized temporal structures and then injecting scenario-specific information, G2S effectively enhances the generalization capability of LLMs in TKG forecasting.
- LETS-C: Leveraging Text Embedding for Time Series Classification
-
LETS-C is proposed: it digitizes time series into text strings, encodes them using a text embedding model, merges them with the original time series via element-wise addition, and feeds them into a lightweight CNN-MLP classification head. With only 14.5% of the trainable parameters, it achieves SOTA, outperforming 27 baselines including OneFitsAll (fine-tuned GPT-2) on 10 UEA multivariate time series datasets.
- Revisiting LLMs as Zero-Shot Time-Series Forecasters: Small Noise Can Break Large Models
-
This paper systematically evaluates the effectiveness of LLMs as zero-shot time-series forecasters and discovers that LLMs are extremely sensitive to input noise—even a small amount of noise can lead to a drastic performance degradation, making them underperform even simple domain-specific models (such as DLinear). The authors suggest that future research should focus on fine-tuning LLMs to better handle numerical sequences.
- Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement
-
Proposes the Time-MQA framework and the TSQA dataset (~200k QA pairs), unifying time series forecasting, imputation, anomaly detection, classification, and open-ended reasoning QA under a natural language question answering paradigm, and endowing LLMs with time series understanding and reasoning capabilities through continual pre-training.
🩺 Medical LLM (31)¶
- AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset
-
This work constructs AfriMed-QA (15,275 questions across 32 specialties in 16 countries), the first large-scale pan-African medical QA benchmark, systematically evaluates 30 LLMs, and reveals significant regional performance gaps and the counter-intuitive phenomenon where domain-specific biomedical models underperform general-purpose models in African healthcare contexts.
- A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment
-
This paper proposes a modular framework to efficiently adapt Small Language Models (SLMs) into clinical domain models. It includes pre-instruction tuning for domain experts (training multiple expert models on medical corpora), model merging (combining multiple experts into a unified MediPhi), and clinical-task alignment based on 2.5 million synthetic instructions (MediFlow). Ultimately, the 3.8B-parameter MediPhi outperforms GPT-4 on several clinical tasks.
- Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine
-
This paper constructs a medical multiple-choice question (MCQ) benchmark centered around a fictitious organ, "Glianorex," to reveal that LLMs predominantly rely on pattern recognition and test-taking heuristics rather than genuine clinical reasoning in medical MCQ tests. Models scored an average of 64% on entirely fictitious medical knowledge, whereas medical doctors scored only 27%.
- RedactX: An LLM-Powered Framework for Automatic Clinical Data De-Identification
-
RedactX is proposed as a fully automated, multimodal clinical data de-identification framework. By combining multi-round LLM extraction, rule-based processing, and retrieval-based relexicalization, it achieves an F1 score (0.9646) comparable to specialized commercial systems on the i2b2 dataset, while optimizing token usage efficiency.
- Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings
-
A systematic benchmark study reveals that LLM performance significantly degrades under high OOV (out-of-vocabulary) and high-novelty medical text summarization scenarios. Through various vocabulary adaptation strategies (MEDVOC, MEDVOC-LLM, ScafFix), it demonstrates that even Llama-3.1 (128K vocabulary size) still suffers from over-fragmentation, and vocabulary adaptation yields remarkable improvements.
- One Size Fits None: Rethinking Fairness in Medical AI
-
This paper conducts a subpopulation performance analysis across three multimodal medical prediction tasks (ICU mortality, graft failure, and emergency triage), exposing performance disparities among groups that are otherwise masked by aggregated metrics. It advocates for tightly coupling fairness with transparency to promote responsible medical AI deployment through routine subpopulation reporting.
- A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directions
-
The first survey to systematically organize and review LLM research in psychotherapy using the APA three-stage (Assessment \(\to\) Diagnosis \(\to\) Treatment) conceptual taxonomy. Covering over 60 works, it comprehensively analyzes four levels from symptom detection to virtual therapists, revealing a four-fold imbalance across disorder coverage, language bias, methodology fragmentation, and theoretical integration.
- LLMs Can Simulate Standardized Patients via Agent Coevolution
-
EvoPatient proposes a multi-agent coevolution framework. Through autonomous simulated dialogues between patient and doctor agents, LLMs learn to simulate standardized patients (SP) without human supervision, surpassing existing reasoning methods by more than 10% in requirement alignment.
- Adaptive-VP: A Framework for LLM-Based Virtual Patients that Adapts to Trainees' Dialogue to Facilitate Nurse Communication Training
-
Proposes the Adaptive-VP framework, which utilizes LLMs to build Virtual Patients (VPs) that dynamically adjust their behavior based on the communication quality of nursing trainees. Through a four-module pipeline of multi-Agent evaluation \(\rightarrow\) dynamic adaptation \(\rightarrow\) dialogue generation \(\rightarrow\) safety monitoring, the framework significantly improves the perceived realism of VP interactions (persona fidelity \(\eta_p^2 = 0.151\), dialogue realism \(\eta_p^2 = 0.254\)) in a between-subjects experiment with 28 nursing experts.
- ANGEL: Learning from Negative Samples in Biomedical Generative Entity Linking
-
The ANGEL framework is proposed, introducing negative sample training to generative Biomedical Entity Linking (BioEL) for the first time. Through a two-stage strategy (positive-only training + negative-aware preference optimization), it significantly improves the model's ability to distinguish between entities with similar surface forms but different semantics, achieving an average top-1 accuracy improvement of 1.7% across five benchmark datasets.
Browse all 31 Medical LLM papers →
🧬 Computational Biology (6)¶
- LADDER: Language Driven Slice Discovery and Error Rectification in Vision Classifiers
-
LADDER "translates" the internal activations of pre-trained vision classifiers into natural language, retrieves error-related sentences, and leverages LLMs to reason out testable hypotheses regarding "which missing attributes cause the model to fail." This enables the discovery and mitigation of multiple biases in any off-the-shelf classifier without requiring any attribute annotations. It consistently outperforms baselines like Domino, Facts, and DFR across 6 natural/medical datasets and over 200 classifiers.
- A Survey on Foundation Language Models for Single-cell Biology
-
This is the first systematic survey of foundation language models for single-cell biology from a language modeling perspective. It categorizes existing works into two major groups: PLMs (pre-trained from scratch) and LLMs (leveraging existing large models). The paper comprehensively analyzes tokenization strategies, pre-training/fine-tuning paradigms, and downstream task systems, while highlighting key challenges in data quality, unified evaluation, and scaling laws.
- Concept Bottleneck Language Models For Protein Design
-
This paper introduces the explainability design principles of Concept Bottleneck Models (CBMs) into protein language models. By utilizing biological concepts in the intermediate layer as a bottleneck, the proposed method achieves a protein generation system that can design functional protein sequences while simultaneously providing human-understandable design rationales.
- Enhancing Safe and Controllable Protein Generation via Knowledge Preference Optimization
-
This paper proposes the KPO framework, which constructs a Protein Safety Knowledge Graph (PSKG) combined with a weighted graph pruning strategy to identify "similar but safe" protein pairs, and fine-tunes protein language models using DPO to steer them away from the hazardous sequence space while maintaining functionality.
- Retrieve to Explain: Evidence-driven Predictions for Explainable Drug Target Identification
-
Proposes R2E (Retrieve to Explain), a retrieval-based framework that scores and ranks candidate answers by retrieving evidence from a literature corpus and faithfully attributes predictions to supporting evidence using Shapley values, outperforming genetics and GPT-4 baselines in drug target identification tasks.
- Align-Pro: Align Protein Representations Through Multi-Modal Learning
-
Align-Pro aligns the representations of three modalities of proteins—sequence, structure, and functional description—into a unified embedding space through a multi-modal contrastive learning framework, thereby enabling cross-modal protein retrieval, classification, and function prediction.
📡 Signal & Communications (1)¶
- WirelessMathBench: A Mathematical Modeling Benchmark for LLMs in Wireless Communications
-
This paper presents WirelessMathBench, a mathematical modeling benchmark for wireless communications featuring 587 problems extracted from 40 cutting-edge papers. It systematically evaluates the capabilities of LLMs in domain-specific mathematical derivations, revealing that even the strongest model, DeepSeek-R1, achieves an average accuracy of only 38.05%, and a mere 7.83% in full formula derivation.
👥 Social Computing (28)¶
- FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steering
-
Proposed FairSteer, an inference-time debiasing framework that detects bias signals in activations using a lightweight linear classifier, and then dynamically adjusts hidden layer activations using a Debiasing Steering Vector (DSV) calculated from contrastive prompt pairs. This effectively mitigates social bias in LLMs across multiple tasks without retraining.
- ImpliHateVid: Implicit Hate Speech Detection in Videos
-
The task of implicit hate speech detection in videos is proposed for the first time. The ImpliHateVid dataset containing 2,009 videos is constructed, and a two-stage contrastive learning framework is designed to integrate text, image, and audio tri-modal features.
- Measuring Social Biases in Masked Language Models by Proxy of Prediction Quality
-
Proposes attention-weighted prediction quality proxy metrics \(\Delta\text{pa}\) and CRRA to evaluate social bias in MLMs under Iterative Masking Experiments (IME), and introduces a model comparison function BSRT to estimate bias introduced by retraining. The proposed methods are found to be more accurate and sensitive than existing methods like CSPS, AUL, and AULA.
- Can Community Notes Replace Professional Fact-Checkers?
-
A large-scale analysis of 664k Twitter/X Community Notes reveals that their reliance on professional fact-checking is 5 times higher than previously reported (\(\ge\)5-7%). Content involving conspiracy theories/false narratives is twice as likely to cite fact-checking sources compared to other content, demonstrating that high-quality community moderation is deeply intertwined with and irreplaceable by professional fact-checking.
- MDiT-Bench: Evaluating the Dual-Implicit Toxicity in Large Multimodal Models
-
Proposes the concept of "dual-implicit toxicity" — bias and discrimination that can only be identified by combining both textual and visual modalities. It constructs the MDIT-Bench benchmark containing 317K questions across 12 categories and 23 subcategories, and reveals a substantial amount of activatable hidden toxicity in mainstream large multimodal models through long-context jailbreaking.
- Evaluation of LLM Vulnerabilities to Being Misused for Personalized Disinformation Generation
-
This study systematically evaluates the capabilities of 6 mainstream LLMs to generate personalized disinformation, finding that most LLMs can generate high-quality personalized fake news. Furthermore, personalization requests actually reduce the trigger rate of safety filters (acting as a form of jailbreak) and slightly decrease the detectability of machine-generated texts.
- A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models
-
This paper proposes a paradigm shift from passive detection to proactive defense, constructing a "three-pillar" framework of knowledge credibility, inference reliability, and input robustness. It systematically maps 127 defense techniques into these three pillars. A meta-analysis of 48 benchmark studies shows that proactive defense improves performance by 42–63% compared to traditional approaches, while identifying non-trivial trade-offs in computational overhead and cross-domain generalization.
- BanStereoSet: A Dataset to Measure Stereotypical Social Biases in LLMs for Bangla
-
This paper introduces BanStereoSet, a Bangla stereotypical bias dataset comprising 1,194 fill-in-the-blank instances covering 9 bias categories (including race, gender, religion, profession, physical appearance, age, caste, and region). It evaluates social biases in multilingual LLMs for Bangla, revealing that GPT-4o exhibits the highest bias while Mistral displays the lowest.
- Behind Closed Words: Creating and Investigating the forePLay Annotated Dataset for Polish Erotic Discourse
-
This work constructs forePLay (24,768 sentences, 5 categories), the first Polish erotic content detection dataset, and proposes a multidimensional annotation framework covering ambiguity, violence, and socially unacceptable behaviors. Evaluation results show that language-specific Polish models significantly outperform multilingual models, with Transformer encoder models demonstrating the strongest performance in handling unbalanced categories.
- Beyond Negative Stereotypes -- Non-Negative Abusive Utterances about Identity Groups and Their Semantic Variants
-
This paper investigates a neglected type of hate speech—abusive expressions that target identity groups without containing explicit negative stereotypes. It systematically analyzes the semantic variants of such "non-negative abusive utterances" and evaluates the processing capabilities of existing detection models.
Browse all 28 Social Computing papers →
📂 Others (184)¶
- In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents
-
This paper proposes the Reflective Memory Management (RMM) mechanism. By combining prospective reflection (multi-granularity memory summarization) and retrospective reflection (reinforcement learning-driven online retrieval optimization), it builds an efficient memory management framework for long-term personalized dialogue systems, achieving an accuracy improvement of over 10% on LongMemEval.
- Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systems
-
This paper proposes a scalable hard negative mining framework for enterprise domain-specific retrieval. By fusing multiple embedding models, PCA dimensionality reduction, and dual semantic conditional filtering, the framework dynamically selects high-quality hard negatives, achieving significant improvements on both internal cloud service datasets and public benchmarks.
- Implicit Reasoning in Transformers is Reasoning through Shortcuts
-
By training GPT-2 from scratch on controlled multi-step mathematical reasoning datasets, this paper systematically investigates the implicit reasoning mechanisms of language models. It reveals that implicit reasoning is fundamentally shortcut learning based on pattern matching—generalizing well on fixed-pattern data but overfitting on unfixed-pattern data, a finding that also holds true for SOTA large language models.
- Zero-Shot Conversational Stance Detection: Dataset and Approaches
-
This work constructs the first zero-shot multi-turn multi-party conversational stance detection dataset, ZS-CSD (280 targets, 17,063 conversational samples), and proposes the SITPCL model. By combining a speaker interaction encoder with target-aware prototypical contrastive learning, SITPCL achieves state-of-the-art performance (F1-macro of 43.81%) in zero-shot conversational stance detection.
- Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction
-
This paper proposes MM-VAP, a multimodal predictive turn-taking model. By incorporating visual cues such as facial expressions, head poses, and gaze direction into a voice predictive model, it improves the hold/shift prediction accuracy from 79% to 84% on a video conferencing corpus.
- Mapping the Podcast Ecosystem with the Structured Podcast Research Corpus
-
This study builds and releases SPoRC, a large-scale dataset containing transcripts of 1.1 million podcast episodes (complete with metadata, inferred speaker roles, and acoustic features for 370k episodes). Through topic analysis, guest co-occurrence network analysis, and responsiveness analysis during the George Floyd protests, it provides the first comprehensive characterization of the content, structure, and responsiveness of the podcast ecosystem.
- Autalic: A Dataset for Anti-Autistic Ableist Language In Context
-
This paper proposes Autalic, the first dataset dedicated to detecting anti-autistic ableist language in context. It contains 2,400 Reddit sentences annotated with context by experts from neurodivergent backgrounds. Experiments reveal that current LLMs (including DeepSeek, Llama3, Gemma2, and Mistral) exhibit severe disagreement with human judgment when identifying anti-autistic ableism (with an average Cohen's Kappa of only 0.091), highlighting the difficulty of this task.
- CoachMe: Decoding Sport Elements with a Reference-Based Coaching Instruction Generation Model
-
CoachMe is proposed to automatically generate sports-specific coaching instruction texts by comparing the differences (across both temporal and physical dimensions) between the learner's motion and a reference motion, outperforming GPT-4o by 31.6% in figure skating and 58.3% in boxing (according to G-Eval).
- Completing A Systematic Review in Hours instead of Months with Interactive AI Agents
-
This paper proposes InsightAgent, a human-centric interactive multi-agent system that reduces the drafting time of medical systematic reviews from months to approximately 1.5 hours through semantic clustering partitioning, multi-agent parallel reading, and real-time user interaction, achieving 79.7% of human drafting quality.
- What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluation
-
This paper systematically studies the automatic evaluation of book-length stories (>100K tokens), constructs the first large-scale long story evaluation benchmark LongStoryEval (600 newly published novels, 340K reader reviews), proposes a hierarchical evaluation criteria framework, compares the effectiveness of three evaluation strategies, and trains a specialized evaluation model NovelCritique-8B, which outperforms GPT-4o in alignment with human ratings.