🗣️ Dialogue Systems¶

🧪 ICML2025 · 2 paper notes

Investigating Non-Transitivity in LLM-as-a-Judge: Reveals the non-transitivity problem in LLM-as-a-Judge preferences (where A > B and B > C do not guarantee A > C), demonstrating that fixed-baseline rankings are highly unreliable, and introducing a Round-Robin Bradley-Terry ranking paradigm alongside an efficient Swim tournament strategy.
Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agents: This position paper argues that the traditional dichotomy between aleatoric and epistemic uncertainty fundamentally fails in interactive LLM scenarios by reviewing the conflicting definitions in literature. It proposes three new research directions: underspecification uncertainty (task/context under-specification), interactive learning (reducing uncertainty through follow-up questions), and output uncertainty (expressing uncertainty via natural language rather than scalar values).