Skip to content

Trip+: Benchmarking Agents in Personalized Interactive Travel Planning

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/junle-chen/trip-plus
Area: LLM Reasoning
Keywords: Interactive Planning, Agent Benchmark, Dynamic Replanning, User Simulation, Personalization Alignment

TL;DR

This paper introduces Trip+, a comprehensive benchmark for evaluating LLM agents in personalized, multi-turn travel planning across a 40-city sandbox with 11 traveler profiles and stateful simulation, revealing that state-of-the-art models systematically generate technically feasible yet physically exhausting itineraries with weak profile alignment.

Background & Motivation

As large language model agents transition from sandboxed question-answering towards long-horizon task automation in the physical world, interactive travel planning has emerged as a premier frontier for evaluating multi-step reasoning, external tool invocation, and dynamic constraint satisfaction. Unlike static one-shot itinerary generation, real-world travel planning is inherently an interactive dialogue of trial, error, and negotiation: travelers frequently revise their destination desires, introduce tight budget and schedule constraints, resolve conflicting personal preferences, and react to unpredictable real-world disruptions such as inclement weather, peak traffic, or venue closures. An effective agent must therefore generate an executable schedule, continuously adapt to shifting user states, and faithfully retain previously established commitments.

Despite rapid advancements, existing travel planning benchmarks suffer from an acute dichotomy between personalization richness and interaction depth. Early grounded benchmarks like TravelPlanner, ChinaTravel, and DeepPlanning pioneered deterministic tool-use evaluation in static environments but entirely disregarded individual user personas and multi-turn negotiation. Conversely, profile-aware benchmarks such as TripCraft and Travel-Sim introduced personalized preference scoring, yet they remain restricted to isolated one-shot generation where intent drift across turns cannot be captured. Meanwhile, recent interactive replanning efforts such as TravelBench, TripTide, and Trip-Bench examine isolated dialogue slices (e.g., clarification or feedback incorporation), failing to model cumulative multi-turn constraints, environmental perturbations, and the resulting downstream impact on subjective physical and mental wellbeing.

The underlying tension resides in the tight coupling of deep personalization and multi-turn state dynamics: verifying that an evolving itinerary remains executable while honoring past commitments and minimizing traveler fatigue requires an end-to-end framework capable of tracking turn-level state deltas alongside fine-grained experiential feedback. To bridge this critical evaluation gap, this work formalizes the planning process via a state-first paradigm and an integrated four-layer evaluation protocol. Core idea: build Trip+, a sandbox-grounded multi-turn travel planning benchmark with 11 traveler profiles across 40 cities, evaluating agents across decoupled response gating, deterministic feasibility, stateful intent retention, and an LLM-based profile simulator that replays minute-level itineraries to diagnose the gap between operational feasibility and authentic human comfort.

Method

Overall Architecture

The Trip+ architecture operates over three core components: an evidence sandbox encompassing 40 Chinese cities accessible via 11 specialized travel tools, a diverse spectrum of 11 parameterized traveler profile archetypes, and a state-first multi-turn interaction generation engine. Each task instance is structured into an agent-visible interaction space and a hidden reference ground-truth state. At each turn, the agent evaluates its visible memory and tool evidence to select an appropriate action mode (emitting a full itinerary, asking for clarification, or declaring infeasibility). The evaluation suite then applies a four-layer verification cascade: validating response mode selection, executing deterministic feasibility checks over the fixed sandbox, measuring turn-level request fulfillment and intent preservation, and finally conducting an activity-by-activity simulation using an LLM judge ensemble.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Traveler Profile + 40-City Sandbox Evidence + Multi-Turn Dialogue"] --> B["Response Mode Decision Gate<br/>Plan / Clarification / NoSolution"]
    B -->|Non-Plan Mode| C["Clarification Quality / Infeasibility Proof Audit"]
    B -->|Plan Mode| D["Deterministic Feasibility & Hard Constraint Check<br/>Structural Completeness / Grounded Entities / Temporal & Budget Logic"]
    D --> E["Stateful Request Fulfillment & Intent Preservation<br/>Newly Added Fulfillment Rate / Active Constraint Retention Rate"]
    E --> F["Profile-Conditioned User Experience Simulation<br/>Physical Comfort / Pacing / Environmental Exposure / Budget Stress"]
    F --> G["Multi-Dimensional Diagnostic Planning Report"]

Key Designs

1. Response Mode Decision Gate: Decoupling Planning, Clarification, and Infeasibility In realistic travel planning, attempting to force a concrete itinerary when encountering ambiguous user directives, severe persona-preference contradictions, or provably unsatisfiable physical constraints inevitably triggers severe model hallucination and poor decision quality. Trip+ enforces an explicit response mode gate at the very front of the evaluation pipeline, constraining the agent to choose between three mutually exclusive modes: <plan> for actionable execution, <clarification> for resolving blocking ambiguities, and <no_solution> when tool evidence demonstrates that hard constraints cannot be satisfied simultaneously. A turn is eligible for downstream itinerary evaluation only when the agent's chosen mode strictly matches the ground-truth hidden expectation; any mode mismatch immediately triggers a gate failure, penalizing overconfident guessing and rewarding disciplined clarification.

2. Deterministic Feasibility and Fine-Grained Hard Constraint Verification: Spatial, Temporal, and Budget Closure To prevent agents from relying on ungrounded or geographically impossible plans, Trip+ leverages a verified sandbox covering 40 Chinese cities (comprising 7.2M local transit links, 309K high-speed train connections, 38K flights, and comprehensive POI catalogs) to perform rigorous, deterministic validation. The evaluation inspects structural completeness (ensuring closed-loop itineraries and continuous overnight accommodations), entity validity (confirming that every hotel, restaurant, and attraction is grounded in the database), and operational execution. The execution verifier enforces strict minute-level constraints: checking facility operating hours, enforcing realistic visit durations, applying mandatory intercity transit buffers (e.g., 90 minutes prior to flights, 15 minutes after train arrivals), and ensuring that all cost calculations across multi-passenger parties are mathematically exact.

3. Stateful Request Fulfillment and Cross-Turn Intent Retention: Separating Adaptation from Memory In extended dialogues, LLM agents frequently suffer from recency bias or catastrophic forgetting, silently overwriting earlier party constraints or lodging preferences when asked to modify a downstream activity. Trip+ formalizes turn transitions via hidden states \(s_t = (m_t, H_t, Q_t, E_t)\), maintaining explicit delta updates \(\Delta \cdot, t^+\) and cancellations \(\Delta \cdot, t^-\). The benchmark decouples multi-turn evaluation into two orthogonal metrics: Request Fulfillment (\(F_t\)), which strictly measures whether the agent incorporates newly introduced requirements, and Intent Preservation (\(I_t\)), which verifies whether still-active prior constraints, preferences, and environmental conditions remain satisfied. This dual tracking eliminates heuristic gaming where models inflate performance by merely echoing previous turns or blindly executing local edits.

4. Profile-Conditioned User Simulation: Minute-Level Holistic Experiential Replay Rule-based checks can establish whether a travel plan is mathematically executable, but they cannot assess whether an itinerary is humanly tolerable: a daily schedule packing six intense attractions with tight 10-minute subway transfers under high midday heat may be 100% feasible yet utterly miserable for a family with toddlers or seniors. Trip+ introduces an activity-level experiential replay evaluated by an ensemble of four diverse LLM judges (Qwen, Gemini, GPT, Claude). Role-playing the profiled traveler across 11 archetypes (e.g., backpackers, seniors, budget students, 3-generation families), the simulator evaluates every sequential activity along five key dimensions: physical comfort, schedule pacing, environmental exposure, budget stress, and preference alignment on a 1–5 rubric, using median aggregation to eliminate individual model calibration drift.

Benchmark & Evaluation Protocol

Trip+ comprises 153 multi-turn planning instances and 570 interaction turns, averaging 5.23 active hard constraints per turn. The tasks cover four realistic interaction archetypes: User-State Evolution (4 turns), Request Resolution (3 turns), Environment-Driven Replanning (3 turns), and Long-Horizon Alignment (5 turns). Eighteen agentic LLMs—spanning frontier closed APIs (Gemini, GPT, DeepSeek, GLM, Kimi, Doubao, MiniMax, Hy) and lightweight open-weight models (Gemma, Qwen, GLM served via vLLM)—are evaluated under a uniform iterative function-calling scaffold without framework-level prompt modifications.

Quantitative Experimental Results

Main Benchmark Comparison

The primary benchmark evaluation highlights the substantial performance gap between rule-based constraint satisfaction and subjective experiential alignment across all 18 evaluated models.

Model Family Evaluated Model Resp. Acc. Req. Fulfill Intent Pres. Feasibility Hard Cons. Soft Pref. User Sim. Plan Avg. Win (%)
Frontier Models Gemini-3.1-Pro-Preview 0.8544 0.7382 0.7697 0.8812 0.9047 0.6389 0.5077 0.7331 42.9
Gemini-3-Flash-Preview 0.8667 0.6885 0.6532 0.7099 0.7179 0.6172 0.4862 0.6328 0.0
GPT-5.4 0.7544 0.5585 0.5489 0.7669 0.8110 0.5854 0.5184 0.6704 14.3
GPT-5.4-Mini 0.7807 0.4879 0.4221 0.6455 0.6126 0.6207 0.5085 0.5968 0.0
GLM-5.1 0.8702 0.7214 0.7301 0.8463 0.7973 0.6136 0.5014 0.6896 0.0
Kimi-K2.6 0.8439 0.6727 0.6811 0.7472 0.7984 0.6380 0.5164 0.6750 0.0
Doubao-Seed-2.0-Pro 0.8246 0.6811 0.6814 0.6731 0.8006 0.6031 0.5128 0.6474 0.0
DeepSeek-V4-Pro 0.7912 0.6657 0.7139 0.6755 0.8375 0.6258 0.5100 0.6622 0.0
DeepSeek-V3.2 0.7702 0.5418 0.5566 0.6461 0.7162 0.5842 0.4755 0.6055 0.0
Hy3-preview 0.8930 0.6708 0.7640 0.7223 0.6476 0.5957 0.4866 0.6130 0.0
MiniMax-M2.7 0.7772 0.5766 0.4921 0.5596 0.6326 0.6076 0.4992 0.5748 0.0
Lightweight Models Gemma-4-31B 0.9035 0.7465 0.7490 0.7427 0.7632 0.5731 0.4923 0.6428 28.6
Gemma-4-26B-A4B 0.8386 0.5989 0.5594 0.6761 0.6925 0.5940 0.4916 0.6136 0.0
Qwen3.5-27B 0.8491 0.6309 0.6218 0.7393 0.7256 0.6375 0.4821 0.6461 0.0
Qwen3.5-122B-A10B-FP8 0.3368 0.2033 0.1028 0.6935 0.7235 0.6319 0.5050 0.6385 0.0
Qwen3.6-27B 0.5649 0.4350 0.3756 0.7521 0.7225 0.6626 0.5167 0.6635 14.3
Qwen3.6-35B-A3B 0.6912 0.4554 0.2463 0.4292 0.4588 0.5839 0.5141 0.4965 0.0
GLM-4-32B 0.5053 0.4656 0.3155 0.5236 0.5605 0.6141 0.4791 0.5443 0.0

In-Depth Diagnostic and Failure Mode Breakdown

Examining the strongest performing model, Gemini-3.1-Pro-Preview, across distinct metric groups uncovers the root causes of itinerary degradation in complex travel domains.

Evaluation Category Detailed Metric / Dimension Raw Score (0-1) Failure Rate (%) Primary Failure Mode & Diagnostic Observation
Hard Constraints Date Consistency 1.00 0% Fixed date range parsing and round-trip alignment are robust
Hotel Grounding 0.97 3% Accurate lodging lookup with minor room-count mapping errors
Party Size Compliance 0.95 5% Reliable headcount tracking with rare child-ticket discrepancies
Budget Cap Adherence 0.95 5% Explicit ceiling respected; occasionally fails under multi-segment taxi
Restaurant Requirements 0.84 10% Geographical anchor proximity and opening window alignment issues
Intercity Transport 0.82 24% Connecting transit segment errors and tight station transfer windows
Attraction Satisfaction 0.81 17% Closure-day collisions and invalid operational hour overlaps
Soft Preferences Transport Convenience 0.85 26% Unnecessary multi-leg transfers and excessive within-city transfers
Interest Match 0.68 60% Inadequate density of profile-aligned cultural/nature activities
Budget & Value Ratio 0.65 31% Selection of overpriced dining or upper-quartile lodging
Comfort & Pacing 0.39 99% Systematic breakdown: overpacked schedules, zero rest buffers
User Simulation Preference Satisfaction 0.60 40% Sub-optimal overall alignment with nuanced personal tastes
Schedule Pacing Comfort 0.52 48% Rushed transitions, dawn departures, and late-night check-ins
Budget Stress 0.50 75% Pushing expenses close to maximum caps causing financial anxiety
Physical Comfort 0.49 97% Severe exhaustion: daily walking and transit limits exceeded
Environmental Comfort 0.43 95% Excessive heat/cold exposure during peak hours without indoor backup

Key Findings

  • The Feasibility–Suitability Chasm: A high feasibility score does not ensure a tolerable vacation. Even Gemini-3.1-Pro-Preview, which achieves a 0.9047 hard-constraint satisfaction rate, scores only 0.5077 on simulated traveler experience. Strikingly, 97% of its valid itineraries contain excessive traveler fatigue, and 99% trigger soft-preference pacing failures, proving that models optimize for technical correctness at the expense of human vitality.
  • Vulnerability to Long-Horizon Intent Decay: In the Long-Horizon Alignment scenario, model reliability deteriorates sharply as constraints accumulate. Upon reaching an ambiguous query at Turn 3, Gemini-3.1-Pro's Response Accuracy drops to 0.78 and Request Fulfillment plummets to 0.41. By Turn 4, Intent Preservation declines to 0.71, demonstrating that agents rapidly forget early commitments when overwhelmed by multi-turn updates.
  • Diminishing Returns of Inference Compute: Increasing inference effort fails to yield monotonic improvements in itinerary quality. While DeepSeek-V4 averages 48.7 LLM calls and 128.5 tool calls per task, its User Simulation score stagnates at 0.5100. In contrast, Gemini-3.1-Pro and GPT-5.4 attain superior planning quality with a fraction of the computational footprint, indicating that disciplined evidence filtering is vastly more impactful than raw trial-and-error generation.

Highlights & Insights

  • State-First Formalization of Agent Interaction: Trip+ establishes an elegant framework by formalizing interactive travel as an explicit hidden state tracking task, cleanly separating new request fulfillment from prior intent preservation to eliminate evaluation confounding.
  • Hybrid Grounded-Simulation Verification: The integration of deterministic sandbox verification (ensuring temporal and physical ground truth) with role-playing LLM multi-judge simulation (capturing subjective human fatigue and environmental exposure) provides a robust blueprint for evaluating real-world planning agents.
  • Actionable Lessons for Embodied and Personal Assistants: The finding that agents routinely generate "feasible but exhausting" schedules offers a vital insight for general AI assistants: real-world planning cannot treat human stamina as an infinite resource; physical limits and buffer margins must be modeled as primary optimization objectives.

Limitations & Future Work

  • Synthetic Dialogue Regularity: The interaction turns are synthesized through controlled state-first templates, lacking the linguistic noise, conversational disfluencies, typos, and mixed intents common in organic human user dialogues.
  • Geographic and Cultural Scope: The current evidence sandbox is restricted to 40 Chinese cities and conducted in English, leaving cross-border multi-currency logistics, international visa constraints, and regional transit nuances for future benchmark expansions.
  • Simulation Calibration Against Human Physiological Data: Although the 4-judge ensemble exhibits strong internal reliability (Cronbach's \(\alpha = 0.833\)) and 94.0% consistency with human audits, physical fatigue scores are simulated rather than calibrated against wearable biometric or post-trip traveler satisfaction data.
  • vs TravelPlanner (Xie et al., 2024) & ChinaTravel (Shao et al., 2026): While these benchmarks established sandbox tool-use for itinerary generation, they are static single-turn evaluations that ignore personalized user profiles; Trip+ extends planning to multi-turn dynamic interaction across 11 detailed traveler personas.
  • vs TripCraft (Chaudhuri et al., 2025) & Travel-Sim (Yang et al., 2025): Although these studies incorporated profile awareness, they evaluated one-shot tasks without multi-turn state transitions or intent preservation tracking; Trip+ formalizes state deltas and tracks cumulative commitments over long horizons.
  • vs TravelBench (Cheng et al., 2026) & TripTide (Karmakar et al., 2026): Existing replanning benchmarks evaluate isolated interaction turns (such as disruption recovery or clarification) without minute-level physical grounding; Trip+ provides an end-to-end framework coupling verifiable sandbox execution with activity-level experiential replay.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering integration of multi-turn state tracking, physical sandbox grounding, and profile-conditioned experiential simulation.
  • Experimental Rigor: ⭐⭐⭐⭐⭐ Exhaustive evaluation of 18 frontier and open models across 570 turns, supplemented by bootstrap confidence intervals and human rationale audits.
  • Writing Quality: ⭐⭐⭐⭐⭐ Highly structured, articulate narrative with rigorous mathematical formulation and clean empirical visualizations.
  • Significance: ⭐⭐⭐⭐⭐ Exposes a foundational blind spot in modern LLM planning agents, providing indispensable methodology and metrics for human-centric agentic systems.