Skip to content

grammar of the wave towards explainable multivariate time series event detection

content_hash: 1470a7531304a0ea ---# Grammar of the Wave: Towards Explainable Multivariate Time Series Event Detection via Neuro-Symbolic VLM Agents

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/cw-wan/SELA
Area: Multimodal VLM
Keywords: Multivariate Time Series, Event Detection, Neuro-Symbolic System, Vision-Language Agents, Event Logic Tree

TL;DR

Addressing the challenge that multivariate time series event annotations are expensive and governed by complex cross-channel logic, this paper introduces the Event Logic Tree (ELT) to compile textual descriptions into structured temporal logic, alongside SELA, a neuro-symbolic multi-agent framework that uses VLMs for active visual inspection and bottom-up fuzzy logic confidence propagation to achieve high-precision zero-shot event localization with faithful proof-tree explanations.

Background & Motivation

Multivariate Time Series Event Detection (TSED) plays a mission-critical role in high-stakes industrial and scientific domains, including oil and gas drilling, energy grid management, clinical monitoring, and extreme weather hazard warning. Unlike conventional sequence-level classification or anomaly detection based on statistical deviation, events are semantically rich processes governed by intricate cross-channel temporal relationships, such as "a sharp rise on channel A followed by a dip on channel B," or "a steady pressure buildup while volume remains strictly constant." In real-world operational environments, dense timestamp-level event annotations are prohibitively expensive to acquire and heavily dependent on specialized human expertise, creating an acute data bottleneck for fully supervised learning and foundation model fine-tuning.

At the same time, domain-specific natural language event specifications naturally exist within standard operating manuals, regulatory guidelines, and meteorological hazard definitions. However, existing symbolic time series representations (such as SAX, Chronicle, and STL) either rely on rigid discrete string tokens that fail to quantify continuous semantic coherence against linguistic predicates, or operate strictly on binary Boolean values lacking topological elasticity under temporal dilation. Directly serializing numerical sequences into LLM prompts triggers severe hallucinations and arithmetic failures on long multi-channel series. Conversely, recent visual-centric VLM baselines (such as VL-Time) reason over fixed-resolution global waveform plots, losing fine-grained localized resolution and lacking structured symbolic constraints to prevent hallucinated misattribution of transient artifacts.

To bridge the gap between unstructured linguistic guidance and continuous multi-channel signals, the central challenge lies in developing an intermediate representation that simultaneously parses hierarchical temporal logic and grounds into neural visual perception with verifiable explanations. Core idea: explicitly compile unstructured natural language specifications into an Event Logic Tree (ELT) that supports topological elasticity and bottom-up fuzzy confidence propagation, and deploy a decoupled "Logic Analyst" and "Signal Inspector" multi-agent framework to turn event detection into the construction of an instantiated, verifiable proof tree.

Method

Overall Architecture

SELA (Symbolic Event Logic Agent) is a neuro-symbolic multi-agent architecture that leverages the Event Logic Tree (ELT) as the core intermediate representation for language-guided time series event detection. The end-to-end execution flows through three coordinated stages: logical schema compilation, active multi-scale visual inspection, and bottom-up fuzzy proof evaluation. First, the Logic Analyst receives solely the natural language event description and compiles it into a normalized hierarchical ELT schema composed of atomic signal primitives and temporal-logic operators. Next, the Signal Inspector inspects the multivariate time series using active visualization tools (global scanning, local window zooming, and temporal marker placement) to detect candidate intervals and score primitive instances. Finally, the central logic engine evaluates physical collision gates and fuzzy compositional operators bottom-up, yielding the final calibrated event boundaries and an interpretable proof tree.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Multivariate Signals X + Textual Specification L"] --> B["Event Logic Tree Compilation<br/>Parse text into normalized hierarchical ELT Schema"]
    B --> C["Active Signal Visual Inspection<br/>Multi-scale window zoom & primitive candidate grounding"]
    C --> D["Bottom-up Fuzzy Logic Proof<br/>Physical collision gating & T-norm confidence propagation"]
    D --> E["Output: Precise Event Intervals & Explainable Proof Tree"]

Key Designs

1. Event Logic Tree (ELT) Knowledge Representation: Decoupling Schema from Signal Instantiation To overcome the inability of existing symbolic systems to align with linguistic descriptions while preserving temporal elasticity, ELT strictly decouples the abstract event schema from its concrete signal-level instantiation. Schema nodes are categorized into primitive leaf nodes \(n_p = \langle \tau, c \rangle\) denoting an atomic morphology \(\tau\) over a physical channel \(c\) (e.g., "a sharp pressure increase on the Pressure channel"), and composite internal nodes \(n_\phi = \langle [n_1, \dots, n_k], op \rangle\). The operator set \(op \in \{\text{SEQ}, \text{SYNC}, \text{GUARD}, \text{OR}\}\), adapted from Allen's interval algebra, captures temporal precedence, cross-channel synchronization, containment, and semantic disjunction, forming a conditionally complete basis for all 13 fundamental interval relations. To rule out physically invalid structures, ELT enforces three constitutive axioms: the Constructive Composition Axiom (\(|N| \ge 2\), banning trivial self-nesting and pure logical NOT composites), the Temporal Compactness Axiom (bounding unexplained semantic gaps within composite spans), and the Physical Exclusivity Axiom (forbidding any single channel from supporting multiple incompatible active primitive states simultaneously).

2. Role-Decoupled Multi-Agent Coordination: Isolating Logic Analysis from Signal Inspection To prevent schema construction from overfitting or being biased by noise in specific time series instances, SELA enforces strict role decomposition between two dedicated agents communicating via structured function calls and typed artifacts (SchemaArtifact and InstanceArtifact). The Logic Analyst accesses only the natural language event manual and compiles it into a canonical binary ELT schema following the standard template \(\text{SEQ}(\text{Pre}, \text{SEQ}(\text{Main}, \text{Post}))\), establishing background and transitional baselines without observing any signal samples. Conversely, the Signal Inspector operates exclusively on the compiled schema and the multivariate signal visualizer. Equipped with an active inspection toolkit—including full-signal inspection (view_full), region-of-interest zooming (view_window), and boundary marker alignment—the inspector overcomes the resolution limits of static image downsampling, autonomously identifying candidate intervals and submitting primitive aliases, temporal spans, and coherence scores.

3. Bottom-Up Fuzzy Logic Proof and Normalized Confidence: Constraint-Gated Back-Traceability To resolve competitive multi-class evaluation and counter compounding conjunctive penalties, the central logic engine evaluates instantiated trees bottom-up using product T-norm semantics combined with hard validity gates. At the primitive level, semantic coherence \(\mu = \mathcal{M}_\tau(X, I, c) \in [0, 1]\) measures the morphological alignment between the signal segment and linguistic predicate. At composite nodes, SEQ enforces causality gate \(G_{\text{causal}}\) and compactness gate \(G_{\text{coh}}\), SYNC applies an IoU-based alignment penalty \(\exp\left(\frac{\text{IoU}(I_A, I_B) - 1}{\kappa}\right)\), GUARD penalizes boundary overflows outside the container, and all operators incorporate the channel-collision operator \((1 - \Psi)\) to satisfy Physical Exclusivity. When mutually exclusive candidate event classes compete for the same sequence, evaluating raw root confidence unfairly penalizes complex, semantically richer trees with larger leaf sets due to repeated multiplication. SELA introduces geometric normalization based on the number of active primitive leaves \(K(\hat{T}) = |L^*(\hat{T})|\): $$ \mu_{\text{norm}}(\hat{T}) = \mu_{\text{root}}(\hat{T})^{1/K(\hat{T})} $$ This normalized score eliminates depth-induced confidence decay, ensuring fair, calibrated comparisons and fully transparent failure diagnosis.

A Worked Example

The operation of SELA is demonstrated through the detection of a "Frost / Freeze" event from the Climate Hazard dataset: 1. Schema Parsing: The Logic Analyst compiles official NWS criteria into a structured three-phase ELT: Pre captures early-night mild cooling synchronized with light breeze (\(\text{SYNC}(\text{MildAboveFreezing}, \text{LightWind})\)); Main captures sustained radiational nocturnal cooling (\(\text{TEMP\_C} \le 2^\circ\text{C}\), \(\text{NocturnalCoolDescent}\)) guarded by calm winds and wide dew-point spread (\(\text{GUARD}(\text{SYNC}(\text{CalmWind}, \text{DryWideSpread}), \text{NocturnalCoolDescent})\)); Post captures sunrise warming rebound synchronized with wind mixing (\(\text{SYNC}(\text{WarmingRebound}, \text{WindMixingRise})\)). 2. Signal Zooming & Inspection: The Signal Inspector executes view_full to identify a plausible temperature dip across timestamps \([100, 300]\), then calls view_window([120, 290]) to inspect localized channel dynamics and calibrate sub-interval boundaries. 3. Fuzzy Proof Construction: The inspector instantiates Pre at \([130, 138]\) (scores 0.75 and 0.25), Main at \([138, 260]\) (cooling score 0.68; calm wind 0.85, dry spread 0.90), and Post at \([260, 285]\) (rebound 0.90, wind mixing 0.85). The central engine aggregates these nodes into root span \([130, 285]\) with raw confidence \(0.075\) and normalized confidence \(\mu_{\text{norm}} = 0.690\). Competing candidates like Extreme Cold are rejected because their abrupt arctic temperature drop and collapsing dew-point spread fail the guard constraints, yielding an accurate detection with an IoU of 0.92 against the ground truth.

Key Experimental Results

Main Results

The authors release KITE, a benchmark featuring expert-verified event descriptions and boundaries across two domains: Pressure Test (oil & gas drilling, 2 channels) and Climate Hazard (NOAA ISD / NWS storm records, 7 channels). SELA is evaluated against random guessing, supervised baselines (fully fine-tuned CNN and Transformer, alongside frozen time series foundation models Timer, Moment, Chronos, and TimesFM with tuned detection heads), zero-shot/few-shot LLM/VLM baselines (Numeric and VL-Time), and human data scientists.

Method Backbone / Setting Pressure Test (PT)
[email protected]
Pressure Test (PT)
[email protected]
Climate Hazard (CH)
\(\text{MF1}_{\text{Cov}}\)
Climate Hazard (CH)
\(\text{MF0.5}_{\text{Cov}}\)
Random Random Guessing 9.0±4.2 0.0±0.0 12.0±0.6 9.2±0.5
CNN Supervised Fine-tuning 38.6±2.3 22.4±2.5 45.1±3.2 40.7±2.2
Transformer Supervised Fine-tuning 27.7±2.1 17.7±1.1 46.4±6.5 40.8±6.6
TimesFM Foundation Model Tuned 46.6±6.1 3.9±0.6 44.5±3.0 47.4±3.4
Numeric GPT-4.1 (Zero-shot) 15.4±3.9 12.6±2.9 38.3±2.6 46.2±2.4
Numeric GPT-5 (Few-shot) 57.8±6.0 54.2±3.1 43.4±1.6 49.4±1.1
VL-Time GPT-4.1 (Few-shot) 40.1±2.2 24.9±3.3 12.8±1.3 15.2±2.1
VL-Time GPT-5 (Few-shot) 53.6±3.2 46.0±2.6 30.0±3.0 33.2±4.0
SELA (Ours) GPT-4.1 (Zero-shot) 43.9±2.3 32.7±5.5 52.2±5.2 53.9±5.2
SELA (Ours) GPT-5 (Zero-shot) 66.8±3.9 60.0±3.6 50.7±1.5 57.1±2.2
Human Domain Data Scientists 82.3 79.7 69.5 75.9

Ablation Study

The ablation study systematically quantifies the contributions of the ELT hierarchical logic structure (replaced by unstructured geometric mean, arithmetic mean, or minimum pooling), confidence normalization (\(1/K\) root scaling), operator penalty gates, and the active inspection Zoom tool.

Variant Configuration Pressure Test (GPT-5)
[email protected]
Pressure Test (GPT-5)
[email protected]
Pressure Test Avg.
(4 Metrics)
Climate Hazard (GPT-5)
\(\text{MF1}_{\text{Cov}}\)
Climate Hazard (GPT-5)
\(\text{MF0.5}_{\text{Cov}}\)
Climate Hazard Avg.
(4 Metrics)
SELA Full Model 66.8 60.0 50.9 50.7 57.1 53.5
w/o ELT (Geometric Mean) 41.8 (↓25.0) 39.1 (↓20.9) 38.7 (↓12.2) 29.4 (↓21.3) 34.5 (↓22.6) 40.5 (↓13.0)
w/o ELT (Arithmetic Mean) 41.8 (↓25.0) 39.2 (↓20.8) 39.2 (↓11.7) 29.6 (↓21.1) 34.7 (↓22.4) 40.1 (↓13.4)
w/o ELT (Minimum) 34.7 (↓32.1) 32.0 (↓28.0) 33.4 (↓17.5) 26.4 (↓24.3) 31.1 (↓26.0) 39.1 (↓14.4)
w/o \(1/K\) Normalization 66.8 (+0.0) 60.0 (+0.0) 50.4 (↓0.5) 49.4 (↓1.3) 55.9 (↓1.2) 52.6 (↓0.9)
w/o Operator Penalties 67.2 (+0.4) 61.8 (+1.8) 49.9 (↓1.0) 51.0 (+0.3) 58.7 (+1.6) 49.7 (↓3.8)
w/o Zoom Inspection Tool 63.2 (↓3.6) 51.8 (↓8.2) 48.5 (↓2.4) 49.9 (↓0.8) 56.0 (↓1.1) 52.1 (↓1.4)

Key Findings

  • Hierarchical Logic Structure is the Core Driver: Discarding the ELT tree structure in favor of unstructured confidence pooling triggers an catastrophic drop of up to 17.5 points on Pressure Test average score and 14.4 points on Climate Hazard. Without structural precedence, containment, and collision gating, models fail to separate local noise from true multi-channel composite events.
  • Superiority in Precise Boundary Localization: Under the stringent [email protected] metric requiring tight temporal overlap, supervised time series foundation models collapse (TimesFM drops to 3.9, Timer to 3.3), whereas SELA (GPT-5) achieves 60.0 [email protected], outperforming few-shot VL-Time by 14.0 points.
  • Active Visual Zooming is Indispensable for Long Sequences: Disabling the interactive Zoom tool degrades [email protected] on Pressure Test from 60.0 to 51.8 (-8.2 points), proving that downsampled global plots cannot resolve sharp transition boundaries without localized re-inspection.

Highlights & Insights

  • Neuro-Symbolic Bidirectional Synergy: The architecture leverages a purely linguistic reasoning agent to compile unbiased axiomatic structures, coupled with an embodied visual inspector agent grounding continuous evidence, synthesizing continuous neural perception with deterministic logical guarantees.
  • "Grammar of the Wave" Cognitive Formulation: Reframing time series event localization as formal language compilation and proof-tree parsing establishes a verifiable, transparent audit trail essential for mission-critical deployments.
  • Zero-Shot Cross-Domain Transferability: Without gradient fine-tuning on scarce industrial data, SELA effectively transfers across disparate domains (oil drilling and meteorology) using textual manuals alone, surpassing heavily supervised models.

Limitations & Future Work

  • Benchmark Scale and Diversity: The KITE benchmark is constrained to 148 multivariate samples across two domains due to the immense effort required from senior domain experts (e.g., 20+ years drilling engineers) for cross-verification.
  • Subjectivity and Hallucination in Primitive Scoring: Primitive-level alignment currently depends on commercial generalist VLMs; developing native time series foundation models capable of calibrated cross-modal embedding alignment is needed to reduce scoring noise.
  • Computational Overhead and Inference Latency: Running iterative multi-agent ReAct loops with window zoom operations incurs significant token consumption (~233k–295k tokens, ~$0.62 per sample with GPT-5), posing challenges for real-time edge streaming applications.
  • vs Chronicle Systems and Signal Temporal Logic (STL): Chronicle and STL enforce strict binary Boolean satisfaction (True/False) over rigid quantitative thresholds, failing to accommodate sensor noise or continuous linguistic predicates; ELT supports fuzzy confidence propagation and topological elasticity.
  • vs VL-Time and Static Visual TS Reasoners: VL-Time prompts VLMs to guess temporal spans directly from single global plots without localized zooming or relational constraints; SELA introduces interactive multi-scale tools and axiomatic verification gates.
  • vs Time Series Foundation Models (TimesFM, Timer, Chronos): Foundation models pre-trained for point forecasting or reconstruction lack innate multi-channel semantic reasoning and language conditioning, resulting in blurred temporal boundaries under scarce fine-tuning supervision.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First formulation of language-guided TSED, introducing Event Logic Trees and an elegant neuro-symbolic multi-agent architecture.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous real-world curation across oil drilling and NOAA meteorological records, with 4-dimension parse evaluation and detailed ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Flawless conceptual exposition, mathematically rigorous axiomatic formulation, and highly clear system schematics.
  • Value: ⭐⭐⭐⭐⭐ Provides an interpretable, highly accurate paradigm for high-stakes, low-resource industrial and scientific event monitoring.