Skip to content

Color When It Counts: Grayscale-Guided Online Triggering for Always-On Streaming Video Sensing

Conference: CVPR 2026
arXiv: 2603.22466
Code: lvgd.github.io/ColorTrigger
Area: Video Understanding
Keywords: Streaming Video Understanding, Edge Devices, Grayscale-Guided Triggering, Energy-Aware, Dynamic Token Routing

TL;DR

Ours proposes a new paradigm of "Grayscale Always-on, RGB On-demand." Through ColorTrigger, color redundancy is detected online using lightweight quadratic programming on the grayscale stream. By using only 8.1% of RGB frames, it maintains 91.6% of the full-color baseline performance, enabling always-on video perception for resource-constrained devices.

Background & Motivation

Always-on perception is a core requirement for next-generation wearable/edge AI systems, but continuous high-fidelity RGB video capture is extremely costly on resource-constrained platforms:

Energy Bottleneck: Devices like smart glasses can only sustain continuous RGB recording for approximately 30-60 minutes, falling far short of the requirements for all-day assistants. Even if inference is offloaded to the cloud, the end-to-end energy bottleneck remains dominated by continuous camera exposure and wireless transmission.

Limitations of Prior Work: Methods such as EgoTrigger use audio cues to trigger the RGB camera, but a failure to trigger results in a complete loss of visual information for extended periods, making critical contexts unrecoverable.

Key Insight โ€” Color is Not Always Necessary: Through pilot experiments on Qwen2.5-VL-7B, the authors found that replacing most frames with grayscale while keeping few RGB frames (maintaining temporal structure) only slightly degrades video understanding performance. This indicates significant color redundancy in natural videosโ€”semantic tasks like action recognition, layout reasoning, and counting mostly do not rely on color, requiring color details only at a few critical moments.

Based on this insight, ColorTrigger proposes: the grayscale stream runs continuously to maintain temporal continuity, while RGB capture is triggered only when necessary, fundamentally reducing perception costs.

Method

Overall Architecture

The goal of ColorTrigger is to enable "Grayscale Always-on, RGB On-demand": the grayscale stream runs continuously to preserve temporal continuity, and expensive RGB capture is triggered only at critical moments. It consists of two components: a causal online trigger that detects whether "this frame brings new information and the budget allows it" within a sliding window of grayscale features, and a dynamic token router that allocates computational power based on frame type. Grayscale frames follow a high-compression path with fewer tokens, while RGB frames follow a high-capacity path with more tokens. These are concatenated temporally and fed into a frozen MLLM decoder. The entire pipeline is training-free, strictly causal, and does not modify any MLLM parameters.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Grayscale Stream (Always-on)<br/>Frozen CLIP ViT CLS Features"] --> B
    subgraph TRIG["Causal Online Trigger (Sliding window W, strictly causal)"]
        direction TB
        B["Sliding Window Affinity Matrix<br/>Quantify redundancy / New info"] --> C["Diversity-driven QP<br/>Current frame novelty score s_t"]
        C --> D["Online Credit Budget Controller<br/>RGB usage hard upper bound b_t"]
    end
    D --> E{"Trigger decision u_t<br/>s_tโ‰ฅฮธ AND b_tโ‰ฅ1"}
    subgraph ROUTE["Dynamic Token Routing"]
        direction TB
        F["Grayscale Frame โ†’ Compressed Path<br/>T_g tokens"]
        G["RGB Frame โ†’ High-capacity Path<br/>T_c tokens (T_c > T_g)"]
    end
    E -->|"u_t=0 No Trigger"| F
    E -->|"u_t=1 Trigger RGB"| G
    F --> H["Temporal Assembly<br/>โ†’ Frozen MLLM decoder"]
    G --> H

Key Designs

1. Sliding Window Grayscale Affinity Matrix: Determining New Information via Similarity

To decide whether to trigger RGB without seeing color, inter-frame redundancy must be quantified. At each time step \(t\), ColorTrigger maintains a causal sliding window \(\mathcal{W}_t\) of size \(W\). Using a frozen CLIP visual encoder, it extracts \(\ell_2\)-normalized features \(\mathbf{f}_i\) from the CLS tokens to construct an affinity matrix \(\tilde{\mathbf{A}}_t = \frac{1}{2}(\mathbf{F}_t \mathbf{F}_t^\top + \mathbf{I}_{n_t}) \in [0,1]^{n_t \times n_t}\). High values of \(\tilde{A}_{ij}\) indicate redundancy (similarity) between frames \(i\) and \(j\), while low values indicate new changes. The matrix is symmetric and positive semi-definite. Using only frames within the current window ensures strict causality, a hard constraint for always-on online scenarios.

2. Diversity-Driven Quadratic Programming: Formulation Frame Selection as Optimization

Similarity scores alone are insufficient; they must be converted into a score for the current frame. ColorTrigger reformulates frame selection as a continuous QP: \(\mathbf{w}_t = \arg\min_{\mathbf{w}} \lambda \mathbf{w}^\top \tilde{\mathbf{A}}_t \mathbf{w}\), subject to \(\mathbf{1}^\top \mathbf{w} = m_t\), assigning weights \(\mathbf{w}_t \in [0,1]^{n_t}\) to frames in the window. The quadratic term \(\mathbf{w}^\top \tilde{\mathbf{A}}_t \mathbf{w}\) penalizes assigning high weights to mutually similar frames, naturally spreading the budget across temporally diverse frames. A higher weight \(s_t = (\mathbf{w}_t)_{n_t}\) for the current frame indicates it contributes new information not covered by recent history, making it a candidate for RGB triggering.

3. Online Credit Budget Controller: Hard Upper Bound on Long-term RGB Usage

Only considering geometric novelty would lead to excessive triggering during rapid scene changes, defeating the purpose of power saving. The controller maintains a scalar credit balance \(b_t \in [0, C]\), accumulating at a target rate \(r\) per frame and consuming one unit per trigger: \(b_{t+1} = \text{clip}(b_t - u_t + r,\; 0, C)\). The final trigger must satisfy both the geometric threshold \(s_t \geq \theta\) and the budget constraint \(b_t \geq 1\): \(u_t = \mathbb{I}[s_t \geq \theta \wedge b_t \geq 1]\). This fixes the long-term RGB usage at \(\sum_{t=1}^T u_t \leq rT + C\), preventing over-budgeting regardless of scene dynamics.

4. Dynamic Token Routing: Allocating Computation Based on Information Density

The trigger decision determines which frames are RGB, and the router then allocates token budgets. Grayscale frames use low-resolution input to produce \(T_g\) tokens, while RGB frames use high-resolution to produce \(T_c > T_g\) tokens, assembled according to the trigger bit: \(\mathbf{Z}_t = (1 - u_t)\psi_g(g_t) \oplus u_t \psi_c(c_t)\). When RGB is sparse, the total cost \(\sum_t [(1-u_t)T_g + u_t T_c]\) is significantly lower than the full-color cost \(T \cdot T_c\), concentrating computation on the few moments with high information density.

Loss & Training

This method is entirely training-free and requires no additional supervision or fine-tuning. All triggering and routing decisions are derived from the geometric relationships in the grayscale stream, allowing direct deployment with frozen MLLMs.

Key Experimental Results

Main Results

Performance on the StreamingBench real-time visual understanding task:

Method #Frames RGB(%) All (Acc) Description
Qwen2.5-VL-7B (Full Color) 1fps 100% 73.68 Full-color Baseline
Human - - 91.46 Human Performance
TimeChat-Online-7B 1fps 100% 75.36 Streaming MLLM SOTA
InternVL-3.5-8B 128 100% - Strong Open-source Model
ColorTrigger 1fps 8.1% 67.49 Only 8.1% RGB frames

ColorTrigger achieves 91.6% of the full-color baseline performance (67.49/73.68) using only 8.1% RGB frames, showing balanced performance across sub-tasks (object perception, causal reasoning, action perception, etc.).

Ablation Study

Configuration RGB(%) All (Acc) Description
Full Grayscale 0% ~60 Grayscale only, significant degradation
Uniform Sample 5% RGB 5% ~63 Randomly inserted low-rate RGB
Uniform Sample 10% RGB 10% ~66 Uniform sampling
ColorTrigger 8.1% 8.1% 67.49 Intelligent trigger outperforms uniform
ColorTrigger + Token Route 8.1% 67.49 Token routing further reduces inference cost
Full Color 100% RGB 100% 73.68 Upper bound

Key Findings

  • There is significant color redundancy in natural videos: 5-10% RGB frames are sufficient to recover most performance.
  • Intelligent triggering is superior to uniform sampling: at the same RGB ratio, ColorTrigger achieves higher accuracy.
  • Dynamic Token Routing further reduces inference cost without sacrificing performance.
  • This paradigm can be directly applied to existing frozen MLLMs without any training.

Highlights & Insights

  • "Color is not always necessary" is a profound observation: It fundamentally challenges the implicit assumption that "more RGB = better performance."
  • The "Grayscale Always-on + RGB On-demand" paradigm is highly significant for real-world edge AI deploymentโ€”extending battery life for smart glasses and security cameras.
  • The QP + Credit Budget design elegantly balances local triggering flexibility with global budget constraints.
  • Fully training-free and plug-and-play, compatible with any frozen MLLM.

Limitations & Future Work

  • Grayscale cameras often have lower resolution and quality than RGB cameras; handling hardware heterogeneity in real-world deployments remains to be verified.
  • The current use of CLIP CLS tokens for affinity analysis might miss local detailed changes.
  • Although QP solving is lightweight, it still incurs computational overhead; real-time performance on ultra-low-power chips requires further evaluation.
  • Validated only on streaming video understanding benchmarks; lacks power consumption and latency evaluations on actual edge devices.
  • Tasks highly dependent on color (e.g., color recognition) may be significantly affected.
  • Similar to event camera priors (activity-driven sampling) but without requiring specialized hardware.
  • Complementary to EgoTrigger (audio-triggered RGB): ColorTrigger maintains grayscale visual continuity, avoiding complete visual information gaps.
  • While Token Pruning/Merging (ToMe, ATP-LLaVA) compresses post-capture, ColorTrigger reduces costs pre-capture.

Rating

  • Novelty: โญโญโญโญโญ The "Grayscale Always-on + RGB On-demand" is a brand new paradigm with profound practical significance.
  • Experimental Thoroughness: โญโญโญโญ Thoroughly validated on StreamingBench, though lacks hardware-level power evaluation.
  • Writing Quality: โญโญโญโญโญ The narrative from pilot study to insight to method is very fluid.
  • Value: โญโญโญโญโญ Direct engineering value for always-on perception in edge AI with strong heuristic potential.