DLGStream: Dynamic Language-embedded Gaussian Splatting for Open-vocabulary Enabled Free-viewpoint Video Streaming

Conference: ECCV2026
arXiv: 2606.28840
Code: https://github.com/kkkzh/DLGStream
Area: 3D Vision
Keywords: Three-Dimensional Gaussian Splatting, Free-Viewpoint Video, Open-Vocabulary Queries, Language Feature Embedding, Interpolated Deformation Field

TL;DR

DLGStream introduces a dual-opacity language Gaussian representation and a deformation field based on temporal feature interpolation, embedding streamable spatiotemporal language features while reconstructing high-quality free-viewpoint video from multiview dynamic scenes, using an average frame size of 43 KB to enable real-time 4D open-vocabulary queries and frame interpolation.

Background & Motivation

Free-viewpoint video (FVV) reconstruction based on Three-Dimensional Gaussian Splatting (3DGS) has advanced rapidly in recent years. By extending 3DGS to dynamic scenes and supporting frame-by-frame streaming, these methods have surpassed NeRF-based approaches in both reconstruction quality and frame rate. However, these FVV representations fundamentally contain only geometry and color parameters, preventing users from interacting with scenes semantically—for example, querying an object in natural language, editing a scene, or understanding an environment for embodied AI. This is the greatest user-experience shortcoming of current 3DGS-based FVV methods compared with conventional video streams: a high-fidelity 4D scene is available, yet one cannot simply ask, "Where is the cup?"

Recently, methods such as LangSplat have enabled open-vocabulary queries in static scenes by distilling CLIP language features into 3DGS. Its successor, 4DLangSplat, extends this idea to dynamic scenes and supports time-aware 4D queries. However, the design of 4DLangSplat is entirely unsuitable for FVV applications. It retains LangSplat's strategy of separately modeling three levels of language features, corresponding to SAM's large/medium/small object segmentation masks, and trains an independent language Gaussian field for each level. This requires transmitting three independent dynamic Gaussian models, tripling the frame size; inference requires three rendering passes to obtain language features at all levels, causing FPS to plummet to 15. More seriously, attempts to jointly optimize language features and color significantly degrade color reconstruction quality. Color and language features have different physical properties, creating a fundamental conflict between their optimization directions when they share an opacity parameter. These conflicts make existing language Gaussian representations unusable for free-viewpoint video streaming.

The paper takes a two-pronged approach. Since color and language features require different opacities to characterize their rendering processes, opacity is decoupled so that each can perform alpha blending independently. Meanwhile, the continuity of object motion in real videos is exploited by maintaining temporal features only at keyframes and interpolating them for non-keyframes, substantially reducing frame size while naturally supporting frame interpolation. Core Idea: a dual-opacity language Gaussian representation (maintaining independent opacities for color and language features) and a deformation field based on linear interpolation of temporal features (storing only \(T/n\) key temporal features and interpolating non-keyframe features) embed language features into a streamable 4D representation while preserving high-quality FVV reconstruction, using an average frame size of 43 KB to enable real-time 4D open-vocabulary queries and 4D frame interpolation.

Method

Overall Architecture

DLGStream's overall architecture consists of four stages. First, multiview videos are divided into GOPs (Groups of Pictures, with 60 frames by default), and static–dynamic Gaussian decomposition follows the approach of Swift4D. An auxiliary binary cross-entropy loss learns a dynamic indicator for each Gaussian, separating Gaussians into a static field \(\mathcal{G}^s\) and a dynamic field \(\mathcal{G}^d\); only dynamic Gaussians need to learn deformations. Next, attribute offsets for the dynamic Gaussian field are predicted by an interpolation-based deformation field. Within each GOP, every \(n=10\) frames are represented by a key temporal feature \(k_i\). Temporal features for non-keyframes are obtained by linearly interpolating adjacent keyframes as \(k_t = \text{Lerp}(k_i, k_{i+1}, t)\), then passed to small per-attribute MLP decoders \(\mathcal{D}_a\) to predict offsets for position, covariance, opacity, color, and language features. The static Gaussians and deformed dynamic Gaussians are then fed into the dual-opacity language Gaussian renderer, which outputs a color image and a 9-dimensional multilevel language feature map in a single rendering pass (each set of 3 dimensions corresponds to one SAM level). For subsequent GOPs, binary voxels encode Gaussian attribute residuals relative to the previous GOP. After decoding with a small MLP, processing continues through the deformation field and rendering pipeline.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multiview video"] --> B["GOP grouping<br/>（60 frames per group）"]
    B --> C["Static–dynamic Gaussian decomposition<br/>（Binary cross-entropy indicator）"]
    C --> D["Static Gaussian field Gˢ"]
    C --> E["Dynamic Gaussian field Gᵈ"]
    E --> F["Key temporal features Kᵢ<br/>One every n=10 frames"]
    F --> G["Linear interpolation<br/>→ Per-frame temporal features"]
    G --> H["Per-attribute MLP decoders<br/>Predict attribute offsets"]
    H --> I["Deformed dynamic Gaussians"]
    D --> J["Dual-opacity language Gaussian<br/>Rendering（single pass）"]
    I --> J
    J --> K["Color image + <br/>9-dimensional multilevel language features"]
    K --> L["4D open-vocabulary queries"]
    C --> M["Binary voxels + small MLP"]
    M --> N["Inter-GOP Gaussian residuals"]
    N --> J

Key Designs

1. Dual-opacity language Gaussian representation: independent opacities decouple conflicting rendering requirements for color and language features

Directly adding 9-dimensional language features as extra Gaussian attributes and jointly optimizing color and language significantly degrades color reconstruction quality. The key insight is that position, rotation, and scale define only the scene's geometric distribution, independent of the content being rendered. Opacity, however, directly participates in alpha-blending rendering, and color and language features have sharply different physical tendencies. Color requires lower opacity to represent subtle texture details and changes in highlights, such as transparent regions or reflective edges. Language features are semantic: responses within an object should be consistent rather than semitransparent. A shared opacity parameter is therefore pulled in two directions and cannot simultaneously satisfy both optimization objectives.

DLGStream addresses this by maintaining independent opacities \(o^c \in \mathcal{O}^c\) and \(o^l \in \mathcal{O}^l\) for color and language features, respectively. Each is rendered using its own alpha-blending formula (both retain the original ordering, differing only in their opacity values):

\[R_c(x) = \sum_{m \in N} c_m o_m^c \prod_{j=1}^{m-1} (1-o_j^c), \quad R_l(x) = \sum_{m \in N} f_m o_m^l \prod_{j=1}^{m-1} (1-o_j^l)\]

To prevent spatial misalignment between the color and language rendering surfaces, language feature rendering reuses the depth ordering and transparency accumulation cutoff from color rendering. The gradients of the two opacities also remain independent during back-propagation. Statistical distributions in the experiments confirm the need for this design: color opacity concentrates around 0.3 (lower values are needed to represent texture details), whereas language feature opacity concentrates around 0.5 (semantic consistency demands higher values). The two opacities of the same Gaussian evolve independently over time and differ markedly.

2. Deformation field based on temporal feature interpolation: reducing redundancy while supporting frame interpolation

Existing FVV methods learn an independent temporal feature for every frame, and introducing language features increases frame size even further. However, object motion in real videos is continuous, and temporal features of adjacent frames are highly redundant. Drawing on video interpolation, DLGStream maintains only \(T/n\) learnable key temporal features within a GOP (by default, 60 frames with an interval of 10, giving 6 key features). At a non-key timestamp \(t\), adjacent key features are linearly interpolated to obtain \(k_t = \text{Lerp}(k_i, k_{i+1}, t)\). The interpolated temporal feature is passed to per-attribute MLP decoders to predict the offset of each dynamic Gaussian attribute, \(\Delta a_t^d = \mathcal{D}_a(k_t)\), yielding the final deformed attribute \(a_t^d = a^d + \Delta a_t^d\).

This design has two key advantages. First, the number of stored temporal features is reduced from \(T\) to \(T/n\) (1/10 of the original frame count). The temporal features are organized into a 2D image sequence and further compressed using standard video codecs (H.264 / HEVC), bringing the temporal-feature portion of frame size down to only 20-30 KB. Second, because interpolation models temporal variation continuously, the method naturally supports 4D frame interpolation: it can be trained at a low FPS (such as 10 FPS) and rendered at a high FPS (such as 30 FPS). Experiments show competitive rendering quality even when 9 frames are skipped during training (training at 3 FPS). A temporal regularization loss \(\mathcal{L}_{tsr} = L1(k_i, k_{i+1})\) is also introduced to encourage continuity between adjacent temporal features.

3. GOP-by-GOP residual training and binary voxel compression: eliminating inter-GOP flicker while supporting parallelism

Existing GOP-based FVV methods exhibit noticeable brightness flicker between GOPs. This occurs because the independently trained Gaussian color parameters of each GOP represent its average color. Adjacent GOPs have different averages without temporal constraints, leading to inconsistent brightness when joined. DLGStream initializes from the Gaussian attributes of the previously trained GOP \(g-1\), so the current GOP learns only residuals relative to the previous one: \(\mathcal{G}_g^s = \mathcal{G}_{g-1}^s + \mathcal{R}_g^s\) (static) and \(\mathcal{G}_g^d = \mathcal{G}_{g-1}^d + \mathcal{R}_g^d\) (dynamic). Only the residual values therefore need to be stored and transmitted.

Residual compression draws on BiRF's binarization approach, using multilevel binary voxels (each voxel value constrained to +1 or -1) with a small MLP to encode the spatial distribution of residuals. For a Gaussian at position \(\mathcal{X}\), trilinearly interpolated binary voxels are passed through a Sigmoid to obtain a spatial feature \(f_g\), after which an MLP predicts the residual. An entropy loss \(\mathcal{L}_e\) is introduced during training to minimize the bit count of binary voxels. Inter-GOP endpoint temporal regularization, \(\mathcal{L}_{tsr} = L1(k_{T/n}^g, k_0^{g-1})\), enforces consistency between endpoint temporal features of consecutive GOPs. Temporal features from different GOPs can be encoded into a single video for further compression. This training scheme also supports parallelization: subsequent GOPs can all be trained using initialization from the first GOP, reducing training time by a factor of 2.7-3.8.

Loss & Training

The total loss for the first GOP is \(\mathcal{L} = \alpha \mathcal{L}_{rgb} + (1-\alpha) \mathcal{L}_{ssim} + \beta \mathcal{L}_{tsr} + \psi \mathcal{L}_l\), where \(\mathcal{L}_l = L1(R_l, \text{concat}(I_l^{small}, I_l^{mid}, I_l^{large}))\) is the L1 loss between rendered language features and concatenated CLIP labels from the three levels. Subsequent GOPs additionally require the binary voxel entropy term \(\mathcal{L}_e\). The default configuration uses GOP = 60 and a keyframe interval of \(n=10\), with temporal features encoded using libx265 (crf=6). DLGStream is compatible with both 3DGS and Scaffold-GS as base representations: the former adds offsets directly to Gaussian attributes, whereas the latter adds offsets to anchor attributes before MLP decoding.

Key Experimental Results

Main Results

DLGStream is comprehensively compared with existing methods on three multiview dynamic scene datasets: N3DV, MeetRoom, and WideRange4D.

Task	Metric	4DLangSplat	Ours(3DGS)	Ours(ScaffoldGS)
4D open-vocabulary queries (N3DV)	mIoU ↑	75.5%	85.8%	84.7%
4D open-vocabulary queries (N3DV)	mAcc ↑	90.4%	96.2%	96.0%
Joint color+language optimization (N3DV)	PSNR ↑	30.99	32.15	32.04
Joint color+language optimization (N3DV)	Frame size (KB) ↓	431	97.11	46.33
Joint color+language optimization (N3DV)	FPS ↑	15	83	48

On pure FVV reconstruction (without language features), DLGStream further achieves SOTA performance:

Method	PSNR ↑ (N3DV)	Frame size (KB) ↓	FPS ↑	PSNR ↑ (MeetRoom)
StreamSTGS	32.30	174	100	27.41
Ours(3DGS)	32.26	92.04	107	27.48
GIFStream	30.76	38.04	97	24.37
Ours(ScaffoldGS)	31.93	42.42	88.27	27.28

Ablation Study

Config	mIoU ↑	PSNR ↑	Note
Ours(3DGS)-single opacity	78.99%	31.73	Shared opacity for color and language features
Ours(3DGS)-dual opacity	85.78%	32.15	full model
Ours(ScaffoldGS)-single opacity	79.62%	31.88	Same as above, with ScaffoldGS as the base
Ours(ScaffoldGS)-dual opacity	84.69%	32.04	full model
Nearest-neighbor interpolation deformation field	—	31.35	Storage: 66.48 KB
Bilinear interpolation (default)	—	31.93	Storage: 42.42 KB
Bicubic interpolation deformation field	—	32.20	Storage: 80.24 KB

Key Findings

• Dual opacity is central to the performance gains: removing dual opacity causes mIoU to plummet from 85.8% to 79.0%, while PSNR also drops from 32.15 to 31.73. This simple change brings approximately 7 points of improvement in semantic segmentation accuracy without sacrificing color quality.
• The opacity distributions are clearly distinct: visual analysis shows that color opacity concentrates around 0.3 (low opacity is needed to represent texture details), whereas language feature opacity concentrates around 0.5 (semantic consistency). They also evolve independently over time, indicating that dual opacity learns different physical meanings.
• The temporal-feature frame size is only 20-30 KB: owing to interpolation-based compression and efficient video codecs, temporal features account for only about 1/3 of the total frame size. The remainder mainly consists of static Gaussian attributes, meaning that advances in static Gaussian compression can be seamlessly integrated into the framework to further reduce frame size.
• 4D frame interpolation performs surprisingly well: even when 9 frames are skipped during training (using only 3 FPS training data), reconstruction SSIM drops only from 0.944 to 0.936, indicating that the interpolated deformation field indeed learns the temporal correlations of motion.

Highlights & Insights

• Dual-opacity decoupling is insightful and broadly applicable: superficially, it merely adds an extra learnable opacity parameter, but it reflects a deep understanding of the different physical properties of color and language features. Statistical analysis of opacity distributions—color favoring transparency and semantics favoring opacity—provides direct evidence for this difference, grounding the change in physical considerations rather than guesswork.
• Temporal feature interpolation serves two purposes at once: it reduces frame size by storing fewer frames and naturally supports 4D frame interpolation. This enables recording or transmission at low FPS and high-FPS rendering during decoding, making it well suited to bandwidth-constrained applications.
• Binary voxel residual compression is lightweight: inter-GOP Gaussian residuals require only about 2 KB, thanks to spatial encoding with ±1 binary voxels and decoding with a tiny MLP. The design is simple and storage-efficient.
• Compatibility with both frameworks is validated on the 3DGS and Scaffold-GS base representations, indicating that the three core designs—dual opacity, the interpolated deformation field, and GOP-by-GOP residual training—are general-purpose language-embedded FVV components that can transfer to future 3D representations.

Limitations & Future Work

• Frame interpolation depends entirely on the method's own deformation field, without external priors: current frame interpolation relies solely on the linearity assumption of the interpolated deformation field, and quality still drops noticeably for long interpolation intervals (skipping 9 frames). Optical flow or video diffusion models could provide external priors to improve interpolation across larger gaps.
• Compression of static Gaussian attributes remains a bottleneck: the storage breakdown (Table 7) shows that Gaussian attributes themselves account for most of the frame size (with the 3DGS base, 64 KB / 92 KB), while language and temporal features together account for only ~30 KB. Directly integrating advanced static Gaussian entropy coding methods such as HAC could therefore further reduce overall frame size.
• Dependence on multiview input: the method assumes multiview video input and is not yet applicable to monocular video or sparse-view scenarios.
• Parallel training slightly underperforms sequential training: although parallel mode significantly accelerates training (by a factor of 2.7-3.8), mIoU and mAcc decline slightly, leaving room to improve the accuracy of the parallel strategy.

Related Work & Insights

• vs 4DLangSplat: 4DLangSplat builds a separate model for each of the three language levels, resulting in 3 times the frame size, rendering that is 3 times slower, and degradation under joint optimization. DLGStream encodes all three levels in a unified 9-dimensional feature vector and renders them in a single pass. Combined with dual opacity to eliminate degradation, this reduces frame size by a factor of 10 and increases FPS by a factor of 5.
• vs StreamSTGS: also an FVV method from Tianjin University, StreamSTGS achieves similar PSNR on pure reconstruction (32.30 vs 32.26) but does not support language interaction. DLGStream additionally supports open-vocabulary queries and frame interpolation while halving frame size.
• vs GIFStream: GIFStream has a smaller frame size (~38 KB), but its RD curve shows lower quality than DLGStream at the same bitrate, and it exhibits inter-GOP flicker. DLGStream's residual GOP-by-GOP training fundamentally resolves this issue.
• Dual opacity has broader application potential: any Gaussian representation involving joint rendering of multimodal attributes—color, semantics, normals, depth, and so on—could draw on this idea, maintaining separate opacities for different attributes to avoid optimization conflicts.

Rating

• Novelty: ⭐⭐⭐⭐ Dual-opacity decoupling is simple, effective, and physically insightful; temporal feature interpolation elegantly transfers video interpolation ideas to 4DGS compression, with a well-composed overall combination of innovations.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comparisons on three datasets (including the large-scale 360° scenes of WideRange4D) against 10+ baselines, ablations covering every key design, and an uncommon statistical analysis of opacity distributions.
• Writing Quality: ⭐⭐⭐⭐⭐ The motivation develops progressively, the problem is clearly defined, the method section combines equations and diagrams effectively, and the appendix is rich and detailed.
• Value: ⭐⭐⭐⭐ Fills the gap in language-embedded FVV, offers high practical deployment value (controllable frame size and real-time FPS), and provides broadly useful insights into similar joint optimization problems through the dual-opacity design.

Related Papers

• [AAAI 2026] StreamSTGS: Streaming Spatial and Temporal Gaussian Grids for Real-Time Free-Viewpoint Video
• [NeurIPS 2025] Motion Matters: Compact Gaussian Streaming for Free-Viewpoint Video Reconstruction
• [ECCV 2026] OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environments
• [CVPR 2026] Stabilizing Streaming Video Geometry via Dynamic Feature Normalization
• [ECCV 2026] MeGAS: Thermomechanical Dynamic Gaussian Splatting for Thermophysical Scene Editing