CL4D: Contrastive Language–4D Pretraining for Vision-Language Reasoning in Dynamic Scenes

Conference: ECCV 2026
Paper: Official page · PDF
Project: https://4d-vision-uom.github.io/
Area: VLM Reasoning / 4D Scene Understanding
Keywords: dynamic point clouds, language alignment, spatiotemporal Transformer, cross-modal retrieval, geometric question answering

TL;DR

CL4D combines per-frame spatial encoding, cross-frame temporal encoding, and language-contrastive learning before connecting its representations to a frozen 7B language model as 4DVLM, achieving 70.32% batch text-to-motion R@1 on HumanOnly, 16.75 percentage points above the strongest reported baseline, while leaving substantial limitations in global retrieval and open-world reasoning evidence.

Background & Motivation

Answering what happens after a person turns, or which hand is raised at the end, requires both three-dimensional body structure and temporal order. RGB video preserves time but projects depth and occlusion relationships into two dimensions; static point-cloud language models such as PointLLM observe geometry directly but do not represent continuous motion. Here, 4D means a sequence of evolving three-dimensional point clouds, not a timestamp attached to a single static cloud.

Existing P4Transformer, PST-Transformer, and Motion PointNet encoders learn motion features from point-cloud sequences, typically for action recognition over a fixed label set. Such classifiers do not naturally support free-form queries describing several steps and a particular limb movement. Skeleton-language alignment offers a compact alternative, but a skeletal topology does not readily unify humans, manipulated objects, and robotic arms. Raw dynamic point clouds provide a shared input representation, at the cost of unordered points, no inherent pointwise correspondence across frames, and scarce paired language data.

The authors therefore build a dataset, train a language-aligned encoder, and test whether its representations support question answering rather than only closed-set recognition. Core idea: first compress spatial structure and motion evolution into dynamic point-cloud representations that can be compared with text, then expose that geometric evidence to a language model through a lightweight projection.

Method

Overall Architecture

Training has two stages. CL4D first learns a shared semantic space from point-cloud sequences and action descriptions using bidirectional retrieval supervision. 4DVLM then generates answers from pretrained visual tokens and questions while training only the projection between the visual encoder and the language model. DynAction4D supplies motion descriptions for the first stage and action, spatial, and temporal questions for the second. At inference, 4DVLM consumes point clouds without RGB video; this does not mean that its training-data construction is independent of video or other VLMs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Motion parameters and objects"] --> B["DynAction4D Construction"]
    B -->|Point-cloud sequences| C["Factorized Spatiotemporal Encoding"]
    B -->|Action descriptions| D["Bidirectional Language Alignment"]
    C --> D
    D -->|Pretrained visual tokens| E["Frozen-Backbone VQA Adaptation"]
    B -->|Synthetic question-answer pairs| E
    E --> F["Answers grounded in dynamic geometry"]

Key Designs

1. DynAction4D Construction: a shared source for geometry, descriptions, and questions

The pipeline converts SMPL motion parameters into Unity animations, introduces variation in body shape, texture, and placement, and uniformly samples meshes frame by frame to obtain point clouds. HumanOnly derives from HumanML3D, with approximately 23k training and 4k test sequences. ObjInteractions derives from Humoto, with 510 training and 219 test sequences. Cluttered inserts HumanML3D motions into scenes populated with Humoto objects, again with approximately 23k training and 4k test sequences. These segments expose the model to isolated motion, interaction geometry, and background clutter. Because segments share upstream motion sources, adding their sizes does not establish an equivalent number of independent motions.

Retrieval descriptions are inherited from the source motion datasets. For 4D-VQA, Gemini 3.0 Flash receives Unity-rendered videos together with HumanML3D descriptions and generates questions about action semantics, body-spatial qualities, and temporal changes. This connects contrastive training to question-answer adaptation, but the annotations are synthetic. The cached text does not specify the VQA pair count, explicit split sizes, or proportion checked by humans, so none is supplied here. Object counts also conflict within the paper: the dataset section reports 72 objects and 735 interactions, whereas the discussion mentions 73 interacting object categories; this discrepancy remains unresolved.

2. Factorized Spatiotemporal Encoding: resolve within-frame disorder before compressing dynamics

Each frame is divided into 64 local groups of 32 points. A PointNet-style encoder produces 512-dimensional group features, and a learnable CLS token is prepended. A spatial Transformer models relationships among local groups within the frame. Its outputs are summarized through max pooling, mean pooling, and CLS extraction, yielding three complementary frame summaries. Max pooling can preserve salient local responses, mean pooling captures the overall feature distribution, and CLS provides learned aggregation. This is an interpretation of their roles, not a claim supported by separate ablations of all three summaries.

The temporal stage does not explicitly track individual points. It stacks the three summaries across frames. With the described 32-frame configuration, the intermediate tensor is \(H_s\in\mathbb{R}^{3\times512\times32}\). Treating the summaries as three channels and patchifying the last two dimensions into \(16\times16\) blocks produces \(32\times2=64\) patches. Linear projection converts them into 768-dimensional tokens for a temporal Transformer initialized from ViT-B/16. Retrieval uses the mean of the output tokens as the sequence embedding. This avoids joint attention over all points and frames while reusing image-ViT initialization, but temporal locality depends on the summaries and patch layout rather than explicit point trajectories.

The text specifies spatial and temporal depths of 4 and 12, respectively, an MLP ratio of 4, and a spatial encoder trained from scratch. It also states that both Transformers have 12 attention heads, although a 512-dimensional spatial representation cannot be evenly divided into 12 conventional heads. The cache does not explain an internal projection that would resolve this. These are therefore reported values, not a verified executable configuration; a replacement head count or hidden width should not be invented to hide the inconsistency.

3. Bidirectional Language Alignment: match free-form descriptions instead of fixed labels

The text branch uses the 768-dimensional CLS representation from distilbert-base-uncased. Each motion and its paired description form a positive pair, while descriptions from other batch items supply negatives. Training optimizes both text-to-motion and motion-to-text matching rather than only one ranking direction. Both branches are trainable, allowing the language space to adapt to semantics involving limb placement and action order instead of forcing the visual encoder alone to match frozen text features.

Equations (5)–(7) in the cache have damaged parentheses and operators. The following standard directional cross-entropies are normalized transcriptions based on the accompanying verbal definition. The sum-versus-average coefficient for the combined loss cannot be confirmed from damaged Equation (7), so no such coefficient is supplied. Let \(s_{ij}=\cos(v_i,t_j)/\tau\), where \(v_i\) denotes a motion embedding, \(t_j\) a text embedding, \(B\) the batch size, and \(\tau\) the temperature:

\[ \mathcal{L}_{m2t}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(s_{ii})}{\sum_{j=1}^{B}\exp(s_{ij})},\qquad \mathcal{L}_{t2m}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(s_{ii})}{\sum_{j=1}^{B}\exp(s_{ji})}. \]

This objective turns open-vocabulary matching into similarity retrieval. However, different samples in a batch may describe similar motions and need not be genuine semantic negatives; the paper does not report a dedicated false-negative treatment. Its use of zero-shot retrieval emphasizes avoiding fixed classification labels. It does not establish that all test action categories, or RH20T itself, were excluded from every training stage.

4. Frozen-Backbone VQA Adaptation: test whether language models can use geometric features through a small interface

The second stage linearly projects spatiotemporal visual tokens to 4096 dimensions, matching the Vicuna 7B language backbone in LLaVA-v1.5-7b, and combines them with the question using LLaVA-style multimodal conditioning. It is important to distinguish mean-pooled sequence embeddings for retrieval from projected visual tokens for question answering. The global retrieval vector should not be presented as a complete specification of the VQA input.

The CL4D spatiotemporal encoder and the language backbone remain frozen, and only the projection is trained. This avoids changing two large models simultaneously but makes performance depend on the existing visual representation and frozen language model: a projection cannot recover a subtle movement already discarded during visual aggregation. The text does not sufficiently specify every VQA token-selection and special-token insertion detail, so no exact prompt template is invented here.

A Worked Example

Figure 7 shows a person walking, turning, and continuing in the opposite direction; the question asks what happens after the turn. Under the described 32-frame configuration, local point groups become spatial summaries, which become 64 temporal patches. This gives the encoder an opportunity to distinguish stopping after turning from continuing to walk. Pretraining first constrains the action semantics through descriptions, and the VQA stage uses projected tokens together with the question to produce an answer.

In the displayed example, 4DVLM describes a 180-degree turn followed by a return to steady walking, while VideoLLaMA 3 and Gemini 3.0 Flash incorrectly describe stopping or standing still. This is a single qualitative example reported in the paper, not an accuracy estimate for all turning questions. The 32-frame count illustrates the stated configuration rather than a separately disclosed sampling record for this figure.

Loss & Training

CL4D is trained with AdamW for 150 epochs. The main retrieval results use training batch size 72, a learning rate of \(10^{-4}\) for the spatiotemporal components and projection heads, and \(10^{-5}\) for the text encoder. The ablation table instead uses training batch size 64, while evaluation batch size remains 32. Absolute numbers from those two tables therefore do not describe one identical configuration.

4DVLM uses autoregressive answer supervision. The following notation makes the visual input explicit in an equivalent rendering of Equation (8). \(X_v\) is the point-cloud sequence, \(X_q\) the question, and \(y_t\) an answer token:

\[ \mathcal{L}_{\mathrm{VLM}}=-\sum_t\log p\!\left(y_t\mid y_{<t},X_q,\mathrm{Proj}(V_{st}(X_v))\right). \]

Only the projection layer is optimized in this stage, using AdamW with learning rate \(10^{-3}\) for 10 epochs. The cache does not provide the temperature value, weight decay, hardware, or training duration; these should not be imported from unrelated CLIP or LLaVA implementations.

Key Experimental Results

Main Results

R@1 is the fraction of queries whose correct paired item ranks first. Batch retrieval uses 32 randomly sampled candidates, while Global retrieval searches the full test set. All retrieval values below are percentages from Table 1. Baselines were retrained with the same contrastive objective rather than directly evaluated as their original classifiers.

Dataset	Model	Text-to-motion Batch	Text-to-motion Global	Motion-to-text Batch	Motion-to-text Global
HumanOnly	P4Transformer	53.57	3.66	58.05	5.33
HumanOnly	CL4D	70.32	8.07	68.62	8.02
ObjInteractions	Motion PointNet	41.37	12.33	36.18	11.87
ObjInteractions	CL4D	49.40	23.29	46.97	22.37
Cluttered	Motion PointNet	41.71	1.23	43.24	1.53
Cluttered	CL4D	55.07	3.11	51.94	2.71
RH20T	Motion PointNet	68.21	31.09	52.28	24.37
RH20T	CL4D-mini	77.24	37.82	57.10	27.73

The table-derived batch text-to-motion gains for HumanOnly, ObjInteractions, and Cluttered are 16.75, 8.03, and 13.36 percentage points. The paper states 8.1 for the second comparison; this note uses the directly reproducible 8.03. The corresponding RH20T gain is 9.03 percentage points, but that real-world robotics result alone does not establish unadapted cross-domain zero-shot transfer.

The next table selects five VQA metrics from Table 3, retaining their decimal scales rather than presenting them as accuracy. Video models receive mesh-rendered videos of the same scenes; M.V denotes three viewpoints, whereas 4DVLM receives point clouds.

Model	BLEU	ROUGE-L F1	METEOR	BERTScore Precision	BERTScore F1
VideoLLaMA 3	0.0437	0.2951	0.2481	0.4769	0.4317
Gemini 3.0 Flash	0.0447	0.2812	0.2673	0.4240	0.4057
Gemini 3.1 Pro	0.0300	0.2427	0.1933	0.4562	0.3736
Gemini 3.1 Pro (M.V)	0.0320	0.2500	0.2064	0.4482	0.3788
GPT 5	0.0184	0.1612	0.1347	0.4111	0.3191
4DVLM	0.0729	0.3324	0.3152	0.4515	0.4459

4DVLM leads on several generation metrics, but its BERTScore Precision is below VideoLLaMA 3 and Gemini 3.1 Pro. The paper's statement that it wins on every metric should therefore not be repeated unqualified. Model names follow the paper and have not been checked online; these results are specific to its synthetic VQA protocol.

Ablation Study

Table 2 uses HumanOnly with training batch size 64 and evaluation batch size 32. The first seven rows are the ViT-Base reference or architecture variants. The last two degradation settings use CL4D-mini as their reference. GFLOPs are reported as in the source table.

Configuration	Text-to-motion Batch R@1	Motion-to-text Batch R@1	GFLOPs
ViT-Base	68.43	67.96	183.28
ViT-Small	69.61	70.41	177.64
Frozen text encoder	56.63	57.39	183.28
SigLIP loss	66.97	67.14	183.28
Randomly initialized ViT	63.32	62.79	183.28
Shuffled frames	60.97	58.97	183.28
CL4D-mini	67.18	66.02	70.22
mini + partial point clouds	66.81	65.66	70.22
mini + reconstruction noise	65.85	64.64	70.22

Key Findings

• Freezing the text encoder reduces batch text-to-motion R@1 by 11.80 percentage points, and randomly initializing ViT reduces it by 5.11 points. Joint language adaptation matters, and the gains cannot be attributed entirely to a 4D architecture learned from scratch.
• Shuffling frames reduces batch motion-to-text R@1 from 67.96 to 58.97, a drop of 8.99 percentage points. The prose reports 11.44, which does not match the ViT-Base reference in Table 2. This ablation supports sensitivity to temporal order, not proof of causal reasoning.
• CL4D-mini loses only 1.25 percentage points in text-to-motion retrieval relative to ViT-Base while reducing computation from 183.28 to 70.22 GFLOPs. The paper's approximately 60% saving instead uses the ViT-Tiny reference of 176.20 GFLOPs, so the denominators should not be mixed.
• Partial point clouds and reconstruction noise with normalized standard deviation 0.01 reduce mini's text-to-motion score by 0.37 and 1.33 percentage points. These controlled degradations do not constitute comprehensive validation of real LiDAR occlusion, sparsity, and motion distortion.
• Table 4 keeps the projection-layer setup unchanged while replacing the visual encoder: BLEU is 0.0394 for PST-Transformer, 0.0320 for Motion PointNet, and 0.0729 for CL4D, supporting an encoder contribution. Figures 5 and 6 also report generally better retrieval with larger training batches and more frames, but the cache does not support reliable transcription of every plotted point.

Highlights & Insights

• Avoiding a human-specific skeleton gives human motion, interacting objects, and robotic actions a shared point-cloud interface. The benefit is representational scope, not demonstrated understanding of every kind of dynamic environment.
• Organizing cross-frame summaries into a three-channel feature map is a practical way to reuse image-ViT initialization for dynamic geometry. The random-initialization ablation supports its contribution, but the feature map should not be interpreted as an ordinary RGB image.
• Contrastive pretraining followed by frozen-backbone VQA adaptation tests both retrieval quality and whether a language model can use the learned representations. Future work can reuse this evaluation separation to locate bottlenecks in visual evidence versus the language interface.

Limitations & Future Work

• The authors explicitly leave broader generalization for future work. The data primarily concerns synthetic human motion; interaction examples and RH20T retrieval do not establish long-horizon multi-agent understanding or open-world robotic question answering.
• HumanOnly global text-to-motion R@1 is only 8.07%, and Cluttered reaches 3.11%. Strong performance within a small candidate batch does not imply reliable large-scale retrieval; larger candidate pools, multiple semantic positives, and near-duplicate motion analysis are needed.
• Gemini-generated VQA annotations, unequal task adaptation, reference-answer style, and different input modalities complicate comparison with video baselines. Human review, action- and subject-disjoint splits, matched adaptation budgets, and explicit temporal/geometric correctness metrics would be more informative than textual similarity alone when claiming superiority over general VLMs.
• Reproducibility remains incomplete: VQA scale and quality control, detailed RH20T training splits, temperature, and projection-token details are not sufficiently specified in the cache. Some equations are corrupted, and attention-head and dimension statements are questionable. Source-code inspection and checking the original PDF layout should precede reproduction; this note does not fill those unknowns.

Related Work & Insights

• Compared with CLIP, VideoCLIP, and ULIP: these align images, video, or static 3D data with language, while CL4D applies a related objective to evolving point clouds. Its novelty lies in the modality, spatiotemporal encoder, and supporting dataset combination rather than a new bidirectional contrastive loss.
• Compared with P4Transformer, PST-Transformer, and Motion PointNet: the baselines are retrained with the same contrastive objective. The comparison therefore principally supports the architecture and initialization, not an explanation attributing every gain to replacing classification with contrastive learning.
• Compared with Motion Patches: both reuse ViT-style processing for motion, but Motion Patches relies on structured skeletons and CL4D constructs frame summaries from point clouds. Unifying nonhuman entities motivates the latter, while sacrificing the simplicity of explicit skeletal topology.
• Compared with PointLLM and LLaVA: PointLLM addresses static point-cloud language understanding, and LLaVA supplies the visual-to-language conditioning pattern. 4DVLM differs through its language-aligned dynamic geometric encoder. The useful research question is how temporal evidence enters a frozen language model, not whether this is a new language-decoding algorithm.

Rating

• Novelty: 4/5. Dynamic point clouds, language alignment, and VQA adaptation form a valuable research interface, although the loss and language-conditioning mechanism largely reuse established techniques.
• Experimental Thoroughness: 3/5. Retrieval, VQA, encoder replacements, and degradation tests are covered, but VQA disclosure, generalization controls, and statistical uncertainty remain insufficient.
• Writing Quality: 3/5. The main narrative is clear, but numerical gains, claims of leading on every metric, and some configurations are inconsistent; damaged equations are a separate cache-readability issue.
• Value: 4/5. The pipeline is useful for dynamic geometry-language research, but deployment value still needs real acquisition, large-candidate retrieval, and independent VQA validation. These are the note author's assessments, not conference review scores.