3DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answering

Conference: ECCV2026
Authors: Changwoo Baek, Kyeongbo Kong
Paper: ECCV official page
PDF: Full paper
Project: 3DZip
Area: VLM Efficiency / 3D Question Answering
Keywords: 3D vision-language models, token compression, feature diversity, determinantal point process, spatially constrained merging

TL;DR

3DZip compresses LLaVA-3D visual tokens before the language model through coarse voxelization, feature-diversity DPP anchor selection, and spatially constrained merging, reducing the reported 1410-token configuration to 128 tokens while retaining a paper-reported 94.7% relative performance and reducing latency from 342 ms to 178 ms per sample on a single RTX 4090.

Background & Motivation

Projection-based 3D vision-language models avoid having to build a strong point-cloud encoder from scratch. They extract visual features from multiple RGB views, use depth and camera poses to place those features in a shared world coordinate system, and add 3D positional embeddings. LLaVA-3D follows this route to answer questions about object attributes, spatial relationships, and scene context. The cost is that many visual tokens from multiple views reach the language model, increasing attention computation and KV cache requirements.

Redundancy occurs at two different scales. Repeated observations of the same physical surface produce tokens at nearly identical locations, which voxel aggregation can consolidate. However, the front, back, and sides of a sofa occupy different locations and can still consume many tokens after voxelization. Spatial coverage therefore does not imply object coverage: large walls, floors, and furniture can dominate the budget while small objects receive no representative. In an analysis of 10 SQA3D scenes using ground-truth object masks, spatial DPP covers 134/259 objects, whereas feature DPP covers 182/259. Coverage means that an object contains at least one selected token, not that the object was correctly recognized.

The authors consequently separate selecting representatives from enriching them with context. Selection should favor feature differences so that similar surfaces do not consume the entire budget; merging should respect geometry so that similar but distant objects are not mixed. Core idea: first aggregate repeated observations spatially, then allocate anchors through feature diversity, and finally merge only spatially compatible context into those anchors.

Method

Overall Architecture

The inputs are multi-view RGB-D observations and camera poses. Image encoding, back-projection, and positional embedding produce tokens that carry both features and 3D coordinates. After compression, these visual tokens are supplied to the language model alongside the question to generate a natural-language answer. Between token construction and language modeling, 3DZip applies coarse voxelization, feature-diversity anchor selection, and spatially constrained merging; the anchor budget \(K\) controls the final visual token count.

Compression does not read question-conditioned LLM attention or use ground-truth object labels to select tokens. Ground-truth masks are used only for the object-coverage analysis. The 1410 tokens in the main table identify the reported full-token configuration, not an intrinsic count for every scene. The raw multi-view token count, occupied voxel count, and final anchor count are distinct quantities.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Multi-view RGB-D<br/>and camera poses"] --> Tokens["Encoding and back-projection<br/>3D features and coordinates"]
    Tokens --> Voxel["Coarse Voxelization"]
    Voxel --> Anchor["Feature-Diversity<br/>Anchor Selection"]
    Anchor --> Merge["Spatially Constrained Merging"]
    Merge --> Answer["Compressed tokens + question<br/>Language model generates answer"]

Key Designs

1. Coarse Voxelization: consolidate repeated observations at the same location

The first stage partitions 3D space into axis-aligned voxels of side length \(\delta\). Tokens in the same voxel are grouped, and their features and coordinates are averaged separately. The output is therefore an aggregate with a mean feature and mean observed position, rather than an artificial token located at the voxel center. The default is \(\delta=0.2\) m. This stage does not enforce the final token budget; its job is to reduce dense multi-view duplication before selection.

Why not apply diversity selection directly to all projected tokens? Repeated observations of a local surface need not have identical features, so feature-only selection can still waste anchors on densely sampled regions. Spatial aggregation first gives each local region a representative before it competes for the scene-wide anchor budget. Removing this stage reduces 64-token SQA3D EM from 52.8 to 51.8. Larger voxels are not automatically better, however: once a small object and its background have been averaged together, DPP cannot recover the lost distinction.

2. Feature-Diversity Anchor Selection: cover different content, not merely different locations

After voxelization, 3DZip L2-normalizes voxel features and constructs a kernel from pairwise cosine similarities. A determinantal point process, or DPP, then selects \(K\) anchors. The key is not an independent importance score for each token, but a set-level assessment of redundancy: similar feature directions add little representational coverage, whereas complementary directions are more valuable. The kernel and selection objective described in the paper can be written as:

\[ L_{ij}=\hat{\mathbf f}_i^\top\hat{\mathbf f}_j, \qquad \mathcal A\approx\underset{|\mathcal S|=K}{\operatorname{argmax}}\det(L_{\mathcal S}). \]

Here, \(L_{\mathcal S}\) is the kernel submatrix for the selected tokens. Its determinant measures the volume associated with the selected normalized feature directions; redundant directions reduce that volume. The paper uses a standard Cholesky-based greedy approximation, not exhaustive subset enumeration or random sampling at inference time. The kernel is built from existing features rather than a separately trained object classifier.

The difference from spatial diversity is particularly important for large objects. Opposite ends of a wall are far apart but may carry similar semantic content, while a small backpack beside a sofa is spatially close yet visually distinct. Feature DPP gives the latter a better chance of representation, but it is not instance segmentation and does not guarantee one token per object. The input features already include 3D positional information, so feature-based selection should not be interpreted as eliminating geometry altogether; it simply does not select using XYZ distance alone.

3. Spatially Constrained Merging: recover context without mixing distant look-alikes

Keeping only anchors would discard details contained in other tokens, so the final stage attempts to merge non-anchors into anchors. For each non-anchor, it first finds the most similar anchor in feature space and then checks whether their Euclidean distance in voxel-grid space is at most \(\tau_g\), whose default is 5. Spatial distance is an admission condition, not the primary criterion for choosing the anchor. Crucially, if the feature-best anchor fails this check, Algorithm 1 discards the token; it does not specify a fallback search for a less similar but closer anchor.

Accepted non-anchor features are averaged together with the anchor feature, while the anchor coordinate remains unchanged. Preserving that coordinate keeps the representative at the selected location instead of allowing coordinate averaging to move it between objects. An anchor with no accepted neighbors retains its own feature. Rejected non-anchors do not become additional outputs, so the final set still contains \(K\) visual tokens.

Following the prose definition of the mean update, the extraction-damaged Equation (18) can be expressed as:

\[ \mathbf f'_a= \frac{\mathbf f_a+\sum_{j\in\mathcal S_a}\mathbf f_j}{1+|\mathcal S_a|}, \qquad \mathbf p'_a=\mathbf p_a. \]

\(\mathcal S_a\) contains only non-anchors whose feature-best anchor is \(a\) and which pass the spatial threshold. This is an average over voxel-level tokens, not a reweighting by how many original observations each voxel contained. The expression above is reconstructed from the textual definition, not copied from the corrupted equation extraction. The operator in Equation (14) and grid-index formatting in Equation (15) are also damaged, so the cache does not justify asserting a particular rounding implementation. Exact reproduction requires checking those indexing details in the PDF or implementation.

A Worked Example

The following is an illustrative scene, not an additional experiment from the paper. A room contains a sofa, a nearby backpack, and another similar sofa farther away; the question asks for the backpack's location. When several views observe the same patch of sofa surface, coarse voxelization first consolidates their nearby observations. Tokens from different sofa surfaces can still remain in different voxels.

Feature-diversity anchor selection does not automatically allocate more anchors to the sofa merely because it occupies more space. Distinct local content, such as the backpack, can compete for representation. Suppose a non-anchor near the backpack finds its feature-best anchor on the backpack and has grid distance 3: it passes threshold 5 and is merged. If another token's feature-best match is on the distant sofa at grid distance 8, it is discarded rather than merged across the room. The values 3 and 8 are illustrative threshold examples only.

The language model receives anchors representing different content, enriched by compatible local context. This example explains why selection and merging use different criteria; it does not guarantee that the backpack is selected or that the compressed model answers correctly.

Loss & Training

The three stages operate on existing 3D tokens. The main paper introduces no dedicated learnable compression module, compression loss, or additional fine-tuning objective. Its statement that compressed inputs should preserve answering behavior is a compression goal, not an implemented distillation loss. Competing methods use the same LLaVA-3D backbone, and generation temperature is set to 0. That fixes the sampling configuration but does not substitute for a complete specification of random seeds and hardware determinism.

Key Experimental Results

Main Results

ScanQA uses validation-set EM, SQA3D uses test-set EM, and OpenEQA uses GPT-4o-based LLM-Match evaluation. These are not interchangeable accuracy measures. The following rows are selected from Table 1 under a shared backbone. Rel. reproduces the paper's reported values: the main text does not clearly define its aggregation formula, so 94.7% should not be read as a task-specific EM, a simple average of the three scores, or a lossless-compression percentage.

Method	Visual tokens	ScanQA EM	SQA3D EM	OpenEQA LLM-Match	Reported Rel.
LLaVA-3D, uncompressed reference	1410	26.5	55.7	60.3	100.0%
FastV	128	21.9	50.9	56.0	88.9%
VisionZip	128	22.2	51.5	56.2	89.8%
VisPruner	128	22.3	52.0	55.8	90.0%
Voxelization only	128	23.6	51.5	54.6	90.7%
DTC	128	22.1	51.3	54.8	88.8%
3DZip	128	24.2	53.2	58.6	94.7%
3DZip	64	23.3	52.8	56.7	92.3%
3DZip	32	21.9	51.1	55.7	88.9%

At 128 tokens, 3DZip improves over the strongest compressed competitor for each task in Table 1 by 0.6, 1.2, and 2.4 points, respectively. Relative to the uncompressed backbone, however, it remains lower by 2.3, 2.5, and 1.7 points. The evidence supports better compression quality, not superiority over the full model.

Ablation Study

The following table combines key rows from Table 3; all values are SQA3D EM. The full configuration uses feature DPP, \(\delta=0.2\) m, and spatially constrained merging. Each intervention should be interpreted within its corresponding subtable, rather than adding the score differences as independent component contributions.

Configuration	64 tokens	32 tokens	Interpretation
Full 3DZip	52.8	51.1	Default configuration
Spatial rather than feature distance for anchors	50.1	49.0	Lower by 2.7 / 2.1 points
Without coarse voxelization	51.8	50.2	Lower by 1.0 / 0.9 points
Voxel side length 0.1 m	52.6	50.9	Lower by 0.2 / 0.2 points
Voxel side length 0.3 m	52.1	50.6	Lower by 0.7 / 0.5 points
Without merging	52.5	50.5	Lower by 0.3 / 0.6 points
Merging without spatial constraints	52.3	50.8	Lower by 0.5 / 0.3 points

Key Findings

• Feature-based selection creates the largest gap among these component comparisons: it exceeds spatial selection by 2.7 EM at 64 tokens. This does not establish that DPP outperforms every alternative diversity algorithm.
• In Table 2, 128-token attribute recognition reaches 64.2 versus 64.0 for the full backbone, but spatial understanding reaches 49.5 versus 54.2. Better semantic coverage does not eliminate losses in geometric reasoning.
• 3DZip does not win every category: at 64 tokens, voxelization alone obtains 48.5 on spatial understanding, compared with 48.1 for 3DZip. The conclusion should concern overall results, not universal category-level dominance.

Table 4 measures efficiency on SQA3D with a single RTX 4090. KV cache size is not total GPU memory usage, and the FLOPs reduction should not be confused with the end-to-end latency reduction.

Method	Tokens	FLOPs (T)	Latency (ms/sample)	KV cache (MB)	SQA3D EM
LLaVA-3D	1410	9.18	342	722	55.7
FastV	128	1.41	191	139	50.9
DTC	128	0.90	196	101	51.3
3DZip	128	0.90	178	101	53.2

Calculated from these values, 342/178 is approximately a 1.92-fold speedup and a 48.0% latency reduction. The visual token count falls by approximately 90.9%, which does not produce a proportional reduction in execution time. Relative to DTC's 196 ms, 178 ms is approximately a 9.2% latency reduction; that percentage should be distinguished from a throughput increase.

Highlights & Insights

• Separating point-level redundancy from object-level duplication gives the method a clear explanation. It shows why feature-space budget allocation remains useful even after voxel aggregation has already removed repeated locations.
• Selection and merging deliberately use different criteria: feature differences determine what survives, while spatial constraints determine what information may be combined. This directly addresses the structure of 3D representations rather than merely ranking attention scores.
• Object-coverage analysis connects token allocation to downstream answering instead of reporting only final scores. However, it covers only 10 scenes and should be treated as mechanistic evidence, not a general guarantee.

Limitations & Future Work

• The authors acknowledge that coarse voxelization can attenuate fine-grained cues for small objects, especially when an object and its background share a voxel. Later diversity selection cannot reconstruct information that averaging has removed.
• Fixed \(\delta\) and \(\tau_g\) may not suit different scene scales, object densities, and distances. Adaptive settings are a future direction explicitly discussed by the authors, not a capability already implemented here.
• Mechanistically, feature diversity is not ground-truth instance coverage, so similar small objects can still be missed. Spatial gating also depends on the quality of depth and camera poses; the main text provides no systematic geometric-noise stress test.
• The cache contains the main paper and references, but not the repeatedly cited supplementary material. Additional backbones, alternative diversity methods, spatial thresholds, small-object analysis, and scene-scale tests are mentioned by the authors, but their detailed results cannot be verified from this cache.
• Tables 1 and 2 disagree on 32-token DTC's OpenEQA average, reporting 52.6 and 52.3, respectively. This note follows Table 1 for its main-results source and does not use the disputed value to derive gains. The undefined Rel. aggregation, damaged equation extraction, and absence of reported error bars also limit precise reproducibility assessment.

Related Work & Insights

• vs LLaVA-3D: LLaVA-3D supplies the backbone that constructs 3D representations from 2D features and answers questions. 3DZip is an upstream compression stage, not a separately trained 3D question-answering model.
• vs FastV / SparseVLM: These methods use LLM attention to retain text-relevant tokens, which can suit narrowly targeted object-state questions. 3DZip instead preserves scene-wide feature diversity and local geometric context without question-conditioned attention.
• vs VisionZip / VisPruner: The paper characterizes these as selection strategies based on vision-encoder CLS attention. 3DZip focuses on representing different content rather than retaining only globally salient regions.
• vs DTC / voxelization only: Spatial grouping can remove duplicated locations but cannot fully suppress repeated representations of large objects across voxels. 3DZip adds scene-wide feature diversity after spatial aggregation and constrains the geometry of subsequent feature merging.

Rating

These are the note author's subjective assessments of the available main paper, not conference review scores.

• Novelty: 4/5. The components are not individually new, but the distinction between two redundancy scales and the separate criteria for selection and merging are well motivated.
• Experimental Thoroughness: 4/5. Three benchmarks, multiple budgets, component ablations, and measured latency provide substantial evidence; unavailable supplementary material and missing error bars limit verification.
• Writing Quality: 4/5. The three-stage argument is clear, but the Rel. metric is insufficiently specified and one baseline score differs across tables.
• Value: 4/5. Large reductions in visual tokens and KV cache with reasonable retained answering quality have clear deployment relevance, subject to scene scale and the quality of the underlying geometry.