M2Tok: Multi-head Multi-codebook Discrete Action Tokenization for Vision-Language-Action Models

Conference: ECCV 2026
Paper: ECCV Official
PDF: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/7815.pdf
Code: https://github.com/cpaaax/M2Tok
Area: Robotics & Embodied AI
Keywords: vision-language-action model, action tokenizer, multi-head subspace decomposition, multi-codebook quantization, bimanual manipulation

TL;DR

Addressing the discretization bottleneck caused by semantic entanglement and limited expressivity in existing VLA action tokenizers, M2Tok introduces orthogonal latent subspace decomposition and combinatorial multi-codebook quantization, reducing the action reconstruction L1 error to 0.0024 and boosting bimanual manipulation success rate to 51% on RoboTwin.

Background & Motivation

Autoregressive language models have emerged as an effective unifying framework for multimodal policy learning, expanding rapidly into robotic control. To project continuous high-frequency physical actions into the discrete token spaces native to Large Language Models (LLMs), Vision-Language-Action (VLA) architectures depend fundamentally on action tokenizers. Early paradigms like RT-1 and OpenVLA performed naive per-dimension uniform binning independently at each control step. This scheme ignores temporal correlations across execution horizons and triggers severe jitter in high-frequency actuators. Subsequent advances shifted toward action chunking to compress future trajectory segments into unified discrete codes, branching into two dominant lines: frequency-domain Discrete Cosine Transform combined with Byte-Pair Encoding (DCT + BPE, e.g., FAST), and Vector Quantized Variational Autoencoders based on Residual Vector Quantization (RVQ-based VQ-VAE, e.g., VQ-BET and VQ-VLA).

However, both paradigms suffer from an acute discretization bottleneck when modeling continuous robotic dynamics. Applying text-derived BPE algorithms to continuous DCT spectrum coefficients enforces an unnatural topological discretization, resulting in substantial reconstruction distortion and variable-length token sequences that break efficient parallel decoding in standard LLM backbones. Conversely, RVQ-based VQ-VAE frameworks collapse heterogeneous signals—such as continuous 6-DoF end-effector poses and binary gripper actuation—into a shared monolithic code space, inducing severe semantic entanglement across distinct physical dimensions. Furthermore, their representational capacity scales only linearly with the codebook vocabulary size \(|V|\), lacking the combinatorial density required to cover the continuous, high-dimensional dual-arm manipulation space without losing fine-grained motor primitives.

This structural entanglement and capacity constraint severely cap the performance ceiling of downstream autoregressive policies. Core idea: decompose the continuous action latent space into multiple orthogonal subspaces (Multi-head) and assign an independent codebook to each head for combinatorial quantization (Multi-codebook), expanding representational capacity exponentially at constant parameter scale, and bridge the discrete tokens back to the VLA via a lightweight action conversion module.

Method

Overall Architecture

The M2Tok framework consists of two core components: the discrete action tokenizer and the downstream M2Tok-based autoregressive VLA policy. During the tokenizer phase, bimanual action trajectories are first processed through Bimanual Factorization, splitting 14-DoF dual-arm chunk sequences into symmetric single-arm 7-DoF primitives. A temporal-aware hybrid encoder—combining causal 1D convolutional layers with Transformer blocks—downsamples the action chunk into continuous latent sequence representations. The latent feature space is then partitioned across channel dimensions into \(h\) orthogonal heads (subspaces), where each head queries an independent codebook to retrieve nearest-neighbor discrete indices. A matched decoder progressively upsamples the concatenated prototypes to reconstruct the continuous control trajectory. For the downstream VLA policy, the pre-trained tokenizer and SigLIP vision backbone are frozen; text tokens, binned proprioceptive state tokens, visual patch tokens, and discrete action tokens are serialized into a unified autoregressive stream, with a dedicated action-token conversion module mapping discrete action indices directly back into continuous prototype embeddings before feeding into the LLM.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input Action Chunk<br/>14-DoF continuous dual-arm trajectory"] --> Split["Bimanual Factorization<br/>Split into 7-DoF single-arm sequences"]
    Split --> Enc["Temporal-Aware Hybrid Encoder<br/>Causal 1D-CNN + Transformer layers"]
    Enc --> Head["Orthogonal Subspace Decomposition<br/>Partition into h independent heads"]
    Head --> Quant["Combinatorial Quantization<br/>Independent nearest-neighbor codebook lookup"]
    Quant --> Trans["Conversion Module & VLA Decoding<br/>Codebook prototype projection + autoregressive rollout"]
    Trans --> Dec["Fidelity-Preserving Decoder<br/>Progressive upsampling to continuous actions"]

Key Designs

1. Bimanual Factorization: enforcing kinematic symmetry and data efficiency

Jointly modeling 14-DoF dual-arm control trajectories directly within a single monolithic space squares the dynamics search complexity and forces the encoder to fit inter-arm coordination patterns simultaneously. M2Tok splits each bimanual action chunk \(a_{1:T} \in \mathbb{R}^{T \times 14}\) into two symmetric single-arm sequences \(a_{\text{left}}, a_{\text{right}} \in \mathbb{R}^{T \times 7}\), populating a shared single-arm trajectory dataset \(\mathcal{A}'\) as the actual input to the tokenizer. This formulation enables the tokenizer to focus strictly on learning arm-agnostic kinematic primitives and high-frequency local compliance, delegating higher-level spatial-temporal coordination between arms to the downstream VLA language model.

2. Orthogonal Subspace Decomposition and Hybrid Encoding: disentangling heterogeneous action semantics

Robot actions exhibit fundamentally heterogeneous dynamics: smooth, continuous 6-DoF end-effector Cartesian translations and rotations coexist with sharp, discrete transitions of binary gripper states. M2Tok employs a hybrid encoder that interleaves 1D causal convolutions—capturing high-frequency local motion while downsampling temporal length—with Transformer layers that incorporate long-range temporal context, yielding a downsampled latent sequence \(\hat{Z} \in \mathbb{R}^{R \times d}\). At each reduced temporal step \(r \in \{1, \dots, R\}\), the latent vector \(\hat{z}_r \in \mathbb{R}^d\) is explicitly split into \(h\) orthogonal head segments: $\(\hat{z}_r = [\hat{z}_{1,r}, \hat{z}_{2,r}, \dots, \hat{z}_{h,r}], \quad \hat{z}_{i,r} \in \mathbb{R}^{d/h}\)$ This multi-head structural partitioning encourages the network to implicitly isolate disparate kinematic attributes (e.g., isolating translation trajectories, rotation changes, and gripper grasp events) into distinct subspaces, preventing mutual interference and semantic smearing across action dimensions.

3. Combinatorial Quantization: exponential representation density under constant codebook capacity

Conventional vector quantization schemes rely on a single codebook of size \(V\), where expressivity is strictly bounded by \(|V|\). In contrast, M2Tok assigns \(h\) independent codebooks \(\mathcal{C} = \{\mathcal{Z}_1, \dots, \mathcal{Z}_h\}\), each containing \(N\) learnable prototypes of dimension \(d/h\), such that \(\mathcal{Z}_i = \{z_{i,n}\}_{n=1}^N\). Quantization is executed independently within each subspace via nearest-neighbor lookup: $\(q_{i,r} = \arg\min_{z_{i,n} \in \mathcal{Z}_i} \|z_{i,n} - \hat{z}_{i,r}\|_2\)$ The quantized latent vector \(z_{q,r} = [z_{1,q_{1,r}}, \dots, z_{h,q_{h,r}}]\) is formed by concatenating the retrieved prototype vectors. By partitioning a budget of \(V\) entries into \(h\) codebooks of size \(V/h\), the combinatorial space of unique action configurations scales exponentially to \((V/h)^h\). In the default configuration (\(h=8\) codebooks with 256 entries each, totaling 2048 parameters), the achievable state capacity reaches \(256^8 \approx 1.84 \times 10^{19}\). This combinatorial density minimizes quantization error down to millimeter-scale precision without inflating vocabulary sizes.

4. Action-Token Conversion Module: bridging discrete codebook geometry and LLM embedding space

In standard VLA autoregressive generation, prior discrete tokens \(q_{i-1,r}\) are passed through a randomly initialized token embedding layer to predict the next token. This standard practice discards the learned metric geometry of the VQ codebooks. To preserve physical continuity, M2Tok replaces generic token lookup with a lightweight conversion module: for each newly generated discrete token index \(q_{i-1,r}\), it directly fetches the pre-trained quantized prototype vector \(z_{i-1,q_{i-1,r}}\) from the M2Tok codebook and projects it via an MLP into the LLM embedding manifold. This mechanism injects rich, uncorrupted kinematic semantics directly into the context stream of the language model during sequential action token rollout.

Loss & Training

M2Tok is optimized in a decoupled two-stage training scheme. The action tokenizer is trained first under a standard VQ-VAE objective incorporating reconstruction, embedding, and commitment losses:

\[\mathcal{L} = \lambda_1 \mathcal{L}_{\text{rec}}(\hat{a}, a') + \lambda_2 \sum_{i=1}^h \sum_{r=1}^R \|\text{sg}[\hat{z}_{i,r}] - z_{i,q_{i,r}}\|_2^2 + \lambda_3 \sum_{i=1}^h \sum_{r=1}^R \|\text{sg}[z_{i,q_{i,r}}] - \hat{z}_{i,r}\|_2^2\]

where \(\text{sg}[\cdot]\) denotes the stop-gradient operator. The tokenizer is optimized with AdamW (learning rate \(5 \times 10^{-5}\), total batch size \(2048 \times 4\)) on 4 RTX 4090 GPUs.

In the second stage, the pre-trained M2Tok and SigLIP visual encoder (siglip-so400m-patch14-224) are completely frozen. The policy employs Qwen2.5-0.5B as the LLM backbone, optimizing the action generation heads via categorical cross-entropy next-token prediction:

\[\mathcal{L}_{\text{AD}} = -\sum_{i=1}^h \sum_{r=1}^R \log P(q_{i,r} \mid q_{<}, o_t, s_t, L)\]

Training is conducted using AdamW with a base learning rate of \(1 \times 10^{-4}\) and a cosine decay schedule (warm-up ratio 0.03), running 10 epochs on RoboTwin and 20 epochs on Simpler-Env.

Key Experimental Results

Main Results

Evaluations were performed across the RoboTwin bimanual manipulation benchmark (12 diverse tasks, 100 rollouts per task), the Simpler-Env benchmark (4 tasks based on BridgeV2), and real-world zero-shot transfer on an AgileX Cobot Magic dual-arm platform.

In terms of trajectory reconstruction fidelity on RoboTwin, M2Tok substantially outperforms all prior tokenizers:

Tokenizer Method	Discretization Paradigm	RoboTwin Reconstruction L1 Loss (\(\downarrow\))
FAST	Frequency DCT + BPE	0.0055
VQ-BET	2-layer Residual Vector Quantization (RVQ)	0.0044
VQ-VLA	2D Temporal Conv + RVQ	0.0032
M2Tok (Ours)	Multi-head Decomposition + Multi-codebook Quantization	0.0024

Downstream task success rates across representative RoboTwin manipulation tasks and overall averages are summarized below:

Tokenizer Model	Average Success (\(\uparrow\))	Move Can Pot	Place Burger Fries	Move Pillbottle Pad	Handover Mic
Binning (OpenVLA baseline)	0.24	0.00	0.01	0.02	0.85
FAST	0.17	0.08	0.00	0.01	0.17
VQ-BET	0.29	0.13	0.15	0.06	0.59
VQ-VLA	0.45	0.30	0.52	0.20	0.93
M2Tok-based VLA (Ours)	0.51	0.58	0.64	0.33	0.94

On Simpler-Env, M2Tok achieves an average success rate of 28% (a 33% relative margin over VQ-VLA's 21%). In the complex "Put Eggplant in Basket" task involving irregular geometry grasping, all previous tokenizers failed completely (0% success), whereas M2Tok achieved 33% success. In real-world zero-shot evaluation on the Cobot Magic platform across 20 trials per task, M2Tok attained an average success rate of 0.33, surpassing VQ-VLA (0.28) and VQ-BET (0.15).

Ablation Study

Ablation experiments conducted on RoboTwin 12 tasks isolate the impact of each core structural innovation:

Configuration	Average Success Rate	Relative Drop	Description
M2Tok (Full Model)	0.51	-	Complete multi-head multi-codebook architecture (\(h=8\)) with conversion module
w/o Multi-head	0.22	-56.9%	Collapse into single-head latent space, inducing severe semantic entanglement
w/o Multi-codebook	0.34	-33.3%	Quantize with a single shared codebook, losing combinatorial expressivity
w/o Conversion	0.42	-17.6%	Revert to generic token embedding lookup without codebook geometric features

Key Findings

• Multi-head subspace decomposition is the most critical pillar for fine-grained control: Removing the multi-head mechanism causes success rates to plummet from 51% to 22%. In precision-demanding tasks such as Move Pillbottle Pad and Place Mouse Pad, success rates drop to zero (0.33 \(\to\) 0.00, 0.10 \(\to\) 0.00). Without explicit subspace separation, continuous end-effector translations and binary gripper triggers smear across shared channels, destroying delicate grasp alignment.
• Codebook partition parameter \(h\) exhibits optimal saturation at \(h=8\): Scaling the number of codebook heads \(h\) from 1 to 8 steadily increases policy performance across all 12 tasks, peaking at \(h=8\). Expanding beyond 8 heads results in diminishing returns and minor degradation, caused by latent over-fragmentation where sub-vectors become too low-dimensional to preserve cohesive local geometry.
• Discrete autoregression unlocks high-throughput real-time deployment: On a single RTX 4090, native PyTorch inference of M2Tok runs at 8.2 Hz (121.6 ms latency), a 4.1\(\times\) speedup over Binning (2.0 Hz, 485.3 ms). Because M2Tok retains a purely discrete tokenized formulation, it is directly compatible with production LLM inference engines like vLLM. Integrated with vLLM, M2Tok achieves 56.2 Hz (17.8 ms latency), vastly exceeding real-time physical control demands (10-20 Hz) while bypassing the multi-step denoising latency inherent to diffusion heads.

Highlights & Insights

• Combinatorial capacity leverage: By partitioning a fixed codebook parameter budget into multiple orthogonal codebooks, M2Tok scales representational state density exponentially (\((V/h)^h\)) rather than polynomially, achieving continuous-level approximation fidelity at discrete token computational cost.
• Bimanual factorization as a foundational inductive bias: Factoring dual-arm 14-DoF trajectories into symmetric 7-DoF single-arm streams effectively doubles training demonstration density and encourages the tokenizer to learn arm-agnostic kinematic primitives, simplifying multi-arm embodied modeling.
• Engineering dividends of discrete autoregression: While many recent VLA models transition toward continuous diffusion heads at the cost of breaking LLM architectural uniformity and inference compatibility, M2Tok demonstrates that overcoming the discretization bottleneck allows pure autoregression to achieve state-of-the-art precision alongside >50 Hz real-time inference via vLLM.

Limitations & Future Work

• Implicit semantic allocation across heads: Subspace decomposition relies on unsupervised optimization without explicit constraints guaranteeing that specific heads dedicate strictly to gripper state versus end-effector rotations. Introducing structured kinematic regularizers could enhance interpretability.
• Sim-to-real gap under physical contact dynamics: While achieving state-of-the-art zero-shot transfer, an average success rate of 33% in physical environments shows that real-world sensor noise, lighting variations, and contact dynamics still challenge open-loop chunk execution.
• Fixed action chunk horizon: The tokenizer operates on a static action chunk window length. Dynamic-horizon tokenization could further improve precision in contact-rich or reactive phases.

Related Work & Insights

• vs FAST [25]: FAST compresses frequency-domain DCT coefficients via text BPE, which results in variable token lengths that impair parallel decoding in LLMs and suffers from substantial rounding distortion. M2Tok operates in the latent temporal domain with fixed-length tokens, reducing reconstruction L1 error by over 56% (0.0055 \(\to\) 0.0024).
• vs VQ-BET [15] & VQ-VLA [33]: Both utilize Residual Vector Quantization (RVQ), suffering from semantic entanglement and vocabulary capacity limits. M2Tok replaces RVQ with orthogonal multi-head decomposition and combinatorial codebooks, outperforming VQ-VLA by 13% relative margin on RoboTwin.
• Takeaway: For heterogeneous multi-axis physical signals (translations, rotations, binary grips), monolithic latent quantization inevitably blurs fine-grained dynamics. Orthogonal subspace decomposition coupled with combinatorial codebooks represents a principled architectural design pattern for continuous-to-discrete embodied tokenization.

Rating

• Novelty: ⭐⭐⭐⭐ [Introduces multi-head subspace decomposition and combinatorial multi-codebook quantization to resolve the long-standing discretization bottleneck in robotic action tokenization]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive validation across 12 bimanual RoboTwin tasks, 4 Simpler-Env cross-domain tasks, zero-shot real-world physical trials, in-depth component ablations, and inference speed benchmarking]
• Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous presentation linking mathematical capacity scaling directly to empirical manipulation performance]
• Value: ⭐⭐⭐⭐ [Offers an open-source, highly efficient, and low-latency discrete action tokenization foundation for the open-source VLA community]