Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/AI9Stars/CHEERS
Area: Multimodal VLM / Image Generation
Keywords: unified multimodal models, semantic-detail decoupling, visual token compression, cascaded flow matching, high-frequency injection

TL;DR

Cheers routes compressed visual semantics through a shared language backbone and injects patch details according to generation progress through a cascaded flow matching head, achieving 0.78 on GenEval with a 1.5B language backbone and 83M training samples while retaining competitive image understanding.

Background & Motivation

Visual understanding and image generation both process images, but they need information at different granularities. Answering questions about object categories, chart contents, or spatial relations requires semantic representations aligned with language; generating images also requires local textures and boundaries. Discretizing images into vocabulary tokens makes it possible to reuse next-token prediction, but quantization errors and limited representation capacity can compromise image fidelity. Continuous diffusion representations alleviate this discretization bottleneck without automatically resolving the different feature requirements of understanding and generation. Unification therefore involves more than attaching two output heads to a backbone: it requires deciding which information belongs in the shared representation and which should remain in the generation process.

Existing approaches make different trade-offs: Janus-style models separate visual encoding for understanding and generation, whereas semantic-centric representations may omit details needed for reconstruction. Directly fusing heterogeneous visual features can instead make understanding and detail reconstruction objectives interfere within the same feature space. Cheers retains a shared semantic interface while routing details closer to the image output, rather than requiring the language backbone to carry all visual information. This also makes compression worth considering: if the decoder can recover details, the expensive language backbone may not need every high-resolution patch token. However, local text can itself carry semantics, so fine-grained information cannot simply be dismissed as noise for understanding.

The paper uses the pretrained pixel interface of SigLIP2 to extract semantics and reuses shallow patch features in the generation head. This is not a pair of entirely independent vision models, but a division of responsibilities between different depths and granularities within the same visual tokenization process. Core Idea: let compressed semantics handle cross-modal modeling and a detail pathway modulated by semantic state handle later visual refinement, reducing representational conflict through the information flow.

Method

Overall Architecture

Cheers comprises a unified vision tokenizer, a hybrid-decoding backbone, and a cascaded flow matching head, alongside a conventional text tokenizer and language modeling head. Inputs can be an image with a question or a text condition for image generation; the former produces text, whereas the latter iteratively transforms latent noise into an image. The understanding pathway only needs semantic tokens to produce an answer through the language backbone; generation re-encodes the current latent variable at every sampling timestep. Generation therefore does not compute a text condition once and delegate all subsequent work to a completely independent diffusion model. The shared backbone repeatedly participates in visual semantic modeling, while the detail pathway bypasses it and enters the cascaded flow matching head.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image or current latent"] --> Tokenizer["Unified Vision Tokenizer"]
    Text["Text condition"] --> Backbone["Hybrid-Decoding Backbone"]
    Tokenizer -->|Compressed semantics| Backbone
    Backbone -->|Autoregressive decoding| Answer["Text answer"]
    Backbone -->|Visual hidden states| Head["Cascaded Flow Matching Head"]
    Tokenizer -->|Patch details at the current timestep| Head
    Head --> Update["Velocity field and latent update"]
    Update -->|Next timestep| Tokenizer
    Update -->|VAE decoding after sampling| Image["Output image"]

The loop represents image-generation data flow, not repeated denoising for understanding tasks. During training, the clean latent supplies flow matching supervision; at inference time, the generation branch does not access the target image or its details. The three designs below follow the order of encoding, shared modeling, and generative decoding.

Key Designs

1. Unified Vision Tokenizer: return to the pixel interface before compressing semantics

The input image is first encoded by a VAE, reducing each spatial dimension by a factor of 16. For generation training, the clean latent is linearly mixed with Gaussian noise; understanding uses the clean endpoint, while language-only tasks use the noise-endpoint convention. The path specified on page 4 is:

\[ z_t=t z_1+(1-t)z_0,\qquad z_0\sim\mathcal{N}(0,I). \]

Here \(z_1\) is the image latent; generation training samples \(t\in(0,1)\), understanding fixes \(t=1\), and language-only tasks specify \(t=0\). The crucial step decodes the current \(z_t\) into pixel space with the VAE decoder before passing it to pretrained SigLIP2-ViT. Unlike replacing the ViT input with a randomly initialized latent patch embedding, this preserves the input format familiar to the pretrained vision model. The authors argue that direct latent processing damages fine-grained features and OCR, but the main text directs the dedicated evidence to supplementary material absent from this cache. Retaining the pixel interface is thus an explicit architectural choice, not an independently quantified OCR improvement established by the main-text ablation.

SigLIP2 uses \(16\times16\) patch embeddings, aligning the patch grid spatially with the VAE latent grid. Deep visual features supply semantic tokens, while shallow patch embeddings supply the generation head with a source of local details. Before entering the language backbone, Pixel-Unshuffle rearranges neighboring spatial positions into channels, followed by projection to the language model hidden dimension. Halving both output-grid dimensions reduces the number of semantic tokens to one quarter. This does not mean losslessly compressing all visual information to one quarter, nor does it establish an exactly fourfold reduction in total computation. The detail branch retains a finer grid, and generation still executes VAE decoding and visual encoding.

2. Hybrid-Decoding Backbone: causal prediction for text and bidirectional context for images

The language backbone is initialized from Qwen2.5-1.5B-Instruct and receives a concatenation of text embeddings and compressed visual semantic tokens. Visual tokens use bidirectional attention to access global image context; text tokens use causal attention to support autoregressive output. Unification therefore occurs in contextual modeling rather than by forcing images and text to share exactly the same output probability model. For visual question answering, a standard language modeling head generates the answer token by token without asking the image head to render it. For image generation, textual conditioning shapes visual hidden states that the cascaded flow matching head converts into a direction of motion in latent space.

This division avoids requiring the language backbone to predict complete high-frequency patch representations directly. It primarily models global layout and relationships between objects and text conditions, leaving detail recovery to a specialized head. Nevertheless, the semantic versus high-frequency distinction is architectural and representational; the main text imposes no strict frequency-domain orthogonality constraint. Shallow patch features are not obtained through an explicit Fourier high-pass filter, so decoupling should not be interpreted as mathematically complete separation of frequency components. Understanding and generation still share parameters, and controlled experiments are needed to assess interference rather than assuming that separate pathways guarantee its absence.

3. Cascaded Flow Matching Head: restore the semantic grid before gated detail injection

The generation head has two cascaded stages containing 7 and 3 DiT blocks, respectively, both using AdaLN-Zero for timestep modulation. The first stage receives low-resolution visual hidden states from the backbone and processes global semantic structure. PixelShuffle then doubles both spatial dimensions and adjusts the channels, realigning the generated features with the original patch grid. The second stage receives shallow patch features extracted from the image decoded from the current latent, then uses subsequent DiT blocks to output a velocity field. Semantics first and details second initially describes two stages inside each velocity prediction, not two independent complete image-sampling procedures.

The gating network reads upsampled semantic features that have already been timestep-modulated and outputs a scalar at each spatial position to control detail injection. The detail pathway consequently depends on semantic state, location, and generation progress instead of using a fixed additive weight throughout sampling. Figure 5 shows relatively weak early injection, a possible mid-stage dip, and stronger late injection; this is not strict monotonic growth at every step. There is no dedicated supervision for that temporal curve; it is an observation of the trained model. The plotted statistics also normalize each run before averaging, making them useful for relative trends rather than absolute cross-model injection comparisons. The gate-update equation on page 6 loses operators in the text extraction, so this note explains gated detail residuals from the prose without inventing an exact author equation.

A Worked Example

Consider a text-to-image request specifying two differently colored objects with a left-right relation; this is an architectural walkthrough, not an additional experimental example. At the paper's \(512\times512\) resolution, the VAE latent grid is \(32\times32\), containing 1024 spatial positions. The tokenizer extracts semantic features on that grid from the current noisy state, then compresses them to \(16\times16\), or 256 semantic tokens, for the backbone. The backbone jointly processes the text condition, objects, colors, and positional relation, while the first head stage further organizes low-resolution semantic structure. After PixelShuffle restores the \(32\times32\) grid, gated patch features supply local information from the current state, and the second stage predicts the velocity field. Numerical integration updates the latent, and the entire pathway repeats until final VAE decoding produces the image. Because early inputs remain noisy, the gate should not be interpreted as copying textures from a clean reference image. For image question answering instead, the image occupies the clean endpoint and follows only semantic encoding, the backbone, and text output.

Loss & Training

Text tasks use conditional autoregressive cross-entropy; the image objective supervises the velocity from noise to the image along the linear path above. The flow matching objective on page 7 can be written as:

\[ \mathcal{L}_{FM}=\left\|v_\theta\left(Z_s'^{(t)}\right)-(z_1-z_0)\right\|_2^2. \]

Here \(Z_s'^{(t)}\) denotes visual features processed by the generation head, and the target velocity is the difference between the clean latent and initial noise. The prose defines the total objective as a weighted sum of text and image losses, with image-loss weight \(\lambda=1\); its extracted equation also loses symbols, so the corrupted expression is not reproduced. Generation inference starts from noise and repeatedly predicts velocity while numerically solving an ODE, with classifier-free guidance and a shifted time schedule. The main text does not provide a CFG strength, schedule-shift value, or uniform sampler configuration verifiable in this cache, so a complete reproduction command cannot be inferred.

Training uses \(512\times512\) images, AdamW, a 0.02 warmup ratio, and 1.0 gradient clipping on 128 A100 GPUs. Stage I trains only the projection, CFM head, and gating modules with 4.5M image-caption pairs and 1.3M relabeled ImageNet samples, repeating the latter 10 times to establish generation capability. Stage II uses 30M samples in an understanding, generation, and text ratio of \(3:6:1\), jointly optimizing all parameters except the VAE. Stage III uses 33M samples at the same \(3:6:1\) ratio, adding synthetic images and instruction data, including 466K compositional instructions constructed from Objects365. Stage IV performs supervised fine-tuning on 3.8M curated samples with an understanding-to-generation batch ratio of \(1:1\). Table 1 gives learning rates of \(10^{-4}\), \(10^{-4}\), \(4\times10^{-5}\), and \(2\times10^{-5}\), and training lengths of 30K, 60K, 65K, and 30K steps. Batch size is 512 for the first three stages and 128 for the last; learning-rate schedules are constant for the first three and cosine for the last. The discussion's claim about not freezing core parameters must be interpreted alongside this table, preserving the frozen VAE and the restricted training in Stage I.

Key Experimental Results

Main Results

The following selection combines Table 2 on page 8 with Tables 3 and 4 on page 9, restricting comparisons to unified models with 1.5B language backbones. These parameter counts refer to the language backbone, not the complete system including the VAE, vision encoder, and generation head. MMBench and ChartQA assess understanding, GenEval measures compositional text-image alignment, and DPG-Bench measures dense-prompt following; higher is better for all listed metrics.

Model	Total training samples	MMBench	ChartQA	GenEval Overall	DPG-Bench Overall
Janus-Pro	162M	75.5	23.4	0.73	82.63
Show-o2	177M	67.4	40.0	0.73	85.02
Harmon	113M	65.5	29.8	0.76	Not reported
Tar	403M	65.6	Not reported	0.76	82.96
Cheers	83M	70.4	75.7	0.78	83.48

Cheers exceeds Tar by 4.8 points on MMBench and 0.02 on GenEval, but does not exceed Janus-Pro on MMBench or Show-o2 on DPG-Bench. The paper describes training cost as approximately 20% of Tar's; the directly verifiable tabular evidence is 83M versus 403M training samples, approximately 20.6%. Without matched GPU hours, training FLOPs, and synthetic-data costs, this does not establish an 80% reduction in total compute or monetary expense.

Ablation Study

The following results come from Table 5 on page 12 and use small-scale controlled fine-tuning after alignment, not the final model trained on the full 83M samples. Joint training uses 858K understanding and 850K generation samples; the first row uses understanding data alone, while the last two compare high-frequency injection, or HFI, under the same joint settings.

Config	SEEDBench	MMBench	ChartQA	POPE	AI2D	GenEval	DPG-Bench
Understanding only, HFI architecture retained	70.8	65.2	58.5	87.0	67.7	Not applicable	Not applicable
Understanding and generation, without HFI	70.0	66.3	58.8	86.2	67.3	0.17	39.11
Understanding and generation, with HFI	69.8	67.1	59.9	87.5	68.1	0.30	51.63

Key Findings

• Under matched joint training, HFI raises GenEval from 0.17 to 0.30 and DPG-Bench from 39.11 to 51.63, gains of 0.13 and 12.52, respectively.
• Understanding gains are not universal: HFI increases MMBench by 0.8, but SEEDBench decreases from 70.0 to 69.8; limited changes in understanding are better supported than improvement on every capability.
• Compared with understanding-only fine-tuning, joint training with HFI scores 67.1 versus 65.2 on MMBench but 69.8 versus 70.8 on SEEDBench; no impairment should mean overall competitiveness rather than no individual regression.
• Figure 4 shows pronounced growth in Stage III, dominated by synthetic, instruction-oriented data; final cross-model tables alone cannot separate architectural gains from data-quality gains.

Highlights & Insights

• Compression and bypassed detail recovery form a coupled design: the backbone processes fewer spatial tokens while the head still accesses a finer grid. This addresses both tasks more directly than minimizing token count alone.
• The gate is derived from timestep-modulated semantic states, avoiding a hand-coded rule for when textures must activate. Figure 5 shows late-stage strengthening, but does not provide strict causal proof of the mechanism.
• Reusing a pretrained pixel encoder reduces the burden of relearning a semantic vision interface. It also requires latent decoding and visual encoding at every generation timestep, which efficiency evaluation should include.

Limitations & Future Work

• The authors acknowledge limited model scale and the lack of direct initialization from a large pretrained VLM, leaving room for better intricate details and stronger understanding.
• They also note reliance on single-image training data; video, cross-image context, and more complex multimodal inputs remain future directions rather than demonstrated coverage.
• Reader assessment: the HFI ablation uses reduced training data, and the main text lacks complete comparisons against fixed injection or different compression ratios, limiting finer attribution of the gains.
• The cache contains the main paper and references but no supplement; extracted gate-update, integration, and total-loss expressions are damaged, and exact author equations were not reconstructed from them.
• The conclusion mentions zero-shot image editing without a corresponding quantitative evaluation in the main text read here; this note does not list it as a thoroughly validated main capability.

Related Work & Insights

• vs Janus / Janus-Pro: these models separate visual encoding spaces, whereas Cheers retains a unified semantic interface and places the distinction in generation-side detail injection. Both address task differences, but decouple at different locations.
• vs Show-o2 / Transfusion: these share the hybrid principle of autoregressive text with image diffusion or flow modeling. Cheers focuses on combining shared semantic compression with decoder-side detail recovery rather than hybrid decoding itself.
• vs TUNA / RAE: TUNA extracts semantics from latents and RAE emphasizes reconstruction from semantic representations; Cheers restores the pixel interface and reuses patch details. Its OCR criticism of direct latent encoding still requires the supplementary evidence.
• Research direction: under matched training samples and total inference budgets, compare compression ratios and injection locations to test whether the detail bypass preserves both OCR and compositional generation. This is a proposed follow-up, not an existing result.

Rating

• Novelty: 4/5. Combining semantic compression with time-adaptive detail injection is well targeted, though hybrid decoding and cascaded generation build on established ideas.
• Experimental Thoroughness: 4/5. Multiple understanding and generation benchmarks and controlled joint-training ablations are included; total cost and the gating mechanism need finer comparisons.
• Writing Quality: 4/5. Architecture and training stages are clear, but some claims exceed individual results; extraction damage to equations is separately identified.
• Value: 4/5. A reusable division of representational responsibilities for small unified multimodal models that merits further testing under matched budgets.