AC3S: Adaptive Conditioning for 3D-Aware Synthetic Data Generation¶

Conference: ECCV 2026
arXiv: 2606.31204
Area: 3D Vision
Keywords: Synthetic Data, Diffusion Models, Adaptive Conditioning, Multi-Agent VLM, Geometric Alignment

TL;DR¶

This paper proposes AC3S, which dynamically attenuates the conditioning injection strength of ControlNet using an adaptive visual prompt modulator, supplemented by a multi-agent VLM system to generate text prompts consistent with geometric structures. While maintaining precise 3D pose alignment, this approach significantly enhances the realism and diversity of synthetic images, reducing the FID by 15.95 points compared to the baseline 3D-DST and achieving notable improvements in downstream classification and pose estimation tasks.

Background & Motivation¶

Synthetic data generation has become a key methodology for alleviating data scale bottlenecks in computer vision. Traditional methods rely on physical rendering engines (such as Blender) to manually construct scenes. Although they allow precise control over lighting, cameras, and object layouts, generating realistic images at scale remains extremely time-consuming and requires extensive artistic resources. The rise of diffusion models offers a new perspective: pretrained text-to-image models can generate high-fidelity photos at minimal cost, but they are inherently 2D generators lacking fine-grained control over 3D geometric structures, preventing them from being directly applied to downstream tasks that require precise 2D/3D annotations.

Recently, 3D-DST proposed a pipeline that balances 3D control with the generation capability of diffusion models. It renders images using CAD models and extracts their Canny edge maps as visual prompts for ControlNet, thereby securing precise alignment between the generated image poses and the rendered models while automatically obtaining 3D annotations. While this scheme shows remarkable results in pose accuracy, it introduces a severe side effect: blurry backgrounds, simplified object textures, and distorted scenes in the generated images. The root cause lies in the over-conditioning of ControlNet. The edge maps extracted from textureless and backgroundless CAD renderings are extremely sparse. When processed through ControlNet, the output features become overly dominant, severely suppressing the rich generation priors originally present in the pretrained diffusion models and causing the images to lose realism. The authors quantified this conflict with an interesting observation: adding text prompts reduces the denoising loss, whereas injecting the ControlNet output conversely increases the loss—indicating that the ControlNet output acts as harmful noise to generation quality and needs to be attenuated rather than fully adopted.

The key insight of this paper is that the optimal amount of attenuation varies by sample—different object categories and random seeds require different ControlNet conditioning strengths, but this optimal value correlates with the magnitudes of the U-Net and ControlNet feature norms and is therefore predictable. Core Idea: A lightweight MLP is utilized to dynamically predict the attenuation coefficient for each image based on the U-Net and ControlNet feature norms at the first timestep of the diffusion process, changing the ControlNet injection from the original f+F(prompt) to f+λ·F(prompt). Concurrently, a 7-agent VLM system is designed to collaboratively generate text prompts consistent with the geometric structure of the edge maps. This simultaneously relieves constraints on the generation prior from both visual and textual conditioning dimensions to maximize image quality and diversity while preserving precise pose alignment.

Method¶

Overall Architecture¶

The complete generation pipeline of AC3S consists of four interconnected modules. The input is a CAD 3D model and a reference image selected arbitrarily from ImageNet, and the output is a high-quality synthetic image precisely aligned with the model's pose along with its corresponding 2D/3D annotations. The core innovation lies in two adaptive components: the visual prompt modulator (which adaptively attenuates the ControlNet injection strength) and the multi-agent VLM system (which automatically generates highly detailed text prompts semantically consistent with the visual prompts).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["CAD 3D Model<br/>+ Reference Image I_ref"] --> B["Rendering + Canny Edge Extraction<br/>I_render → Edge Map I_prompt"]
    B --> C["ControlNet"]
    C --> D["ControlNet Output"]
    D --> E["Adaptive Modulator<br/>3-layer MLP"]
    B --> F["Multi-Agent VLM<br/>7-Agent Collaboration"]
    F --> G["Text Prompt T<br/>Positive + Negative Prompts"]
    B --> H["Diffusion U-Net"]
    H --> I["U-Net Intermediate Feature f"]
    I --> E
    E --> J["Attenuation Coefficient λ"]
    J --> K["Attenuated Injection<br/>f̂ = f + λ·F"]
    K --> H
    G --> H
    H --> L["Synthetic Image<br/>+ 2D/3D Annotations"]

Key Designs¶

1. Adaptive Visual Prompt Modulator: Dynamically Attenuating ControlNet Injection Based on Feature Strength

The standard ControlNet injection adds the side network output element-wise to the U-Net features (f̂ = f + F). For sparse edge maps extracted from CAD renders, this direct addition is overly dominant, completely suppressing the pretrained diffusion model's original ability to generate rich textures and backgrounds. The authors' core idea is to introduce a sample-adaptive attenuation coefficient λ, modifying the injection to f̂ = f + λ·F. The range of λ is set to [0.3, 1.0] (experimentally, values below 0.3 lead to complete pose misalignment, yielding no effect).

The modulator is implemented as a three-layer MLP. At the first timestep of the diffusion process, the MLP takes the L2 norms of the intermediate U-Net feature f and the ControlNet output F (‖f‖ and ‖F‖) as inputs to predict the value of λ, which is then shared across all subsequent timesteps. The rationality of this design lies in the strong correlation between the optimal λ and these two feature norm magnitudes: a larger ControlNet output feature norm relative to the U-Net feature norm indicates a higher attenuation requirement, and vice versa.

The training strategy of the modulator is one of the most elegant aspects of this work. Because λ balances the conflicting objectives of 'pose alignment' and 'image quality', direct supervision with denoising loss would lead to a degenerate solution where the model learns to always output λ=0 (minimizing denoising loss but yielding complete pose misalignment). The authors design a self-supervised pseudo-label generation algorithm: for each rendered image, images are generated using λ values from 0.3 to 1.0 (with a step of 0.1, totaling 8 candidate values), producing a sequence that naturally clusters into two categories—misaligned poses at low λ values and aligned poses at high λ values. Then, k-means clustering (k=2) is performed on pixel-level features, marking the cluster containing the λ=1.0 image as the 'aligned cluster' and selecting the minimum λ within this cluster as the optimal λ. This essentially detects a 'Just Noticeable Difference' (JND) threshold: the pose is observed to align only after λ exceeds this threshold, and further increases only needlessly degrade image quality. These pseudo-labels are used to train the MLP for an 8-class classification task, requiring no manual annotation.

2. Multi-Agent VLM Text Prompt Generator: Eliminating Semantic Conflicts Between Text and Visual Conditions

The quality and consistency of text prompts represent another bottleneck in scaling synthetic data. Fixed templates (e.g., 'a high-quality photo of [category]') are too coarse, whereas descriptions automatically generated by VLMs from real images, though rich in detail, may describe scenes that conflict with the geometric structure shown in the edge maps. For instance, the real image might be a close-up while the edge map displays a global view, and such conflicts can lead to hallucinations and degradation during the generation process. This paper addresses this by decomposing prompt generation into a collaborative pipeline of 7 specialized agents.

The seven agents each have distinct responsibilities. The Subject Identifier receives both the rendered image and the reference image, performing cross-validation to output a 1-3 word subject label (relying solely on ImageNet class names would lose critical semantic context). The Genre Selector chooses the style that best matches the subject from predefined photographic genres (wildlife, macro, landscape, etc.). The Pose Extractor extracts the subject's pose and camera viewpoint from the rendered image, writing this information directly into the prompt to ensure perspective consistency with the edge map. The Scene Composer extracts visual attributes such as the subject's color and texture, along with background scene descriptions, from the reference image. The outputs of the first three are compiled by the Positive Prompt Synthesizer to assemble a coherent positive prompt. The Protected Terms Identifier identifies key terms in the positive prompt to put on a protection list, preventing negative prompts from erroneously excluding them. Finally, under the constraint of protected words, the Negative Prompt Synthesizer composes meaningful negative prompts tailored to the current subject and scene.

The system is based on Qwen3-VL-23B-Instruct (4-bit quantized), orchestrated as a directed acyclic graph via LangGraph's StateGraph and utilizing MetaGPT's shared message pool pattern for inter-agent communication. The prompt template for each agent contains 7 structured fields (role, goal, context, constraints, comparison examples, output format, acceptance criteria), bounding the tasks within a scope with clear success criteria to make the outputs reliable and automatically parsable. Upstream agents like the Subject Identifier and Scene Composer can execute in parallel, while downstream agents run sequentially depending on upstream outputs.

Loss & Training¶

The modulator is trained using a self-supervised classification strategy: by discretizing the consecutive values of λ into 8 candidate values and generating pseudo-labels for each rendered image via the JND detection algorithm, the MLP is trained to perform cross-entropy classification over these 8 classes. This paper explicitly avoids utilizing the denoising loss—since the denoising loss is trivially minimized at λ=0, leading the model to learn a degenerate solution of always outputting 0. The pseudo-label scheme bypasses this pitfall by directly modeling the pose alignment threshold. Additionally, for the generated dataset, AC3S employs DreamSim to replace the filtering schemes that require training individual pose estimators for each category, significantly reducing computational overhead.

Key Experimental Results¶

Main Results¶

Ablation experiments are conducted on 50 ImageNet categories, generating 1,000 synthetic images per category (totaling 50,000 images). Stable Diffusion v1.5 with 20-step sampling is uniformly used as the diffusion model. Image quality is evaluated using FID against the real ImageNet distribution.

Method	Animal FID↓	Object FID↓	Overall FID↓
3D-DST (Baseline)	72.67	90.38	87.19
+ Visual Prompt Modulator	60.93	81.18	77.53
+ VLM Text Prompt Generator	54.68	84.23	78.93
AC3S (Both Combined)	48.87	76.15	71.24

Top-1 accuracy of downstream classification tasks (pretraining followed by ImageNet fine-tuning):

Method	ResNet-18	ConvNeXt-S	Swin-S
Real Data Only (No Pretraining)	76.34	80.12	81.59
w/ Text2Img Pretraining	77.08	79.21	81.36
w/ 3D-DST Pretraining	76.96	81.89	83.17
w/ AC3S Pretraining	77.33	82.75	84.19

Ablation Study¶

Config	Overall FID↓	Note
3D-DST (Baseline)	87.19	No adaptive control, leading to degradation of both textures and backgrounds
+ Visual Prompt Modulator	77.53	-9.66 FID, significantly richer texture and background details
+ VLM Text Prompt Generator	78.93	-8.26 FID, improved color and scene plausibility
AC3S (Full)	71.24	-15.95 FID, animal categories show the largest improvement (-23.8)

Pose estimation experiments (pretraining followed by PASCAL3D+ fine-tuning, π/6 threshold):

Method	Sofa	Dining Table	Car
ResNet Baseline (No Pretraining)	80.79	79.86	87.60
w/ 3D-DST Pretraining	86.70	80.20	88.51
w/ AC3S Pretraining	88.18	81.91	89.03

Key Findings¶

The visual prompt modulator and VLM text prompts each contribute an 8-10 point improvement in FID, with their synergistic combination reaching a 15.95 point improvement. This demonstrates that dual-channel adaptive control from both visual and textual sources is indispensable. Animal categories exhibit a greater improvement than object categories (-23.8 vs. -14.2 FID), because animals require higher texture detail and background complexity and are thus more severely affected by over-conditioning.
In pose estimation, the sofa category under the fine-grained π/18 threshold shows the largest improvement (+7.39%), indicating that precise attenuation resolves the generation quality bottleneck while preserving pose alignment.
The purely synthetic data experiments are the most impressive: without any real-data fine-tuning, the model pretrained with AC3S achieves an accuracy of 34.48% on PASCAL3D+ (Sofa, π/6), compared to only 10.84% for 3D-DST. This demonstrates that adaptive conditioning significantly narrows the gap between synthetic data and real distributions, making purely synthetic data initially viable for the first time.

Highlights & Insights¶

Self-supervised JND pseudo-labels as the highlight: Employing k-means clustering on the λ scan sequence to detect the threshold for pose alignment provides high-quality training signals without requiring any human annotation. This concept is highly inspiring for any conditioning control problem that requires automatic threshold calibration.
From 'one condition fits all' to 'personalized condition for each sample': While traditional ControlNet imposes the same conditioning injection strength across all images, AC3S is the first to demonstrate that the optimal intensity varies across samples and is predictable via feature norms, paving the way for finer adjustments in conditioning.
Multi-agent decomposition to resolve text-geometry conflicts: Decomposing the error-prone 'end-to-end description generation' into 7 verifiable, specialized roles—each with explicit inputs, outputs, and acceptance criteria. The Pose Extractor is specifically designed for pose alignment, and the Protected Terms Identifier prevents internal contradictions in positive and negative prompts. This systems engineering approach is readily transferable to other image generation scenarios requiring fine-grained conditioning control.
DreamSim replacing per-category pose estimator training: The filtering step in 3D-DST requires training a separate pose estimator for each category, which is computationally expensive. AC3S uses the perceptual similarity of DreamSim to perform a single forward ranking pass to complete the filtering, vastly improving efficiency and generalizing much better.

Limitations & Future Work¶

This work is restricted to Stable Diffusion v1.5. Whether the modulator requires retraining on stronger base models (e.g., SDXL, SD3, FLUX) remains to be verified, as different models may exhibit varying sensitivity to ControlNet conditioning strength.
The modulator only predicts λ at the first timestep, which is then shared across all subsequent timesteps. The authors note that the optimal λ is correlated with the random seed, suggesting potential timestep dependency. A time-varying multi-step prediction scheme could theoretically further improve quality but would increase computational overhead.
The multi-agent VLM relies on Qwen3-VL-23B inference, which incurs high costs. Exploring lightweight models or distilling parts of the agents (such as relatively simple roles like the Subject Identifier or Genre Selector) is an important direction for scaling up application.
The CAD model coverage is limited (approximately 30 models per category), restricting object diversity to the quality and richness of the model repository itself. Integrating with larger-scale 3D datasets (such as Objaverse-XL) is a natural direction for extension.

vs. 3D-DST: 3D-DST established the 'CAD rendering → edge map → ControlNet' 3D-aware synthetic paradigm but lacks conditioning attenuation, resulting in severe over-conditioning. AC3S is a direct improvement upon it, resolving the generation quality issue with two adaptive modules: the visual prompt modulator and the multi-agent VLM.
vs. original ControlNet: ControlNet's standard injection (λ=1) works well for real depth map or real edge map prompts, but is overly dominant for sparse, textureless visual prompts like CAD renderings. AC3S essentially builds a tailored, adaptive conditioning intensity calibrator specifically for the domain of synthetic data.
vs. other prompt modulation methods: Concurrent works like TASR and Navigating also employ similar modulators but rely on denoising loss supervision. The key distinction of AC3S is revealing the degenerate solution problem at λ=0 and designing a self-supervised pseudo-label scheme to circumvent it.

Rating¶

Novelty: ⭐⭐⭐⭐ — The idea of adaptively attenuating ControlNet is simple yet effective, with the self-supervised JND pseudo-labels serving as a notable highlight. While structurally built upon 3D-DST, showing strong incremental attributes, it delivers highly solid solutions to key issues.
Experimental Thoroughness: ⭐⭐⭐⭐ — The evaluation is comprehensive, featuring FID ablations, classification experiments across three architectures, and pose estimation experiments with purely synthetic data comparisons. The lack of generalization testing on additional datasets (such as COCO or LVIS) is a minor drawback.
Writing Quality: ⭐⭐⭐⭐ — The logical reasoning is clear, with a complete narrative of 'problem discovery → conflict quantification → proposed solution' readily visible in the first two sections. The analysis and circumvention of the modulator's degenerate solution are thoroughly explained.
Value: ⭐⭐⭐⭐ — Synthetic data generation is a high-demand topic in the computer vision community, and AC3S offers a directly practical scalable solution. The significant advantage of purely synthetic training (Sofa 34.48% vs. 10.84%) holds notable real-world value.