Objects as Audio-Visual Modal Sound Fields¶

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: Impact Sound Synthesis, Audio-Visual Learning, 3D Gaussian Splatting, Modal Sound Fields, Physical Priors

TL;DR¶

Audio-Visual Modal Sound Field (AV-MSF) couples 3D Gaussian Splatting visual geometry priors with compact physical modal parameters to reconstruct continuous object-level impact sound fields from multi-view RGB images and few-shot recordings, enabling novel-position sound synthesis, contact localization, and text-guided sound editing.

Background & Motivation¶

Human perception of physical objects relies not only on visual inspection but also on rich acoustic feedback generated during physical interaction. In daily life, passive visual observation often fails to distinguish objects of identical appearance but distinct mechanical properties; for instance, a transparent glass goblet and an identical plastic cup look nearly indistinguishable, yet a light tap immediately reveals the difference—glass emits a bright, resonant ring with prolonged decay, whereas plastic produces a dull, heavily damped thud. These impact sounds intrinsically convey key material properties, including elastic modulus, density, stiffness, and structural thickness. Nevertheless, mainstream 3D digitization frameworks, such as NeRF and 3D Gaussian Splatting, remain overwhelmingly appearance-centric, neglecting physical interaction acoustics and limiting their utility in high-fidelity VR contact feedback and embodied multimodal simulation.

Existing approaches to modeling object impact sounds fall into two main categories, both facing significant limitations in practical adoption and physical fidelity. Physics-based simulation methods rely on linear modal analysis applied to detailed volumetric meshes with known physical parameters (Young's modulus, Poisson's ratio, and Rayleigh damping). However, accurate material constants are hard to measure, and finite element modeling (FEM) simulations are computationally expensive; while recent differentiable inverse-rendering frameworks like DiffSound optimize physical parameters from audio, their long optimization chains and high-order FEM steps remain prone to local minima and numerical errors. Conversely, data-driven generative models like SonicGauss bypass mesh simulation but require massive training corpora, lack explicit physical constraints, and frequently suffer from hallucinated distortions and poor generalization in few-shot contact regimes.

Bridging this gap requires leveraging the synergy between physical principles and visual priors. Crucially, object impact sounds vary smoothly across the surface and correlate strongly with local geometry (such as curvature and thickness), while rigid-body vibrations naturally decompose into object-intrinsic global modal frequencies/dampings and position-dependent excitation gains. Core Idea: Decouple the object impact sound field into global intrinsic modal parameters and a continuous neural gain field, guided by 3D Gaussian Splatting geometric features and symmetry alignment, enabling physically grounded impact sound synthesis, contact localization, and material editing from multi-view images and only a few audio recordings.

Method¶

Overall Architecture¶

AV-MSF takes calibrated multi-view RGB images and few-shot impact sound recordings at sparse surface locations to reconstruct a continuously queryable object-level modal sound field. The pipeline comprises three stages: first, in the visual processing stage, a 3D Gaussian Splatting (3DGS) representation is constructed, lifting pre-trained DINOv2 features onto 3D Gaussian centers with geometric symmetry alignment and spatial cluster pooling to provide compact visual priors; second, in the acoustic processing stage, narrowband decomposition via STFT and Hilbert transform log-linear regression extracts intrinsic modal frequencies and dampings from reference recordings, alongside a learnable filter bank modeling non-modal environmental noise; third, in the reconstruction stage, a neural gain field conditions on local visual features and global attention context to predict position-specific modal gains, driving a differentiable modal synthesizer for end-to-end optimization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    InV["Multi-view RGB Images"] --> D1["Geometry- and Symmetry-Aware 3D Visual Feature Field<br/>3DGS reconstruction + DINOv2 lifting + symmetry pooling"]
    InA["Few-shot Impact Audio & Coordinates"] --> D2["Physical Modal Parameter Extraction & Residual Noise Decoupling<br/>Narrowband STFT + Hilbert decay regression + bandpass noise"]
    D1 --> D3["Local-Global Context-Conditioned Neural Gain Field<br/>Spatial clustering + offset PE + cross-region attention"]
    InA --> D3
    D2 --> D3
    D2 --> Syn["Differentiable Modal Sound Synthesizer<br/>Decaying sinusoidal modes summation + filtered noise residual"]
    D3 --> Syn
    Syn --> Out["Novel-Position Sound Rendering / Contact Localization"]
    Out --> D4["Physically Grounded Hierarchical Frequency Sound Editing<br/>Global pitch scaling + residual modal tuning + Audio-SDS"]

Key Designs¶

1. Geometry- and Symmetry-Aware 3D Visual Feature Field: Eliminating viewpoint inconsistencies and enforcing physical geometric symmetry

Directly lifting 2D visual features into 3D space frequently creates inconsistencies across symmetric object regions due to viewpoint changes and lighting variations, distorting the predicted acoustic mode shapes. AV-MSF first reconstructs a dense 3D Gaussian Splatting point cloud \(\mathcal{P} = \{o_i\}_{i=1}^O\) and assigns each Gaussian center a feature vector \(f_i \in \mathbb{R}^D\) projected from pre-trained DINOv2 embeddings. To ensure physical consistency, the system detects rotational and planar reflection symmetries by evaluating candidate transformation errors over the point cloud, performing orbit and reflection pooling across symmetric counterparts to produce refined features \(f_i^{\text{refined}}\). Subsequently, the Gaussian centers are grouped into \(K\) spatial clusters via Euclidean distance, and cluster-averaged descriptors \(\bar{f}_k = \frac{1}{|\mathcal{C}_k|} \sum_{i \in \mathcal{C}_k} f_i\) are computed to establish a compact, topologically aware global visual prior.

2. Physical Modal Parameter Extraction & Residual Noise Decoupling: Anchoring non-convex optimization in physical reality

Directly optimizing the modal sound field from few-shot recordings is ill-posed and prone to local minima, especially given unmodeled ambient noise and contact force variations. Leveraging the principle from elastodynamics that vibration frequencies and dampings are intrinsic global properties, the audio waveforms \(s_j(t)\) are decomposed into narrowband components \(s_{j,n}(t)\) via STFT and inverse-STFT. After filtering spurious peaks, log-linear regression is applied to the Hilbert transform envelope:

\[\log |\mathcal{H}\{s_{j, n}(t)\}| \approx \log(g_n) - d_n \cdot t\]

yielding candidate damping coefficients \(d_n\) and gains \(g_n\). Persistent frequencies across recordings determine the shared modal frequencies \(\{f_i\}_{i=1}^N\) and mean dampings \(\{d_i\}_{i=1}^N\). Furthermore, unpredictable non-modal artifacts are captured via a differentiable noise residual \(\epsilon(m, t) = \sum_{i=1}^F m_i (b_i * \epsilon(t))\) with learnable per-band magnitudes \(m\), isolating environmental noise from the underlying modal physics.

3. Local-Global Context-Conditioned Neural Gain Field: Predicting spatially continuous modal excitations

Given an arbitrary contact coordinate \(x \in \mathbb{R}^3\), the network must predict the mode-specific excitation gains \(\mathcal{G}_\theta(x) = [g_1(x), \dots, g_N(x)]^\top\). The architecture integrates fine-grained local geometry with global structural context: the model finds the nearest Gaussian center \(i^\star = \arg\min_i \|x - x_i\|_2\) to retrieve its local feature \(f_{i^\star}\), and computes relative offset vectors \(r_k = x_{i^\star} - c_k\) to all \(K\) cluster centers encoded via sinusoidal positional encodings \(\text{PE}(r)\). A query vector \(q = W_q [\text{PE}(r); f_{i^\star}]\) attends to key-value projections \(k_k = W_k \bar{f}_k, v_k = W_v \bar{f}_k\) of region descriptors:

\[\alpha_k = \text{softmax}_k \left( \frac{k_k^\top q}{\sqrt{d}} \right), \quad z = \sum_{k=1}^K \alpha_k v_k\]

A lightweight multi-layer perceptron \(\mathcal{G}_\theta(x) = \text{MLP}_\theta(z; f_{i^\star})\) then outputs the modal gains, harmonizing micro-scale surface geometry with macro-scale shape context.

4. Physically Grounded Hierarchical Frequency Sound Editing: Preserving spatial mode shapes under cross-material diffusion distillation

To enable semantic material sound editing via text prompts, AV-MSF adapts Audio-SDS distillation from a pre-trained text-to-audio diffusion model. Directly optimizing sparse modal frequencies independently across large spectral gaps (e.g., transforming ceramic into wood) leads to severe optimization instability. The method introduces a hierarchical frequency parameterization: \(f_i = \exp(\log \alpha + \log f_{i,\text{init}} + \Delta f_i)\), where \(\alpha\) acts as a global pitch-scaling factor capturing primary material shifts, while \(\Delta f_i\) provides per-mode residual refinement. Crucially, linear modal analysis dictates that altering Young's modulus, density, and damping scales the frequencies and decay rates while leaving the underlying spatial mode shapes unchanged:

\[a_k^{(2)}(x) = a_k^{(1)}(x) \frac{f_k^{(1)}}{f_k^{(2)}}\]

Because frequency-dependent scaling cancels out, the relative gain ratios between arbitrary contact locations remain invariant. This physical preservation law allows the model to perform text-driven sound editing simply by updating global frequency and decay scalars, keeping the learned spatial gain field intact.

Loss & Training¶

The framework is trained in two stages. In the warm-up stage, the neural gain field is initialized by minimizing the mean-squared error against the extracted target gains \(g_i\):

\[\mathcal{L}_{\text{warmup}} = \frac{1}{S} \sum_{i=1}^S \|\mathcal{G}_\theta(x_i) - g_i\|^2\]

In the end-to-end training stage, the differentiable synthesizer renders the synthesized waveform \(\hat{s}_i(t) = \sum_{k=1}^N \mathcal{G}_\theta(x_i) e^{-d_k t} \sin(2\pi f_k t) + \epsilon(m, t)\). The parameters are optimized via a multi-scale STFT loss applied to both the complete waveform and the early transient segment \(s_i[:t]\) where impact energy concentrates:

\[\mathcal{L}_{\text{spec}} = \sum_r \left( \|\text{STFT}_r(s_i) - \text{STFT}_r(\hat{s}_i)\| + w_c \|\text{STFT}_r(s_i[:t]) - \text{STFT}_r(\hat{s}_i[:t])\| \right)\]

where \(r\) indexes FFT resolutions and \(w_c\) weights the early transient reconstruction.

Key Experimental Results¶

Main Results¶

Quantitative comparisons on novel-position impact sound rendering across ObjectFolder Real (100 objects, 20% training data) and RealImpact (50 objects, leave-one-out cross-validation). Metrics include spectral L1 distance, log-mel spectral L1 distance (L1 Log), envelope distance (ENV), and perceptual audio similarity (CDPAM, lower is better).

Dataset	Method	L1 ↓	L1 Log ↓	ENV ↓	CDPAM ↓
ObjectFolder Real	White Noise	3.774	6.859	0.305	1.35e-3
	Random Impact	0.035	1.367	0.019	2.47e-4
	KNN (\(K=3\))	0.014	0.930	0.014	1.53e-4
	DiffSound (Physics-based)	0.031	1.298	0.030	2.53e-4
	SonicGauss (Data-driven)	0.033	1.281	0.018	2.01e-4
	Ours (AV-MSF)	0.013	0.951	0.014	1.35e-4
RealImpact	White Noise	3.789	7.258	0.308	1.23e-3
	Random Impact	0.039	1.582	0.022	2.71e-4
	KNN (\(K=3\))	0.026	1.036	0.017	2.25e-4
	DiffSound (Physics-based)	0.029	1.533	0.024	2.39e-4
	SonicGauss (Data-driven)	0.039	1.580	0.025	2.43e-4
	Ours (AV-MSF)	0.021	0.996	0.017	2.16e-4

Ablation Study¶

Ablation experiments on an ObjectFolder Real subset validate the necessity of each architectural component:

Config	L1 ↓	L1 Log ↓	ENV ↓	CDPAM ↓	Note
Full model (Ours)	0.019	0.927	0.015	2.00e-4	full model configuration
w/o visual	0.028	1.148	0.015	4.20e-4	removes DINOv2 features; perceptual score degrades sharply
w/o init	0.045	1.077	0.026	2.97e-4	removes parameter extraction and warm-up; traps in local minima
w/o align	0.019	1.002	0.016	2.62e-4	removes geometric symmetry alignment; degrades symmetric responses
w/o residual	0.031	5.764	0.026	3.22e-4	removes residual noise filter; log spectral loss severely spikes

Case studies and downstream evaluations further reveal: - Non-Symmetric Objects: On an asymmetric subset, KNN degrades significantly due to the absence of geometric repetition (L1: 0.0135, L1 Log: 0.9724), whereas AV-MSF consistently maintains superior accuracy (L1: 0.0110, L1 Log: 0.9259, CDPAM: 1.53e-4). - Contact Localization (RMED ↓): Inverting impact coordinates via cosine similarity between observed gains and the gain field yields an error of 34.61% for AV-MSF, significantly outperforming DiffSound (41.78%). - Sound Editing (UMAP Distance ↓): For semantic material transfer, direct generation yields 3.077 and standard Audio-SDS yields 4.234, while AV-MSF achieves 2.753.

Key Findings¶

Physics-visual synergy breaks few-shot barriers: In the sparse 20% training setting, AV-MSF achieves a 2× performance boost over SonicGauss (even after pretraining on ObjectFolder 2.0), showing that physical inductive biases prevent catastrophic generative hallucinations.
Residual noise decoupling is critical for gain fidelity: Omitting the noise residual causes the L1 Log error to surge from 0.927 to 5.764 and degrades localization error from 38.4% to 43.8%, demonstrating that unmodeled acoustic noise otherwise forces the spatial gain field to overfit environmental artifacts.
Visual semantics convey structural acoustics: Ablating DINOv2 embeddings doubles the CDPAM perceptual error, proving that high-level visual features encode subtle acoustic cues such as local curvature, edge boundaries, and material texture that raw 3D coordinates fail to convey.

Highlights & Insights¶

Decoupled modal field for sample efficiency: Separating global object-intrinsic vibration modes from a spatial continuous neural gain field drastically reduces optimization degrees of freedom, achieving dense surface coverage from sparse strikes.
Mode shape invariance for physical editing: Leveraging the mechanical invariance of modal excitation ratios under uniform material changes allows cross-material text-driven synthesis without retraining the spatial field.
Geometric symmetry pooling: Detecting and pooling features across rotational and planar reflection orbits resolves view-dependent lighting noise and ensures physically consistent sound rendering across symmetric regions.

Limitations & Future Work¶

Uniform material assumption: Current formulation assumes homogeneous material composition across the object; extending to multi-material assemblies (e.g., wooden hammers with metal heads) requires learning spatially varying modal frequencies and dampings.
Simplified contact mechanics: The model relies on calibrated force sensor profiles for normalization; integrating striker material properties and contact elastodynamics into the differentiable pipeline remains an open challenge.
High-frequency transient modeling: The modal synthesis formulation assumes linear decay from rest, leaving room for improved modeling of non-linear chaotic impacts, friction, and complex phase dynamics.

vs DiffSound (SIGGRAPH 2024): DiffSound performs differentiable inverse rendering over high-order volumetric FEM meshes, which is computationally prohibitive and prone to optimization collapse. AV-MSF bypasses FEM simulations by optimizing modal parameters and neural gain fields guided by 3DGS visual priors.
vs SonicGauss (2025): SonicGauss relies on diffusion-based cross-modal generation, requiring heavy training data and frequently generating physically inconsistent sounds. AV-MSF enforces linear modal dynamics to achieve superior physical fidelity with minimal data.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Elegant integration of 3D Gaussian Splatting, visual foundation model priors, and classical linear modal acoustics.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks across two real-world datasets, complete ablation suite, asymmetric case studies, and dual downstream validations.
Writing Quality: ⭐⭐⭐⭐⭐ Clear physical formulation, well-structured mathematical notation, and compelling visual-acoustic motivation.
Value: ⭐⭐⭐⭐⭐ Offers a practical, low-cost paradigm for digitizing interactive visual-tactile-acoustic assets for VR and embodied AI.