ECCV2024 ECCV2024 accepted papers ECCV2024 paper list AI paper notes top conference papers Image Generation Human Understanding 3D Reconstruction & Rendering Autonomous Driving Segmentation Video Understanding Multimodal VLM Image Restoration

🎞️ ECCV2024 Accepted Papers¶

869 ECCV2024 paper notes covering Image Generation (119), Human Understanding (70), 3D Reconstruction & Rendering (61), Autonomous Driving (61), Segmentation (54), Video Understanding (51), Multimodal VLM (46), Image Restoration (33) and other 46 areas. Each note has TL;DR, motivation, method, experiments, highlights, and limitations — 5-minute reads of core ideas.

💡 LLM Reasoning (1)¶

Controllable Navigation Instruction Generation with Chain of Thought Prompting: This paper proposes C-Instructor, which leverages chain-of-thought prompting of LLMs to achieve style- and content-controllable navigation instruction generation. Through three core mechanisms—Chain of Thought with Landmarks (CoTL), Spatial Topology Modeling Task (STMT), and Style-Mixed Training (SMT)—the method comprehensively outperforms existing approaches on four indoor and outdoor navigation datasets.

🦾 LLM Agent (3)¶

VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding: This paper proposes VideoAgent, a memory-augmented multimodal Agent. By constructing structured memory (temporal memory storing event descriptions and object memory storing object tracking states) and utilizing four tools to interact with the memory, it performs zero-shot long video QA tasks. It achieves an average gain of +6.6% on NExT-QA and +26.0% on EgoSchema, approaching the performance of Gemini 1.5 Pro.
Agent3D-Zero: An Agent for Zero-shot 3D Understanding: Agent3D-Zero proposes a VLM-based zero-shot 3D scene understanding agent framework. By utilizing Set-of-Line visual prompting on the bird's-eye view (BEV) to guide the VLM to actively select observation viewpoints and synthesizing multi-view images for 3D reasoning, it outperforms fine-tuned 3D-LLM methods on tasks like ScanQA.
HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoning: (Note: Brief note based on abstract) This paper proposes HYDRA, a multi-stage dynamic compositional visual reasoning framework. Through the collaboration of three modules—a Planner, a reinforcement learning cognitive controller (RL Agent), and a Reasoner—it achieves reliable and progressive visual reasoning, reaching SOTA performance on multiple datasets including RefCOCO/RefCOCO+, OK-VQA, and GQA.

🔒 Safety (19)¶

CLIP-Guided Generative Networks for Transferable Targeted Adversarial Attacks: This paper proposes CGNC, which leverages the CLIP text encoder to inject target-category semantic information into a conditional generative network. Combining cross-attention modules with masked fine-tuning, this method significantly improves the black-box transfer success rate of both multi-target and single-target directed adversarial attacks.
One-stage Prompt-based Continual Learning: This paper proposes the OS-Prompt framework. By directly utilizing the token embeddings of ViT intermediate layers as prompt queries (rather than relying on an extra query ViT forward pass), it reduces the computational-cost of prompt-based continual learning by approximately 50%. It further compensates for the loss in representation capacity with a Query-Pool Regularization (QR) loss, outperforming CodaPrompt by about 1.4% on CIFAR-100, ImageNet-R, and DomainNet.
Towards Multi-modal Transformers in Federated Learning: The FedCola framework is proposed, which utilizes complementary local training and collaborative aggregation to realize cross-modal knowledge transfer for multi-modal Transformers in federated learning, effectively bridging the gap between uni-modal and multi-modal clients without requiring public data.
MAGR: Manifold-Aligned Graph Regularization for Continual Action Quality Assessment: Proposes the MAGR method, which utilizes a manifold alignment projector and an Intra-Inter-Joint graph regularizer to address the misalignment between old and current feature manifolds caused by feature replay in Continual Action Quality Assessment (CAQA), significantly outperforming existing baselines across four datasets.
Noise-Assisted Prompt Learning for Image Forgery Detection and Localization: This paper proposes CLIP-IFDL, a CLIP-based image forgery detection and localization model. By employing instance-aware dual-stream prompt learning and a forgery-enhanced noise adapter, it addresses CLIP's lack of domain-specific prompts and forgery sensitivity in forgery detection, successfully transferring CLIP's open-world generalization capability to the forgery detection task.
Preventing Catastrophic Overfitting in Fast Adversarial Training: A Bi-level Optimization Perspective: Analyzes the causes of catastrophic overfitting in fast adversarial training from a bi-level optimization perspective, and proposes the FGSM-PCO method. By adaptively fusing historical and current adversarial examples along with custom regularization loss, it effectively prevents and corrects the collapse of inner optimization.
Bi-TTA: Bidirectional Test-Time Adapter for Remote Physiological Measurement: This paper proposes the Bi-TTA framework, which introduces Test-Time Adaptation to remote photoplethysmography (rPPG) tasks for the first time. By leveraging a spatiotemporal consistency self-supervised prior and a prospective-retrospective bidirectional adaptation strategy, the proposed method achieves model domain adaptation at test-time using only unlabeled single-instance data during inference.
Fisher Calibration for Backdoor-Robust Heterogeneous Federated Learning: This paper proposes Self-Driven Fisher Calibration (SDFC), which utilizes Fisher information to measure differences in parameter importance across different distributions. SDFC effectively distinguishes malicious backdoor clients and performs parameter calibration in heterogeneous federated learning scenarios, overcoming the limitations of existing defense methods that rely on data homogeneity and minority malicious node assumptions.
Any Target Can Be Offense: Adversarial Example Generation via Generalized Latent Infection: GAKer is proposed as the first targeted adversarial attack generator that generalizes to unseen target classes. By injecting target features (latent infection) into the intermediate layers of a UNet and employing a class-agnostic cosine distance loss instead of cross-entropy, it outperforms HGN on unseen classes by 14.13% in attack success rate.
SkyMask: Attack-Agnostic Robust Federated Learning with Fine-Grained Learnable Masks: This paper proposes SkyMask, which utilizes parameter-level learnable binary masks at the server side to detect malicious client model updates, achieving attack-agnostic robust federated learning that remains resilient even when up to 80% of clients are malicious.

Browse all 19 Safety papers →

👻 Hallucination Detection (2)¶

BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-Language Models: BEAF proposes a "before-after comparison" hallucination evaluation paradigm: by observing changes in VLM responses after removing objects through image editing and introducing four change-aware metrics (TU/IG/SB/ID), it reveals hallucination behaviors that cannot be detected by traditional text-axis evaluations.
LiDAR-Event Stereo Fusion with Hallucinations: This paper proposes the first framework to fuse sparse LiDAR depth points with event stereo cameras. By "hallucinating" (inserting fictitious events) within the event stack representations (VSH) or the raw event stream (BTH), the framework compensates for the missing information of event cameras in motion-free or textureless regions, significantly improving event stereo matching accuracy.

📊 LLM Evaluation (19)¶

Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrieval: A Slerp-based ZS-CIR method is proposed, which directly fuses the image and text embeddings of VLP models via Spherical Linear Interpolation (Slerp) to construct composed query representations. Combined with Text-Anchored-Tuning (TAT), which fine-tunes the image encoder using LoRA to narrow the modality gap, this method achieves state-of-the-art (SOTA) performance on CIRR, CIRCO, and FashionIQ.
OGNI-DC: Robust Depth Completion with Optimization-Guided Neural Iterations: This paper proposes OGNI-DC, which achieves both SOTA accuracy and strong generalization in depth completion through an "Optimization-Guided Neural Iteration" (OGNI) framework, combining a ConvGRU for iterative depth gradient field refinement with a Differentiable Depth Integrator (DDI).
PetFace: A Large-Scale Dataset and Benchmark for Animal Identification: A large-scale animal face recognition dataset, PetFace, is constructed, containing 13 animal families, 319 breeds, and 257,484 individuals (over 1 million images). Two benchmark tests are established: seen individual re-identification (Re-ID) and unseen individual verification, providing infrastructure for non-invasive automatic animal identification.
SIGMA: Sinkhorn-Guided Masked Video Modeling: This paper proposes SIGMA, which upgrades the reconstruction target of masked video modeling from the pixel level to learnable deep feature cluster assignments by introducing a projection network. Employing optimal transport with the Sinkhorn algorithm to enforce high-entropy regularization, SIGMA avoids representation collapse and comprehensively outperforms state-of-the-art methods like VideoMAE across 10 datasets and 3 benchmarks.
ColorMNet: A Memory-based Deep Spatial-Temporal Feature Propagation Network for Video Colorization: This paper proposes ColorMNet, a memory-based deep spatial-temporal feature propagation network. By incorporating three components—Pre-trained Vision-Guided Feature Extraction (PVGFE), Memory-based Feature Propagation (MFP), and Local Attention (LA)—this method achieves video colorization performance superior to state-of-the-art (SOTA) models while significantly reducing GPU memory consumption to only 1.9 GB.
Distribution Alignment for Fully Test-Time Adaptation with Dynamic Online Data Streams: A Distribution Alignment (DA) loss is proposed to pull the test-time feature distribution back to the source domain distribution. Combined with a domain shift detection mechanism, this method significantly outperforms existing TTA methods under non-i.i.d. dynamic data streams and continuous domain shift scenarios.
Gradient-Regularized Out-of-Distribution Detection: This paper proposes GReg/GReg+, which learns the local smoothness of the scoring manifold by regularizing the input gradient norm of the OOD scoring function, and incorporates an energy-score-based clustering sampling strategy to select highly informative auxiliary samples, achieving SOTA on CIFAR and ImageNet OOD detection benchmarks.
Eliminating Warping Shakes for Unsupervised Online Video Stitching: Defines a new problem in video stitching termed "warping shake" (temporal shaking in non-overlapping regions when extending image stitching to video). This work proposes StabStitch, the first unsupervised online video stitching framework. By simultaneously generating and smoothing stitching trajectories, it achieves both video stitching and stabilization, reaching a real-time speed of 28.2 ms/frame.
Learn from the Learnt: Source-Free Active Domain Adaptation via Contrastive Sampling and Visual Persistence: The LFTL (Learn from the Learnt) framework is proposed. Consisting of two core modules—Contrastive Active Sampling (CAS) and Visual Persistence-guided Adaptation (VPA)—it achieves highly efficient domain adaptation under source-free and extremely low target annotation budgets (\(\le 5\%\)), reaching 87.4% accuracy on VisDA-C with only 1% annotation.
EvSign: Sign Language Recognition and Translation with Streaming Events: This work constructs the first event-camera benchmark dataset, EvSign, for Continuous Sign Language Recognition (CSLR) and Sign Language Translation (SLT) tasks, and proposes an efficient sparse Transformer-based framework that achieves comparable or superior performance to SOTA RGB methods using only 0.34% FLOPs and 44.2% of the parameters.

Browse all 19 LLM Evaluation papers →

📚 Pretraining (8)¶

Prompting Language-Informed Distribution for Compositional Zero-Shot Learning: This paper proposes the PLID method, which leverages sentence-level category descriptions generated by LLMs to construct language-knowledge-driven Gaussian distributions. Combined with vision-language primitive decomposition and randomized logit fusion, it achieves state-of-the-art (SOTA) performance on the Compositional Zero-Shot Learning (CZSL) task.
DragAPart: Learning a Part-Level Motion Prior for Articulated Objects: DragAPart proposes an image generator that uses dragging as an interactive interface, capable of responding to part-level interactions (such as opening/closing drawers/doors) rather than merely moving the entire object. Through the new synthetic dataset Drag-a-Move, multi-resolution drag encoding, and domain randomization strategies, the model generalizes well to real images and unseen categories despite being trained solely on synthetic data.
Plan, Posture and Go: Towards Open-Vocabulary Text-to-Motion Generation: This paper proposes PRO-Motion, a divide-and-conquer framework that decomposes text-to-motion generation into three stages: LLM-driven motion planning (Plan), script-based posture diffusion generation (Posture), and global translation and rotation estimation (Go). By reducing the complexity of each stage, it achieves high-quality open-vocabulary motion generation.
I Can't Believe It's Not Scene Flow!: Reveals that the catastrophic failure of existing scene flow methods on small objects like pedestrians is masked by current evaluation metrics, and proposes a category-aware and velocity-normalized Bucket Normalized EPE evaluation protocol, alongside a simple yet SOTA baseline, TrackFlow (generating scene flow from a detector + tracker), achieving a 1.5x improvement in pedestrian motion description.
Scaling Backwards: Minimal Synthetic Pre-training?: Proposes 1p-frac—achieving pre-training performance comparable to the ImageNet-1k level using minute perturbations of a single fractal image. This challenges the conventional wisdom that "pre-training requires large-scale datasets" and reveals that the essence of pre-training might be closer to weight initialization than visual concept learning.
Cross-Domain Learning for Video Anomaly Detection with Limited Supervision: Proposes a weakly-supervised cross-domain learning (CDL) framework that integrates unlabeled external videos into training via an uncertainty-driven pseudo-labeling mechanism, significantly improving the cross-domain generalization capability of video anomaly detection.
PreLAR: World Model Pre-training with Learnable Action Representation: This paper proposes PreLAR to bridge the gap between action-free pre-training and action-conditioned fine-tuning for world models. By encoding implicit action representations from adjacent frames and designing an action-state consistency loss during unsupervised pre-training on action-free videos, PreLAR significantly improves the sample efficiency of downstream visual control tasks.
Learning to Obstruct Few-Shot Image Classification over Restricted Classes: The Learning to Obstruct (LTO) algorithm is proposed, which modifies pre-trained backbone parameters via a MAML-like meta-learning approach to make them a "bad initialization" for specific restricted classes. This hinders the fine-tuning performance of few-shot classification methods on restricted classes while maintaining normal performance on other classes.

💬 LLM (Other) (11)¶

AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detection: By concurrently incorporating static (globally shared) and dynamic (instance-specifically generated) learnable prompts into CLIP, and using auxiliary anomaly detection data for optimization, this method establishes a zero-shot SOTA on 14 industrial and medical anomaly detection datasets. The core innovation lies in the hybrid prompt design that achieves dual-tier adaptation at both the "task" and "instance" levels.
FunQA: Towards Surprising Video Comprehension: The authors construct a large-scale counter-intuitive video question answering benchmark, FunQA (consisting of 4.3K videos and 312K QA pairs), covering three categories of surprising videos: Humor, Creativity, and Magic. They also propose the FunMentor agent, which enhances the counter-intuitive reasoning capabilities of VLMs through multi-turn dialogue.
Zero-Shot Object Counting with Good Exemplars (VA-Count): Proposes VA-Count, a vision-association-based zero-shot object counting framework. It establishes robust visual associations between high-quality exemplars and images for arbitrary categories through a Grounding DINO-driven exemplar enhancement module and a contrastive learning noise suppression module.
PromptIQA: Boosting the Performance and Generalization for No-Reference Image Quality Assessment via Prompts: The authors propose PromptIQA, which uses a small number of "image-score pairs" (ISPs) as prompts. This allows the trained NR-IQA model to adapt to new quality assessment requirements without fine-tuning, achieving SOTA performance and generalization capabilities across 12 datasets and 5 categories of IQA tasks.
FreestyleRet: Retrieving Images from Style-Diversified Queries: This work proposes the first Style-Diversified Query-Based Image Retrieval (Style-Diversified QBIR) task and the DSR dataset. It designs FreestyleRet, a lightweight, plug-and-play framework that extracts texture/style features of queries using Gram matrices to construct a style space. These style features then initialize prompt tokens, enabling a frozen vision encoder to adapt to various query styles such as texts, sketches, low-resolution images, and artistic paintings.
RoadPainter: Points Are Ideal Navigators for Topology Transformer: RoadPainter is proposed, which adopts a two-stage strategy of first regressing lane centerline points and then refining them using instance masks. Combined with a hybrid attention mechanism and a real-virtual lane separation strategy, it achieves SOTA topology inference performance on the OpenLane-V2 dataset.
Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videos: VidAssist proposes a three-step framework of "Propose-Assess-Search", leveraging LLMs as a knowledge base and evaluation tool combined with a breadth-first search algorithm. It outperforms fully supervised SOTA in a zero/few-shot manner on goal-oriented planning tasks in instructional videos, achieving a +7.7% SR improvement on COIN compared to the fully supervised VLaMP in the few-shot setting.
Cultural Value Differences of LLMs: Prompt, Language, and Model Size: This paper systematically investigates the behavioral patterns of LLMs in expressing cultural values utilizing the Hofstede cultural dimensions questionnaire. It finds that prompt language (Chinese vs. English) and model size have a far greater impact on cultural value disparities than differences in model architecture and question order.
APL: Anchor-based Prompt Learning for One-stage Weakly Supervised Referring Expression Comprehension: This paper proposes APL (Anchor-based Prompt Learning), which designs an Anchor-based Prompt Encoder (APE) to generate distinctive prompts across three categories: location, color, and category. By dynamically integrating these prompts into anchor features to enrich visual semantics, alongside text reconstruction and visual alignment losses, APL achieves precise vision-language alignment. It outperforms existing weakly supervised methods on four REC benchmarks (e.g., exceeding RefCLIP by 6.44% on RefCOCO).
Reprojection Errors as Prompts for Efficient Scene Coordinate Regression: This paper proposes the Error-Guided Feature Selection (EGFS) mechanism, which leverages low reprojection error regions as point prompts for SAM to expand into semantic masks. By iteratively filtering reliable training samples, the method outperforms existing 3D-free SCR methods on the Cambridge Landmarks and Indoor6 datasets with a smaller model size and less training time.

Browse all 11 LLM (Other) papers →

📖 NLP Understanding (1)¶

SLIMER: Show Less, Instruct More - Enriching Prompts with Definitions and Guidelines for Zero-Shot NER: SLIMER enhances the zero-shot named entity recognition capability of LLMs by injecting entity definitions and annotation guidelines into prompts. Trained on only 391 entity categories, it achieves performance comparable to State-of-the-Art (SOTA) methods trained on 13,000+ entity categories when evaluated on unseen entity tags.

🗣️ Dialogue Systems (1)¶

BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generation: This paper proposes the BI-MDRG framework, which bridges image history information to enhance the image-grounding capability of textual responses and the object consistency in sequential image responses in multimodal dialogues.

🔍 Information Retrieval & RAG (3)¶

OneRestore: A Universal Restoration Framework for Composite Degradation: OneRestore is proposed as a Transformer-based universal image restoration framework. Driven by a scene-descriptor-guided cross-attention mechanism and a composite degradation restoration loss, it adaptively handles low-light, haze, rain, snow, and their arbitrary composite combinations within a single model, supporting controllable restoration under both text and visual modes.
Multi-Label Cluster Discrimination for Visual Representation Learning: This work proposes MLCD (Multi-Label Cluster Discrimination), which assigns multiple cluster pseudo-labels to each image and designs a disambiguated multi-label classification loss. Pre-trained on LAION-400M, the ViT model under MLCD comprehensively outperforms OpenCLIP, FLIP, and UNICOM in linear probe, zero-shot classification, and retrieval tasks.
Towards Open-Ended Visual Recognition with Large Language Model: This paper proposes the OmniScient Model (OSM)—a generative mask classifier based on a frozen CLIP-ViT, a trainable MaskQ-Former, and a frozen LLM (Vicuna-7B). It shifts visual recognition from "selecting categories from a predefined vocabulary" to "directly generating category names," eliminating the dependency on predefined vocabularies during both training and testing. It outperforms DaTaSeg by +4.3 PQ on COCO panoptic segmentation.

💻 Code Intelligence (1)¶

DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation: Proposes using code generation to synthesize structured visual data (slides and UIs) to train understanding models, thereby reducing the need for manual annotation.

🎨 Image Generation (119)¶

Pixel-Aware Stable Diffusion for Realistic Image Super-Resolution and Personalized Stylization: This paper proposes PASD (Pixel-Aware Stable Diffusion), which enables the diffusion model to perceive local image structures at the pixel level through a Pixel-Aware Cross Attention (PACA) module. Combined with a degradation removal module and an adjustable noise schedule, it achieves a unified framework for realistic image super-resolution and personalized stylization, where the style can be switched simply by replacing the base model.
ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs: ZipLoRA proposes a cheap and efficient LoRA merging method. By learning column-wise merging coefficients and minimizing the cosine similarity between columns, it achieves hyperparameter-free merging of independently trained subject LoRAs and style LoRAs, generating personalized "any subject × any style" images in diffusion models.
MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model: Proposes MotionLCM, which first introduces consistency distillation into human motion generation, achieving real-time motion generation (~30ms/sequence) via single-step/few-step inference in the motion latent space, and realizes real-time controllable motion generation in the latent space through Motion ControlNet.
Implicit Style-Content Separation using B-LoRA: This paper proposes B-LoRA. By analyzing the SDXL architecture, it is discovered that implicitly separating style and content of a single image can be achieved by jointly training the LoRA weights of only two specific transformer blocks (Block 4 controls content, and Block 5 controls style), supporting various tasks such as style transfer, text-based stylization, and consistent style generation.
ZigMa: A DiT-style Zigzag Mamba Diffusion Model: ZigMa proposes a DiT-style Zigzag Mamba diffusion model. By employing a heterogeneous layer-wise zigzag scanning scheme, it maintains spatial continuity, achieving superior generation quality compared to Mamba baselines with zero parameter or memory overhead, while retaining the linear complexity advantage over Transformers.
TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering: TextDiffuser-2 utilizes two language models for layout planning and layout encoding respectively, achieving more flexible, automated, and diverse visual text rendering, significantly enhancing font style diversity while maintaining text accuracy.
DiffiT: Diffusion Vision Transformers for Image Generation: DiffiT (Diffusion Vision Transformer) is proposed, which introduces a Time-dependent Multi-head Self-Attention (TMSA) mechanism to dynamically adjust self-attention behaviors at different stages of the denoising process, achieving a state-of-the-art (SOTA) FID score of 1.73 on ImageNet-256 with 16-20% fewer parameters than DiT/MDT.
ReNoise: Real Image Inversion Through Iterative Noising: Proposed the ReNoise iterative renoising method to improve the image inversion quality of diffusion models. By applying the UNet multiple times at each inversion timestep and averaging the predictions, it improves trajectory estimation accuracy, which is particularly effective for few-step diffusion models (such as SDXL Turbo and LCM).
RingID: Rethinking Tree-Ring Watermarking for Enhanced Multi-Key Identification: This paper deeply analyzes the source of robustness of the Tree-Ring watermarking method (discovering that distribution shift is an unexpected hidden helper in its verification task), reveals its severe limitations in multi-key identification tasks, and proposes RingID—a multi-channel heterogeneous watermarking framework. Through discretization, lossless embedding, and a more circular ring design, RingID improves the identification accuracy for 2048 keys from 0.07 to 0.82.
DreamDiffusion: High-Quality EEG-to-Image Generation with Temporal Masked Signal Modeling and CLIP Alignment: This paper proposes DreamDiffusion, which leverages temporal masked signal modeling for large-scale pre-training of an EEG encoder to learn robust brainwave representations. It then aligns the EEG-text-image space using additional supervision from a CLIP image encoder, and finally utilizes a pre-trained Stable Diffusion model to generate high-quality images directly from EEG signals, achieving portable and low-cost "thoughts-to-image" generation.

Browse all 119 Image Generation papers →

🎬 Video Generation (15)¶

Evaluating Text-to-Visual Generation with Image-to-Text Generation: The authors propose VQAScore, which uses Visual Question Answering (VQA) models instead of CLIP to evaluate text-to-visual generation quality. It significantly outperforms CLIPScore on complex compositional prompts and releases the GenAI-Bench benchmark.
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion: SV3D is proposed to adapt image-to-video diffusion models for multi-view synthesis and 3D generation, leveraging the generalization capability and multi-view consistency of video models while introducing explicit camera control.
PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation: Estimates spatially-varying Young's modulus material fields for static 3D Gaussian objects by leveraging physical dynamics priors implicit in video generation models, enabling physically plausible interactive 3D dynamics synthesis.
DragAnything: Motion Control for Anything using Entity Representation: This paper proposes DragAnything, which utilizes the latent space features of diffusion models as Entity Representations to achieve entity-level motion control. It addresses the issue of existing trajectory-driven methods only dragging pixels without being able to precisely control the motion of target objects. DragAnything achieves state-of-the-art (SOTA) FVD/FID metrics on VIPSeg, outperforming DragNUWA by 26% in motion control votes in a user study.
FreeInit: Bridging Initialization Gap in Video Diffusion Models: This work identifies a training-inference initialization discrepancy in video diffusion models (where low-frequency information leakage during training leads to temporally correlated initial noise, whereas uncorrelated Gaussian noise is used during inference). It proposes FreeInit, which bridges this gap by iteratively refining the spatiotemporal low-frequency components of the initial noise, thereby significantly improving the temporal consistency of generated videos.
MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model: This paper proposes MOFA-Video, which equips a frozen image-to-video diffusion model (SVD) with controllable motion capabilities by designing multiple domain-specific motion field adapters (MOFA-Adapters). It supports various control signals and their combinations, such as hand-drawn trajectories and facial landmarks, to achieve open-domain controllable image animation.
VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models: This work proposes utilizing a pre-trained video diffusion model (EMU Video) as a multi-view data engine. By fine-tuning it to generate 3-D consistent multi-view videos, the authors construct approximately 3 million synthetic data points to train a feedforward 3D generative model, VFusion3D. This enables generating 3D assets from a single image in seconds, achieving a user preference rate of over 90%.
RealViformer: Investigating Attention for Real-World Video Super-Resolution: This paper systematically investigates the behavioral differences between spatial and channel attention in real-world video super-resolution (RWVSR). It is found that channel attention is more robust to degradation artifacts but leads to feature redundancy. Based on this, RealViformer is proposed with Improved Channel Attention (ICA) and Channel Attention Fusion (CAF) modules, achieving SOTA performance with fewer parameters and faster speed.
DreamMotion: Space-Time Self-Similar Score Distillation for Zero-Shot Video Editing: DreamMotion, a zero-shot video editing framework based on Score Distillation, is proposed. By employing space-time self-similarity regularization, it injects target appearances while preserving the structural and motion integrity of the original video, applicable to both cascaded and non-cascaded video diffusion models.
Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation: This paper is the first to explore the visual features of pre-trained text-to-video (T2V) diffusion models for video understanding tasks. It proposes the VD-IT framework, which extracts visual features with superior temporal-semantic consistency from a frozen T2V diffusion model using two key designs: text-guided image projection and video-specific noise prediction. VD-IT outperforms state-of-the-art methods using discriminatively pre-trained video backbones (such as Video Swin Transformer) across four major R-VOS benchmarks.

Browse all 15 Video Generation papers →

🧩 Multimodal VLM (46)¶

MMBench: Is Your Multi-modal Model an All-Around Player?: Proposes MMBench—a bilingual (English/Chinese) multimodal benchmark comprising 3,217 multiple-choice questions across 20 fine-grained ability dimensions, featuring a CircularEval evaluation strategy and an LLM-based choice extraction mechanism to significantly improve evaluation robustness and fairness.
ShareGPT4V: Improving Large Multi-Modal Models with Better Captions: ShareGPT4V constructs a high-quality descriptive caption dataset comprising 1.2M entries (seed of 100K generated by GPT4-Vision + expanded to 1.2M via Share-Captioner). By using this dataset to train ShareGPT4V-7B (a model based on the LLaVA architecture) in both pre-training and SFT stages, it achieves state-of-the-art performance on 9 out of 11 multi-modal benchmarks. This demonstrates that high-quality captions are the key bottleneck in multi-modal alignment for LMMs.
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?: This paper introduces MathVerse, a multimodal mathematical reasoning benchmark containing 2,612 visual math problems (transformed into 6 versions totaling 15K test samples). By systematically manipulating the allocation of information in text and images, MathVerse assesses whether MLLMs truly "understand" mathematical diagrams. The authors also propose a CoT evaluation strategy for fine-grained reasoning process scoring, revealing that most MLLMs rely heavily on text rather than visual diagrams for mathematical reasoning.
BLINK: Multimodal Large Language Models Can See but Not Perceive: Introduces BLINK—a multimodal evaluation benchmark containing 14 classic computer vision perception tasks (3,807 multiple-choice questions) that humans can solve "in a blink" (95.7% accuracy), but the strongest GPT-4V achieves only 51.26% (only 13.17% above random guessing), revealing a severe deficiency of current MLLMs in core visual perception capabilities.
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training: Apple systematically ablates the three primary axes of MLLM construction (architecture, data, and training), deriving key design principles: Image Resolution > Model Size > Training Data; the choice of VL connector type has minimal impact; and the meticulous blending of caption, interleaved, and text-only data is crucial. This systematically constructed MM1 model family (ranging from 3B-30B dense models to up to 64B MoE models) achieves state-of-the-art performance in few-shot pre-training evaluations.
AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting: The AdaShield framework is proposed, which comprises a meticulously designed static defense prompt (AdaShield-S) and an LLM-based adaptive iterative optimization framework (AdaShield-A). Without fine-tuning MLLMs or training additional modules, it effectively defends against structure-based jailbreak attacks, reducing the attack success rate from over 75% to below 15% while maintaining normal task performance.
Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation: ECSO (Eyes Closed, Safety On) is proposed: a training-free MLLM defense method that detects the safety of its own responses and adaptively converts images in unsafe queries into text descriptions, thereby restoring the intrinsic safety mechanism of pre-aligned LLMs. It achieves up to a 71.3% safety improvement on MM-SafetyBench without compromising general performance.
Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild: Nymeria is currently the world's largest in-the-wild human motion dataset (300 hours, 264 participants), providing synchronized and co-localized multi-device multimodal egocentric data (Project Aria glasses + wristbands + motion capture suits) for the first time, accompanied by 310.5K hierarchical motion-language descriptions.
Towards Open-ended Visual Quality Comparison: This work proposes Co-Instruct, the first large multimodal model for open-ended visual quality comparison. By constructing a 562K instruction-tuning dataset from two "weakly supervised sources" (LLM-merged single-image descriptions + GPT-4V pseudo-labels), Co-Instruct achieves higher accuracy in multi-image quality comparison than its teacher model, GPT-4V, and introduces MICBench, the first multi-image comparison benchmark.
Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models: This paper proposes the Vary method, which scales up the vision vocabulary of Large Vision-Language Models (LVLMs) by generating and integrating a new vision vocabulary. This empowers the model with new fine-grained visual perception capabilities, such as document-level OCR and chart understanding, while maintaining its original general capabilities.

Browse all 46 Multimodal VLM papers →

🧠 VLM Reasoning (1)¶

NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models: NavGPT-2 closes the performance gap between LM-based agents and VLN-specific models while retaining the LLM's interpretable navigational reasoning capabilities by feeding the hidden layer representations of a frozen LLM into a topological map navigation policy network as vision-language features, showcasing excellent data efficiency.

⚡ VLM Efficiency (4)¶

Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models: Groma proposes a new paradigm that embeds localization capabilities directly into the visual tokenization process. By discovering regions of interest (ROIs) via a region proposer and encoding them into region tokens, Groma enables MLLMs to perform high-accuracy referring and grounding without relying on LLM-generated coordinates or external modules. It also leverages GPT-4V with visual prompting to construct Groma Instruct, the first grounded chat dataset featuring dual visual-textual prompts.
IVTP: Instruction-Guided Visual Token Pruning for Large Vision-Language Models: IVTP proposes utilizing textual instruction information to dynamically assess the importance of each visual token and prune redundant tokens during the inference of Large Vision-Language Models (LVLMs). This achieves task-related adaptive visual info compression, significantly reducing computational overhead while maintaining or even improving model performance.
Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understanding: This work proposes ClassAct/ActiveCLIP, which utilizes small, low-cost proxy models to compute "learnability" scores for data points to prioritize training data. This reduces training updates for large-scale visual classifiers and multimodal models by 46% and 51% respectively, achieves up to 25% total compute savings, and stands as the first active learning method to achieve net positive compute savings in large-scale pre-training.
Quantized Prompt for Efficient Generalization of Vision-Language Models: By treating quantization error as a form of regularization noise, this work applies ultra-low-bit quantization (down to 1-bit) to the learnable prompts of VLMs. This significantly reduces storage overhead (up to \(16\times\) compression) while markedly improving the model's generalization capability to unseen classes. QCoOp achieves superior performance over various state-of-the-art (SOTA) methods using only 0.26KB of storage.

🎵 Audio & Speech (8)¶

ControlLLM: Augment Language Models with Tools by Searching on Graphs: This paper proposes the ControlLLM framework, which plans multimodal tool execution by performing graph search (Thoughts-on-Graph) on a pre-built Tool Graph. This significantly improves the accuracy of tool selection and parameter assignment in complex tasks.
Label-Anticipated Event Disentanglement for Audio-Visual Video Parsing: This paper proposes the LEAP (Label semantic-based Projection) decoding paradigm, which utilizes the text embeddings of event categories as semantic anchors. Using a cross-modal attention mechanism, potentially overlapping event semantics within audio/visual latent features are disentangled into independent label embeddings. Combined with an EIoU-based audio-visual semantic similarity loss, LEAP achieves SOTA performance on the AVVP task.
Beat-It: Beat-Synchronized Multi-Condition 3D Dance Generation: The Beat-It framework is proposed to achieve beat-synchronized and keyframe-controllable 3D dance generation by decoupling beat conditions from music and designing a hierarchical multi-condition fusion mechanism, significantly outperforming existing methods on AIST++.
Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos: AV-LDM is proposed to implicitly disentangle foreground action sounds and background ambient sounds by introducing audio from different time segments of the same video as an ambient sound condition during training. Combined with retrieval-augmented generation (RAG) to select appropriate ambient sound conditions during inference, it significantly outperforms existing methods on Ego4D and EPIC-KITCHENS.
Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation: This paper proposes CSTS (Contrastive Spatial-Temporal Separable), an audio-visual fusion method that introduces audio signals to egocentric gaze anticipation for the first time. It models spatial co-occurrence and temporal correlation of audio-visual signals separately through spatial and temporal separable fusion modules, and enhances representations using post-fusion contrastive learning, surpassing SOTA on Ego4D and Aria datasets.
Siamese Vision Transformers are Scalable Audio-Visual Learners: The AVSiam framework is proposed, which uses a single weight-shared ViT backbone to simultaneously process both audio and visual inputs. Combined with a multi-ratio random masking strategy and a dual-objective pre-training scheme (contrastive plus reconstruction), AVSiam achieves state-of-the-art (SOTA) performance on audio-visual classification and retrieval at an extremely low cost (28.9 times faster than MAViL).
CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing: This work proposes the CoLeaF dual-branch learning framework, which explicitly optimizes the integration of cross-modal context through event-aware contrastive learning, achieving an average improvement of 1.9% F-score on the weakly supervised audio-visual video parsing task.
Latent-INR: A Flexible Framework for Implicit Representations of Videos with Discriminative Semantics: This paper proposes the Latent-INR framework. By learning an implicit latent code for each video frame and combining it with a hypernetwork for low-rank weight modulation, the framework decouples the spatial and temporal modeling of video INR. While maintaining competitive compression performance, it equips video representations with semantic discriminative capabilities, supporting various downstream tasks such as retrieval, video frame interpolation, and arbitrary-resolution inference.

🧊 3D Reconstruction & Rendering (61)¶

3D Reconstruction of Objects in Hands without Real World 3D Supervision: This paper proposes the HORSE framework, which trains an occupancy network to reconstruct the 3D shape of hand-held objects from a single RGB image. This is achieved by extracting multi-view 2D mask supervision from in-the-wild videos (using hand pose as an object pose proxy) and learning a 2D slice adversarial shape prior from a synthetic 3D shape collection. Without using any real-world 3D annotations, it outperforms 3D-supervised methods by 11.6% on the MOW dataset.
3DEgo: 3D Editing on the Go!: 3DEgo compresses the traditional three-stage 3D editing pipeline (COLMAP pose estimation \(\rightarrow\) unedited scene initialization \(\rightarrow\) iterative editing and update) into a single-stage framework: first performing multi-view consistent 2D editing on video frames using an autoregressive noise blending module, and then directly reconstructing the 3D scene from the edited frames using COLMAP-free 3DGS, boosting the speed by approximately 10x and supporting videos from arbitrary sources.
3iGS: Factorised Tensorial Illumination for 3D Gaussian Splatting: 3iGS replaces the independently optimized spherical harmonics (SH) coefficients of each Gaussian in 3DGS with a continuous incident illumination field based on tensor decomposition. Combined with learnable BRDF features and a lightweight neural renderer to model the outgoing radiance, it significantly improves the rendering quality of view-dependent effects such as specular reflections while maintaining real-time rendering speeds.
A Compact Dynamic 3D Gaussian Representation for Real-Time Dynamic View Synthesis: This work models the position and rotation parameters in 3DGS as continuous functions of time (Fourier approximation for position, linear approximation for rotation), reducing the storage complexity of dynamic scenes from \(O(TN)\) to \(O(LN)\). It achieves rendering quality comparable to NeRF-based methods on the D-NeRF, DyNeRF, and HyperNeRF datasets while maintaining real-time rendering speeds over 118 FPS.
A Probability-guided Sampler for Neural Implicit Surface Rendering: This paper proposes a probability-guided ray sampler (Probability-guided Sampler) that models a probability density function in a 3D image projection space to guide ray sampling toward regions of interest. Simultaneously, a novel surface reconstruction loss comprising near-to-surface and empty space components is designed. This sampler can be integrated as a plug-and-play module into existing neural implicit surface renderers, significantly improving reconstruction accuracy and rendering quality.
Analysis-by-Synthesis Transformer for Single-View 3D Reconstruction: Proposes the Analysis-by-Synthesis Transformer (AST), which models pixel-to-shape and pixel-to-texture relationships using Shape Transformer and Texture Transformer in a unified framework, achieving high-quality mesh reconstruction and texture generation with only 2D annotations, outperforming existing methods on CUB-200-2011 and ShapeNet.
Analytic-Splatting: Anti-Aliased 3D Gaussian Splatting via Analytic Integration: By analytically approximating the integration of Gaussian signals over pixel windows using a conditional logistic function, instead of pixel-center point sampling in 3DGS, alias-free 3D Gaussian Splatting is achieved, outperforming Mip-Splatting in multi-scale rendering.
BAD-Gaussians: Bundle Adjusted Deblur Gaussian Splatting: This work introduces a physical motion blur imaging model into the 3D Gaussian Splatting framework, jointly optimizing scene Gaussian parameters and camera trajectories during exposure to restore sharp 3D scenes from blurry images and achieve real-time rendering.
BeNeRF: Neural Radiance Fields from a Single Blurry Image and Event Stream: BeNeRF is proposed to jointly recover a neural radiance field and camera motion trajectory from only a single blurry image and its corresponding event stream. High-quality deblurring and novel view synthesis are achieved without requiring multi-view inputs or known poses.
CaesarNeRF: Calibrated Semantic Representation for Few-Shot Generalizable Neural Rendering: This paper proposes CaesarNeRF, which introduces scene-level semantic representations built upon generalizable NeRF (such as GNT). By leveraging camera pose calibration (aligning feature rotations with the target view) and sequential refinement (gradually updating global features across Transformer layers), CaesarNeRF improves PSNR by 1.74dB (on LLFF) compared to GNT in a 1-view setup, and seamlessly enhances other baselines like IBRNet and MatchNeRF.

Browse all 61 3D Reconstruction & Rendering papers →

🧊 3D Generation & Editing (26)¶

AEDNet: Adaptive Embedding and Multiview-Aware Disentanglement for Point Cloud Completion: AEDNet is proposed, which conducts global embedding and local disentanglement of point clouds in the encoder and decoder respectively through the Adaptive Embedding and Multiview-Aware Disentanglement (AED) module. By utilizing 3D viewpoints generated from a unit sphere to observe the point cloud from the outside, it achieves a comprehensive understanding of 3D object geometry, reaching SOTA on the MVP and PCN datasets.
AnimatableDreamer: Text-Guided Non-rigid 3D Model Generation and Reconstruction with Canonical Score Distillation: This work proposes AnimatableDreamer, which extracts skeletons and motion from monocular videos and generates text-guided animatable 3D non-rigid models via Canonical Score Distillation (CSD), comprehensively outperforming existing methods in both generation quality and temporal consistency.
ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidance: This paper proposes ComboVerse, a compositional 3D asset generation framework. It first decomposes an input image containing multiple objects into individual elements and reconstructs them independently as single-object 3D models. Then, it optimizes the position, scale, and rotation parameters of the objects guided by Spatially-Aware Score Distillation Sampling (SSDS), enabling high-quality multi-object compositional 3D asset creation. It significantly outperforms existing methods in both CLIP Score and human evaluation.
Compress3D: a Compressed Latent Space for 3D Generation from a Single Image: This paper proposes a highly compressed triplane latent space autoencoder, paired with a two-stage diffusion model (generating a shape embedding first, followed by a triplane latent). It generates high-quality 3D assets from a single image in just 7 seconds, utilizing significantly less training data and time than comparable methods.
CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model: This paper proposes CRM (Convolutional Reconstruction Model), which leverages the spatial alignment prior between triplanes and six orthographic views. It replaces the Transformer with a U-Net to directly map the six views to a triplane, and utilizes FlexiCubes for end-to-end training. CRM generates high-fidelity textured meshes from a single image in under 10 seconds, with only 1/8 of the training cost of LRM.
DreamDissector: Learning Disentangled Text-to-3D Generation from 2D Diffusion Priors: The DreamDissector framework is proposed to disentangle text-to-3D NeRFs containing multi-object interactions into independent textured meshes using a Neural Category Field and Deep Concept Mining, achieving object-level 3D editing control.
DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting: Proposes DreamScene360, which utilizes panoramic images as an intermediate representation, combined with a GPT-4V self-refinement mechanism and panoramic 3D Gaussian Splatting, to achieve rapid generation of immersive 360° 3D scenes from text.
DreamView: Injecting View-specific Text Guidance into Text-to-3D Generation: DreamView proposes an adaptive text guidance injection module to collaboratively inject view-specific and global text descriptions into a diffusion model, achieving customizable and multi-view consistent text-to-3D generation.
External Knowledge Enhanced 3D Scene Generation from Sketch: Proposes the SEK framework, which integrates freehand sketches and an external object relation knowledge base as conditions for a diffusion model. Through knowledge-enhanced graph reasoning and spectrum filtering, it end-to-end simultaneously generates the layout and object geometry of 3D indoor scenes.
FlashTex: Fast Relightable Mesh Texturing with LightControlNet: Proposed LightControlNet, an illumination-aware variant of ControlNet. Combined with a two-stage texture optimization pipeline, it generates high-quality, relightable PBR textures for 3D meshes in approximately 4 minutes, which is 3-10 times faster than existing methods.

Browse all 26 3D Generation & Editing papers →

🧊 3D Representation & Scene Understanding (20)¶

3×2: 3D Object Part Segmentation by 2D Semantic Correspondences: Proposes a training-free 3D object part segmentation method, 3-By-2, which utilizes 2D semantic correspondences from diffusion models (DIFT) to transfer part labels from annotated 2D datasets or a small number of 3D annotated objects to 3D, achieving state-of-the-art (SOTA) performance under both zero-shot and few-shot settings.
Bi-directional Contextual Attention for 3D Dense Captioning: This paper proposes BiCA, which decouples and parallelly decodes instance queries and context queries via a bi-directional contextual attention mechanism. This solves the objective conflict between localization and caption generation in 3D dense captioning, achieving state-of-the-art (SOTA) performance on both the ScanRefer and Nr3D benchmarks.
Click-Gaussian: Interactive Segmentation to Any 3D Gaussians: This work proposes Click-Gaussian, which learns a discriminative 3D feature field with two-level granularity (coarse/fine) and combines it with Global Feature-guided Learning (GFL) to address cross-view mask inconsistency. It achieves real-time interactive 3D Gaussian segmentation at only 10ms per click, which is 15-130 times faster than existing methods while significantly improving segmentation accuracy.
CloudFixer: Test-Time Adaptation for 3D Point Clouds via Diffusion-Guided Geometric Transformation: This paper proposes CloudFixer, the first test-time input adaptation method for 3D point clouds. By optimizing geometric transformation parameters guided by a pre-trained diffusion model, it transforms out-of-distribution test point clouds back to the source domain. It avoids backpropagation through the diffusion model, achieving single-instance adaptation in under 1 second.
DG-PIC: Domain Generalized Point-In-Context Learning for Point Cloud Understanding: DG-PIC is proposed, representing the first point cloud understanding framework that simultaneously addresses multi-domain and multi-task learning in a unified model. Through dual-level source prototype estimation and a test-time feature shifting mechanism, it enhances generalization capability to unseen domains without requiring model updates.
Dual-level Adaptive Self-Labeling for Novel Class Discovery in Point Cloud Segmentation: This paper proposes a dual-level adaptive self-labeling method that addresses the class imbalance problem through semi-relaxed optimal transport and incorporates region-level representations to enhance pointwise classifier learning, achieving efficient novel class discovery in point cloud segmentation.
FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimally: This paper reformulates the 2D-to-3D segmentation problem of 3D Gaussian Splatting as an integer linear program. Leveraging the linearity of alpha blending, it obtains a closed-form optimal solution in just 30 seconds, achieving a 50x speedup over existing methods.
Gaussian Grouping: Segment and Edit Anything in 3D Scenes: Learns a 16-dimensional Identity Encoding for each Gaussian in 3D Gaussian Splatting to achieve instance-level grouping, utilizing SAM + DEVA video tracking to generate multi-view consistent 2D pseudo-labels for supervision. It achieves 69-77% mIoU on LERF-Mask open-vocabulary segmentation (outperforming LERF by over 2x) and outperforms Panoptic Lifting by 4.9% mIoU in panoptic segmentation while being 14x faster, while simultaneously supporting various editing operations such as 3D object removal, inpainting, colorization, and style transfer.
GPSFormer: A Global Perception and Local Structure Fitting-Based Transformer for Point Cloud Understanding: This paper proposes GPSFormer, which learns short-range and long-range dependencies through a Global Perception Module (GPM) and accurately models local geometric information using a Taylor-series-inspired Local Structure Fitting Convolution (LSFConv). With only 2.36M parameters, it achieves 95.4% accuracy on ScanObjectNN, outperforming all supervised learning and pre-training methods.
Heterogeneous Graph Learning for Scene Graph Prediction in 3D Point Clouds: The 3D-HetSGP framework is proposed to model 3D scene graph prediction as a heterogeneous graph learning problem. By utilizing a two-stage process of Heterogeneous Graph Structure Learning (HGSL) and Heterogeneous Graph Reasoning (HGR), it addresses the suboptimal performance issue caused by indiscriminate message passing in existing homogeneous fully-connected graph methods.

Browse all 20 3D Representation & Scene Understanding papers →

🧊 Depth, Normals & Stereo (11)¶

Binomial Self-compensation for Motion Error in Dynamic 3D Scanning: This work proposes a binomial self-compensation (BSC) algorithm. By performing a weighted sum of motion-affected phase sequences based on binomial coefficients, the algorithm exponentially eliminates motion errors in four-step phase-shifting profilometry without requiring any intermediate variables, thereby achieving high-precision dynamic 3D scanning at the same frame rate as the camera.
Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions: Leverages a text-to-image diffusion model (ControlNet/T2I-Adapter) to transform clean-weather images into adverse-condition images while preserving the same 3D structure. The existing monocular depth estimation networks are fine-tuned via self-distillation to uniformly address out-of-distribution challenges like adverse weather and non-Lambertian surfaces.
DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimation: This paper introduces the diffusion denoising process to monocular depth estimation for the first time. By performing visually conditioned iterative denoising in the latent depth space, and proposing a self-diffusion mechanism to resolve the mode collapse issue caused by sparse Ground Truth (GT) depths, it achieves SOTA performance on KITTI and NYU-Depth-V2.
FutureDepth: Learning to Predict the Future Improves Video Depth Estimation: This paper proposes FutureDepth, which injects implicit motion and scene features into the depth decoder via a Future Prediction Network (F-Net) to learn motion cues and a Reconstruction Network (R-Net) to learn multi-frame correspondences. It achieves state-of-the-art (SOTA) accuracy and temporal consistency on four datasets (NYUDv2, KITTI, DDAD, and Sintel), with inference efficiency significantly surpassing existing video depth estimation methods.
GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image: The authors present GeoWizard, a foundational model for geometry estimation based on Stable Diffusion priors. It jointly predicts depth and normals using a unified model with a Geometry Switcher, and eliminates layout ambiguity in mixed scenes via a Scene Distribution Decoupler, achieving state-of-the-art results on zero-shot depth and normal benchmarks.
High-Precision Self-Supervised Monocular Depth Estimation with Rich-Resource Prior: This paper proposes RPrDepth, which utilizes features and predictions of "rich-resource" models (such as multi-frame and high-resolution models) as priors during training. Through a prior depth fusion module and a rich-resource guided loss, the model achieves or even exceeds the depth estimation accuracy of multi-frame high-resolution models while performing inference using only a low-resolution single image.
Improving Domain Generalization in Self-Supervised Monocular Depth Estimation via Stabilized Adversarial Training: Proposed the SCAT framework, which reduces the sensitivity of UNet skip connections to perturbations via the Scale Depth Network (SDN) and introduces Conflict Gradient Surgery (CGS) to resolve the dual optimization conflict caused by adversarial augmentation, successfully applying adversarial data augmentation to self-supervised monocular depth estimation for the first time to enhance cross-domain generalization.
ProDepth: Boosting Self-Supervised Multi-Frame Monocular Depth with Probabilistic Fusion: Proposes ProDepth, a probabilistic fusion framework that infers dynamic region uncertainty via an auxiliary decoder to adaptively fuse single-frame and multi-frame depth probability distributions using a weighted geometric mean. This directly corrects erroneous matching costs in the cost volume, and combined with an uncertainty-aware loss reweighting strategy, achieves SOTA performance in self-supervised multi-frame monocular depth estimation.
SEDiff: Structure Extraction for Domain Adaptive Depth Estimation via Denoising Diffusion Models: Proposes SEDiff, which for the first time leverages diffusion models to extract domain-invariant structural information, eliminating the domain gap between synthetic and real data through structure-consistent style transfer to achieve high-performance domain adaptive monocular depth estimation.
SpectraM-PS: Spectrally Multiplexed Photometric Stereo Under Unknown Spectral Composition: A spectrally multiplexed photometric stereo method (SpectraM-PS) is proposed that eliminates the need for physical model constraints. Under conditions where the spectral composition of the light source is completely unknown, it recovers surface normals from a single RGB image in a data-driven manner, achieving a breakthrough from traditional multi-shot photometric stereo to single-shot.

Browse all 11 Depth, Normals & Stereo papers →

🧊 Geometric Matching & Localization (18)¶

3D Congealing: 3D-Aware Image Alignment in the Wild: 3D Congealing aligns a set of unannotated, semantically similar internet images into a shared 3D canonical space. By combining SDS guidance from a pre-trained diffusion model to obtain the 3D shape and DINO semantic feature matching to estimate poses and coordinate mappings, it requires no templates, pose annotations, or camera parameters.
3D Single-Object Tracking in Point Clouds with High Temporal Variation: HVTrack is the first to explore 3D single-object tracking under high temporal variation scenarios. It addresses coordinate-wise cloud shape variations, distractor interference, and background noise via three modules: Relative-Pose-Aware Memory (RPM), Base-Expansion Feature Cross-Attention (BEA), and Contextual Point Guided Self-Attention (CPA). On the KITTI-HV dataset with a 5-frame interval, it improves Success/Precision by 11.3%/15.7% over the state-of-the-art (SOTA).
6DGS: 6D Pose Estimation from a Single Image and a 3D Gaussian Splatting Model: Proposes 6DGS, which inverts the 3DGS rendering workflow—casting rays uniformly from the surfaces of the ellipsoids (Ellicell), using an attention mechanism to bind rays to target image pixels, and then utilizing weighted least squares to solve for camera pose in closed form. Requiring no iterations or initial poses, it improves rotation accuracy by 12% and translation accuracy by 22% on real-world scenes, achieving near-real-time performance at 15fps.
A Direct Approach to Viewing Graph Solvability: This paper proposes a more direct reformulation of the viewing graph solvability problem than prior works, introduces new concepts to understand the solvability of real-world SfM graphs, and presents more efficient algorithms for detecting and decomposing unsolvable scenarios.
CG-SLAM: Efficient Dense RGB-D SLAM in a Consistent Uncertainty-Aware 3D Gaussian Field: This paper proposes CG-SLAM, an efficient dense RGB-D SLAM framework based on a consistency- and geometric-stability-optimized uncertainty-aware 3D Gaussian field, achieving state-of-the-art (SOTA) performance in both localization accuracy and reconstruction quality with a tracking speed of up to 15Hz.
Deep Patch Visual SLAM: Based on the DPVO visual odometry system, this work extends it into a complete SLAM system, DPV-SLAM, by introducing efficient proximity loop closure and classical loop closure mechanisms, achieving real-time, high-precision, and low-memory monocular visual SLAM on a single GPU.
Equi-GSPR: Equivariant SE(3) Graph Network Model for Sparse Point Cloud Registration: This paper proposes Equi-GSPR, a sparse point cloud registration method based on SE(3) equivariant graph neural networks. By utilizing equivariant message passing, low-rank feature transformation (LRFT), and implicit feature space similarity matching, it achieves SOTA registration performance on indoor and outdoor datasets with low model complexity.
Event-based Mosaicing Bundle Adjustment: This work proposes EMBA, the first photometric Bundle Adjustment method for rotation-only event cameras. It formulates the problem as a regularized non-linear least squares optimization based on a linearized event generation model, designs an efficient solver by exploiting the block-diagonal sparse structure of the normal equation matrix, and simultaneously optimizes the camera rotation trajectory and the panoramic gradient map.
FastCAD: Real-Time CAD Retrieval and Alignment from Scans and Videos: This work proposes FastCAD to achieve CAD model retrieval and alignment for all objects in a scene within 50ms through contrastive learning embedding space distillation and direct parameter prediction, which is 50 times faster than existing methods with superior accuracy.
GaussReg: Fast 3D Registration with Gaussian Splatting: This work presents the first exploration of registration between 3D Gaussian Splatting scenes, proposing a coarse-to-fine GaussReg framework. The coarse stage utilizes point cloud registration to estimate the initial transformation, while the fine stage extracts volumetric features from rendered images for fine-grained alignment, achieving comparable accuracy to HLoc while being 44x faster.

Browse all 18 Geometric Matching & Localization papers →

🎯 Object Detection (31)¶

YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information: YOLOv9 proposes Programmable Gradient Information (PGI) and Generalized Efficient Layer Aggregation Network (GELAN) to address the information bottleneck problem in deep networks. It comprehensively outperforms existing real-time object detectors on MS COCO with fewer parameters and computation, surpassing methods pre-trained on large-scale datasets while training from scratch.
DAMSDet: Dynamic Adaptive Multispectral Detection Transformer: DAMSDet proposes a dynamic adaptive infrared-visible object detection method based on the DETR architecture. By utilizing Modality Competitive Query Selection (dynamically selecting the dominant modality feature as the initial query for each object) and Multispectral Deformable Cross-Attention (adaptively sampling and aggregating bi-modal features across multiple semantic levels), it simultaneously addresses the dual challenges of complementary information fusion and modality misalignment, significantly outperforming the state-of-the-art (SOTA) on four public datasets.
Portrait4D-v2: Pseudo Multi-View Data Creates Better 4D Head Synthesizer: A new learning paradigm utilizing pseudo multi-view videos to train a feed-forward, one-shot 4D head synthesizer is proposed. It first learns a 3D head synthesizer from synthetic data to convert monocular videos into multi-view videos, and then trains the 4D synthesizer through cross-view self-reenactment using these pseudo multi-view videos. This avoids over-reliance on 3DMMs and significantly outperforms prior methods in reconstruction fidelity, geometric consistency, and motion control accuracy.
Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching: Constructed the first natural language-guided drone geolocalization benchmark GeoText-1652 (276K bbox-text pairs, 316K descriptions), and proposed a blending spatial matching method that achieves region-level spatial relation matching via grounding loss + spatial relation loss, achieving a text retrieval Recall@10 of 31.2%.
TAPTR: Tracking Any Point with Transformers as Detection: TAPTR reformulates the Tracking Any Point (TAP) task as a DETR-like detection problem. It represents each tracking point as a point query containing both position and content, which is layer-wise optimized through a multi-layer Transformer decoder. Combined with a cost volume and sliding-window feature update strategy, it achieves SOTA performance on the TAP-Vid benchmark with faster inference speed.
BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videos: Proposes the Boundary-Aligned Moment Detection Transformer (BAM-DETR), which models moments using an anchor-boundary triplet \((p, d_s, d_e)\) instead of the traditional center-length duplet \((c, l)\). Combined with a dual-pathway decoder and a quality-based ranking mechanism, it effectively addresses the issue of imprecise localization caused by center ambiguity.
Visible and Clear: Finding Tiny Objects in Difference Map: SR-TOD introduces the image self-reconstruction mechanism into object detection for the first time, discovering a strong correlation between reconstruction difference maps and tiny objects. It designs a Difference Map Guided Feature Enhancement (DGFE) module, achieving significant improvements on the self-built anti-UAV dataset DroneSwarms as well as VisDrone2019 and AI-TOD.
Self-supervised Feature Adaptation for 3D Industrial Anomaly Detection: This paper proposes the LSFA (Local-to-global Self-supervised Feature Adaptation) framework. It performs task-oriented adaptation of pretrained features through two self-supervised strategies: Intramodal Feature Compactness (IFC) optimization and Cross-modal Local-to-global Consistency (CLC) alignment. LSFA achieves 97.1% I-AUROC on MVTec-3D AD, outperforming the state-of-the-art (SOTA) by +3.4%.
Adaptive Bounding Box Uncertainties via Two-Step Conformal Prediction: This paper proposes a two-step conformal prediction framework for uncertainty quantification in multi-object detection: the first step generates conformal prediction sets of class labels to handle classification errors, and the second step produces adaptive bounding box uncertainty intervals based on ensembles and quantile regression, providing practically useful tight prediction intervals while guaranteeing coverage.
GRA: Detecting Oriented Objects Through Group-Wise Rotating and Attention: A lightweight Group-wise Rotating and Attention (GRA) module is proposed. By grouping and rotating convolution kernels and applying group-wise spatial attention, it outperforms the previous SOTA method ARC with nearly 50% fewer parameters, achieving new state-of-the-art performance on DOTA-v2.0.

Browse all 31 Object Detection papers →

✂️ Segmentation (54)¶

Rotary Position Embedding for Vision Transformer: This work systematically investigates the application of RoPE (Rotary Position Embedding) from 1D language models to 2D vision tasks. It proposes RoPE-Mixed (mixed learnable frequencies) to replace the conventional Axial frequency allocation. This approach achieves significant resolution extrapolation performance gains on ViT and Swin Transformer, yielding consistent improvements in ImageNet classification, COCO detection, and ADE20k segmentation.
SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference: This work discovers that the failure of CLIP in dense prediction stems from spatial misalignment caused by self-attention. It proposes the Correlative Self-Attention (CSA) mechanism, which modifies only the computation of the last self-attention layer (training-free). This improvement elevates CLIP's zero-shot semantic segmentation performance from 14.1% average mIoU to 38.2%, surpassing all existing methods.
ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback: This paper proposes ControlNet++, which explicitly optimizes the quality of conditional controllable generation through pixel-level cycle consistency loss. A pre-trained discriminative model is used to extract conditions from generated images and align them with the input conditions. To avoid the massive GPU memory overhead of multi-step sampling, an efficient single-step denoising reward strategy is designed. This significantly improves controllability (e.g., +11.1% segmentation mIoU) under various conditional controls such as segmentation masks, edges, and depth.
VISA: Reasoning Video Object Segmentation via Large Language Models: Proposes the new ReasonVOS task and the VISA model, utilizing the world knowledge reasoning capabilities of multi-modal LLMs to achieve video object segmentation and tracking based on implicit text queries.
Frequency-Spatial Entanglement Learning for Camouflaged Object Detection: The Frequency-Spatial Entanglement Learning (FSEL) framework is proposed. By conducting entanglement learning between the frequency and spatial domains, it utilizes global frequency features to compensate for the locality and sensitivity limitations of spatial features, outperforming 21 state-of-the-art (SOTA) methods across three COD benchmarks.
DreamLIP: Language-Image Pre-training with Long Captions: By generating long captions for 30M images using MLLMs, this work proposes multi-positive contrastive learning with dynamic subcaption sampling and a subcaption-specific grouping loss to achieve fine-grained vision-language alignment, matching or even surpassing the performance of CLIP 400M on retrieval and semantic segmentation tasks using only 30M data.
SiLC: Improving Vision Language Pretraining with Self-Distillation: Introduces the SiLC framework, which incorporates local-to-global self-distillation into CLIP-style image-text contrastive learning, significantly boosting performance on dense prediction tasks (detection, segmentation) while also improving classification and retrieval.
ReMamber: Referring Image Segmentation with Mamba Twister: This paper introduces the Mamba architecture to the Referring Image Segmentation (RIS) task for the first time. It proposes the Mamba Twister module, which achieves efficient vision-language feature fusion through a "twisting" mechanism of channel and spatial scanning. It achieves competitive results surpassing Transformer-based methods on RefCOCO/RefCOCO+/G-Ref benchmarks while maintaining linear computational complexity.
Diffusion Models for Open-Vocabulary Segmentation: This paper proposes OVDiff, which leverages a pre-trained text-to-image diffusion model to generate support image sets for arbitrary textual categories. From these, multi-level prototypes (class-level, instance-level, and part-level) are extracted and combined with background prototypes to achieve training-free open-vocabulary semantic segmentation, outperforming prior methods by over 10% on PASCAL VOC.
DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs: Revisiting the dense concatenation shortcut of DenseNet, this paper proposes RDNet (Revitalized DenseNet) through a systematic modernization (wider and shallower architecture, modernized blocks, enlarged intermediate dimensions, more transition layers, etc.). RDNet outperforms Swin Transformer, ConvNeXt, and DeiT-III on ImageNet-1K, proving the powerful potential of concatenation as an underestimated paradigm.

Browse all 54 Segmentation papers →

🖼️ Image Restoration (33)¶

MambaIR: A Simple Baseline for Image Restoration with State-Space Model: This paper introduces Mamba (Selective State-Space Model) to low-level image restoration tasks for the first time. By designing local convolution enhancement and channel attention mechanisms within the Residual State-Space Block (RSSB), the proposed method addresses the issues of local pixel forgetting and channel redundancy in vanilla Mamba on 2D images. It achieves comparable or even superior performance to Transformer-based methods with linear complexity on image super-resolution and denoising tasks (outperforming SwinIR by 0.45dB on SR).
BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion: Proposes BrushNet, a plug-and-play dual-branch diffusion model architecture for image inpainting. By decoupling masked image feature extraction and image generation into separate branches, it achieves layer-wise pixel-level feature injection, thoroughly outperforming existing methods in image quality, masked area preservation, and text alignment.
Restoring Images in Adverse Weather Conditions via Histogram Transformer: Proposed Histoformer, an efficient Transformer based on histogram self-attention. By sorting and binning spatial features according to pixel intensity, it performs self-attention within and across bins to establish dynamic-range spatial attention for efficiently processing weather-degraded pixels. Combined with dynamic-range convolution and Pearson correlation loss, it achieves a unified modeling and reaches SOTA performance on three major tasks: desnowing, deraining/dehazing, and deraindropping.
BAMM: Bidirectional Autoregressive Motion Model: BAMM (Bidirectional Autoregressive Motion Model) is proposed. By unifying generative masked modeling and autoregressive modeling through a hybrid attention masking strategy, it simultaneously achieves high-quality motion generation, adaptive length prediction, and zero-shot motion editing within a single framework, comprehensively outperforming SOTA on HumanML3D and KIT-ML.
Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration: This paper proposes FPro, guiding image restoration via prompting from a frequency-domain perspective. Employing a Gated Dynamic Decoupler to decouple features into low-frequency and high-frequency components, the method injects learnable prompts into both bands using a Dual Prompt Block (HPM + LPM) to interact with decoder features. It outperforms state-of-the-art methods comprehensively across five tasks: deraining, raindrop removal, demoireing, deblurring, and dehazing.
Contourlet Residual for Prompt Learning Enhanced Infrared Image Super-Resolution: To address the unique challenges of infrared image super-resolution, this paper proposes the CoRPLE framework. It utilizes the Contourlet transform for multi-scale and multi-directional infrared spectral residual enhancement, and introduces a prompt learning paradigm based on vision-language models to capture the inherent features of infrared images, achieving SOTA performance on infrared SR tasks.
Teaching Tailored to Talent: Adverse Weather Restoration via Prompt Pool and Depth-Anything Constraint: This paper proposes T3-DiffWeather, a diffusion-based all-in-one adverse weather restoration framework. It utilizes a prompt pool to allow the network to autonomously combine sub-prompts to construct instance-level weather-prompts for modeling diverse weather degradations. Concurrently, it leverages Depth-Anything features to constrain general prompts to model scene information. The method achieves state-of-the-art (SOTA) performance in only 2 sampling steps, with a computational cost of only 1/52 of WeatherDiffusion.
Towards Real-world Event-guided Low-light Video Enhancement and Deblurring: This paper introduces the joint task of event-guided low-light video enhancement and deblurring for the first time. It constructs a beam-splitter-based real-world dataset, RELED, and designs an end-to-end framework consisting of two core modules: Event-Guided Deformable Temporal Feature Alignment (ED-TFA) and Spectrum Frequency-based Cross-Modal Feature Enhancement (SFCM-FE), outperforming previous state-of-the-art methods by over 1.2 dB in PSNR.
Raindrop Clarity: A Dual-Focused Dataset for Day and Night Raindrop Removal: The authors propose Raindrop Clarity, a large-scale real-world raindrop removal dataset containing 15,186 high-quality image pairs/triplets. For the first time, it covers raindrop-focused (clear raindrops with blurred background) and nighttime raindrop scenarios, both of which are missing from existing datasets.
Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators: Discovering that query-key interactions in Diffusion Transformers exhibit significant redundancy (especially in the early stages of denoising), this work proposes the Attention Mediator mechanism to reduce attention complexity to linear. It further designs a step-wise dynamic adjustment strategy, achieving a state-of-the-art FID of 2.01 on SiT-XL/2 while reducing computational overhead.

Browse all 33 Image Restoration papers →

🛰️ Remote Sensing (6)¶

ConGeo: Robust Cross-View Geo-Localization Across Ground View Variations: This paper proposes ConGeo, a model-agnostic single-view + cross-view contrastive learning framework. By enforcing feature consistency across different ground view variations at the same location, it enables a single model to achieve robust cross-view geo-localization under arbitrary orientations and fields of view (FoV).
Masked Angle-Aware Autoencoder for Remote Sensing Images: The authors propose MA3E, which explicitly introduces angle variations into MAE pre-training (by constructing rotational crops via scaling center crop) and automatically assigns reconstruction targets using an optimal transport loss. This allows the model to perceive the diverse angles of remote sensing objects and learn rotation-invariant representations.
Cross-Platform Video Person ReID: A New Benchmark Dataset and Adaptation Approach: This work constructs G2A-VReID, the first ground-to-aerial cross-platform video person re-identification dataset, and proposes the VSLA-CLIP method, which adapts CLIP to video ReID tasks through visual-semantic alignment and a parameter-efficient Video Set-Level-Adapter.
Weakly-Supervised Camera Localization by Ground-to-Satellite Image Registration: Proposes the first weakly-supervised ground-to-satellite image registration localization method. By training an orientation estimator on satellite-to-satellite pairs in a self-supervised manner and training a translation estimator via contrastive learning, it achieves the best cross-area generalization performance without requiring accurate ground-truth (GT) pose labels, outperforming most fully-supervised SOTA methods.
Adapting Fine-Grained Cross-View Localization to Areas without Fine Ground Truth: To address the performance degradation of fine-grained cross-view localization models when deployed in new areas, a weakly-supervised learning method based on knowledge self-distillation is proposed. Employing three strategies—mode-based pseudo GT generation, coarse-level supervision, and outlier filtering—it reduces localization errors by 12% to 20% on VIGOR and KITTI using only ground-to-aerial image pairs from the target area (without requiring precise GT).
Learning Representations of Satellite Images From Metadata Supervision: This paper proposes SatMIP (Satellite Metadata-Image Pretraining), which represents satellite image metadata (such as time, geographic location, sensor information, etc.) as text descriptions to align images and metadata in a shared embedding space via an image-metadata contrastive learning task. This constructs satellite image representations that encode both visual features and semantic information. It further introduces SatMIPS (combining image self-supervision and metadata supervision), which outperforms purely visual self-supervised methods like SimCLR on multiple remote sensing downstream tasks.

🧑 Human Understanding (70)¶

Motion Mamba: Efficient and Long Sequence Motion Generation: This paper proposes Motion Mamba, which is the first to introduce Selective State Space Models (Mamba) to human motion generation. Through two core components, Hierarchical Temporal Mamba (HTM) and Bidirectional Spatial Mamba (BSM), it reduces FID by 50% (0.473 \(\rightarrow\) 0.281) on HumanML3D while achieving a 4x inference speedup (0.217s \(\rightarrow\) 0.058s).
FoundPose: Unseen Object Pose Estimation with Foundation Features: FoundPose leverages a frozen DINOv2 foundation model to extract patch descriptors, establishing 2D-3D correspondences via bag-of-words template retrieval and kNN matching. It achieves zero-shot 6D pose estimation of unseen objects without any task-specific training, significantly outperforming existing RGB methods on BOP benchmarks.
EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis: Proposes EDTalk, an efficient disentanglement framework based on learnable orthogonal basis vectors, which decomposes facial dynamics into three independent latent spaces: mouth shape, head pose, and emotional expression. It simultaneously supports both video-driven and audio-driven emotional talking head generation.
CoMo: Controllable Motion Generation Through Language Guided Pose Code Editing: This paper introduces CoMo, which decomposes motion sequences into semantically explicit pose codes (e.g., "left knee slightly bent") to achieve text-based controllable motion generation and LLM-based zero-shot motion editing.
Large Motion Model for Unified Multi-Modal Motion Generation: This paper proposes the Large Motion Model (LMM), the first motion-centric unified multimodal motion generation foundation model. By constructing the MotionVerse benchmark containing 10 tasks, 16 datasets, and 320K sequences, designing a body-part-aware ArtAttention mechanism, and incorporating a pre-training strategy with random frame rates and masking, LMM achieves high-quality motion generation across diverse tasks.
Multi-Memory Matching for Unsupervised Visible-Infrared Person Re-Identification: A Multi-Memory Matching (MMM) framework is proposed for unsupervised visible-infrared person re-identification. It establishes reliable cross-modal correspondences through three modules: Cross-Modal Clustering (CMC), Multi-Memory Learning and Matching (MMLM), and Soft Cluster-level Alignment loss (SCA), achieving a Rank-1 accuracy of 61.6% on SYSU-MM01 and 89.7% on RegDB.
FreeMotion: A Unified Framework for Number-free Text-to-Motion Synthesis: The FreeMotion framework is proposed to recursively decompose the joint distribution of multi-person motions into conditional single-person motion generation through conditional probability decomposition. This achieves text-driven motion synthesis for an arbitrary number of individuals for the first time, while supporting multi-person spatial control.
Wear-Any-Way: Manipulable Virtual Try-on via Sparse Correspondence Alignment: The Wear-Any-Way framework is proposed, establishing a strong baseline for high-fidelity virtual try-on based on a dual U-Net diffusion model. By introducing a point control mechanism via Sparse Correspondence Alignment, it enables users to precisely manipulate wearing styles (e.g., rolling up sleeves, opening/closing coats, tucking in hems) through click-and-drag interactions, achieving state-of-the-art performance in both standard and manipulable try-on scenarios.
3D Hand Pose Estimation in Everyday Egocentric Images: By systematically investigating four practices—cropped inputs, Intrinsics-aware Positional Encoding (KPE), auxiliary supervision (hand segmentation + grasp labels), and multi-dataset joint training—this work proposes the WildHands system. Under the constraint of using only a ResNet50 backbone and a small amount of data, WildHands achieves robust 3D hand pose estimation in in-the-wild egocentric images. Its zero-shot generalization outperforms FrankMocap across all metrics and competes closely with HaMeR, which is \(10\times\) larger.
Diffusion Model is a Good Pose Estimator from 3D RF-Vision: Proposes mmDiff, a diffusion-based framework for millimeter-wave (mmWave) radar human pose estimation. By employing global-local radar context extraction and structural-temporal motion consistency constraints, it effectively addresses the challenges of sparse, noisy, and inconsistent radar point clouds, significantly outperforming existing SOTA methods.

Browse all 70 Human Understanding papers →

📹 Video Understanding (51)¶

VideoMamba: State Space Model for Efficient Video Understanding: This work innovatively adapts Mamba's selective state space model to the video domain, proposing VideoMamba, a pure SSM architecture. It achieves efficient spatiotemporal context modeling with linear complexity, demonstrating superior performance on both short and long video understanding tasks.
SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow: SEA-RAFT achieves SOTA accuracy while maintaining a simple architecture through three improvements: Mixture of Laplace (MoL) loss, direct regression of initial optical flow, and rigid-flow pre-training, achieving a speedup of over 2.3× compared to existing methods.
Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance: LoRAT introduces LoRA to visual object tracking for the first time. Through two LoRA-friendly designs—decoupled position encoding (shared spatial components + independent type embeddings) and a pure MLP detection head—it enables training a tracker with a ViT-g backbone using laboratory-level resources. It achieves a SUC of 0.762 on LaSOT (new SOTA), while the lightest variant, LoRAT-B-224, runs at 209 FPS.
Local All-Pair Correspondence for Point Tracking: This paper proposes LocoTrack, which achieves all-pair correspondence matching for any points in a video via a local 4D correlation volume. Combined with a lightweight correlation encoder and a length-generalizable Transformer, it obtains state-of-the-art accuracy across all TAP-Vid benchmarks while executing nearly 6 times faster than SOTA methods.
DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video: This paper proposes DINO-Tracker, which combines the semantic features of pretrained DINOv2 with test-time single-video optimization. Through Delta-DINO residual fine-tuning and multi-source self-supervised losses, it achieves long-range dense point tracking. It reaches state-of-the-art (SOTA) performance among self-supervised methods and is comparable to supervised trackers, particularly outperforming existing methods by a wide margin in long-term occlusion scenarios.
Goldfish: Vision-Language Understanding of Arbitrarily Long Videos: This work introduces the Goldfish framework, which achieves efficient understanding of arbitrarily long videos by segmenting them into short clips and utilizing a text-similarity-based retrieval mechanism to select the top-k segments most relevant to the question. It also presents the MiniGPT4-Video short video model and the TVQA-long benchmark for long video evaluation.
R²-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding: This paper proposes R²-Tuning, which appends a lightweight R² Block (only 1.5% of total parameters) recursively in a backward manner onto the last several layers of a frozen CLIP model. It enables query-modulated spatial pooling and coarse-to-fine temporal refinement, outperforming state-of-the-art (SOTA) methods that require additional temporal backbones on 6 VTG benchmarks across 3 tasks with only 2.7M parameters.
Vamos: Versatile Action Models for Video Understanding: Proposes the Vamos framework, which uses Large Language Models as reasoners to flexibly unify visual embeddings and general text descriptions as video representations. It discovers that text-only representations consistently achieve competitive or even superior performance across multiple video understanding benchmarks. Furthermore, it designs a Token Bottleneck Model to achieve interpretable evidence selection and a 5x inference speedup.
EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywhere: Proposes EgoPoser to robustly estimate full-body poses from sparse and intermittent tracking signals of the head and hands from head-mounted devices. Through four core designs—global motion decomposition, realistic field-of-view modeling, SlowFast temporal fusion, and body-shape-aware pose optimization—it achieves state-of-the-art performance in large-scale real-world scenarios while running at over 600 fps.
Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects: Based on the HANDS23 challenge (using the AssemblyHands and ARCTIC datasets), this study systematically benchmarks and deeply analyzes 3D pose estimation methods for egocentric hand-object interactions, revealing the effectiveness of distortion correction, high-capacity Transformers, and multi-view fusion, while highlighting unresolved challenges such as rapid motion, severe occlusion, and object reconstruction under narrow viewpoints.

Browse all 51 Video Understanding papers →

🚗 Autonomous Driving (61)¶

OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving: OccWorld proposes learning a world model in 3D occupancy space. It tokenizes 3D occupancy via VQ-VAE and predicts future scene evolution and ego-vehicle trajectories autoregressively using a GPT-style spatial-temporal generative Transformer, achieving competitive planning performance on nuScenes without requiring instance or HD map annotations.
GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction: This paper proposes an object-centric 3D semantic Gaussian representation to replace traditional dense voxels. It describes driving scenes using a set of sparse 3D semantic Gaussians and generates occupancy predictions via Gaussian-to-voxel splatting, reducing memory consumption by 75%–82% while yielding comparable performance.
Reason2Drive: Towards Interpretable and Chain-Based Reasoning for Autonomous Driving: This paper constructs the Reason2Drive benchmark dataset (600K+ video-text pairs, covering perception-prediction-reasoning chain tasks), proposes ADRScore as a new metric to evaluate the correctness of chain-based reasoning, and designs a Prior Tokenizer + Instructed Vision Decoder framework to enhance the object-level perception and reasoning capabilities of VLMs, significantly outperforming all baselines on autonomous driving reasoning tasks.
GraphBEV: Towards Robust BEV Feature Alignment for Multi-Modal 3D Object Detection: To address the feature misalignment caused by calibration errors between LiDAR and cameras in multi-modal BEV fusion, this paper proposes the GraphBEV framework. It introduces two modules: LocalAlign (KD-Tree-based neighborhood depth graph matching) and GlobalAlign (global alignment via learnable offsets). GraphBEV achieves 70.1% mAP on nuScenes (outperforming BEVFusion by 1.6%) and outperforms BEVFusion by 8.3% in noisy misalignment scenarios.
Fully Sparse 3D Occupancy Prediction: This work proposes SparseOcc, the first fully sparse 3D occupancy prediction network, which achieves efficient occupancy prediction via a sparse voxel decoder and a mask-guided Mask Transformer, and designs a RayIoU evaluation metric to address the depth-direction inconsistent penalty of traditional mIoU.
MapTracker: Tracking with Strided Memory Fusion for Consistent Vector HD Mapping: Redefines online vector HD mapping as a tracking task. It achieves temporally consistent HD map reconstruction through a strided memory buffer fusion mechanism with dual representations (BEV grid + road element vector), significantly outperforming existing methods on nuScenes and Argoverse2 with 76.1 and 76.9 mAP, respectively.
UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction: UniTraj establishes a unified framework for vehicle trajectory prediction by standardizing multiple datasets (nuScenes, Argoverse 2, WOMD), models (AutoBot, MTR, Wayformer), and evaluation strategies. The study reveals a significant drop in cross-dataset generalization for individual models, but demonstrates that scaling up training data volume and diversity substantially boosts performance, achieving 1st place on the nuScenes leaderboard via joint training.
NeuroNCAP: Photorealistic Closed-Loop Safety Testing for Autonomous Driving: This paper proposes NeuroNCAP, a photorealistic closed-loop safety testing framework for autonomous driving based on NeRF rendering. Inspired by the Euro NCAP collision avoidance protocols, three types of safety-critical scenarios (stationary, frontal, and side collisions) are designed. It reveals that current state-of-the-art (SOTA) end-to-end planners (UniAD, VAD) fail catastrophically in closed-loop safety scenarios—with collision rates as high as 88-92%—despite their perception modules functioning accurately.
Safe-Sim: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries: Safe-Sim proposes a closed-loop safety-critical simulation framework based on diffusion models. By introducing an adversarial term and a Partial Diffusion mechanism into the diffusion denoising process, it achieves fine-grained control over adversarial vehicle behavior types (collision angles, relative velocities, and collision categories), validating its effective assessment capabilities over multiple planners on nuScenes and nuPlan.
Stream Query Denoising for Vectorized HD-Map Construction: This paper proposes the Stream Query Denoising (SQD) strategy. By adding noise to the ground truth (GT) of the previous frame and training the network to reconstruct the current frame's GT, temporal consistency modeling in streaming HD map construction is enhanced. This approach consistently outperforms StreamMapNet on nuScenes and Argoverse2.

Browse all 61 Autonomous Driving papers →

🤖 Robotics & Embodied AI (14)¶

GraspXL: Generating Grasping Motions for Diverse Objects at Scale: GraspXL is proposed, an RL-based grasp motion generation framework that generalizes to over 500,000 unseen objects after training on only 58 objects, while simultaneously supporting multi-objective motion control (grasp region, heading, wrist rotation, and hand position) and multiple dexterous hand platforms.
QUAR-VLA: Vision-Language-Action Model for Quadruped Robots: This paper proposes the first vision-language-action (QUAR-VLA) paradigm for quadruped robots, constructing a multi-task dataset QUARD with 259K episodes and the QUART model based on a pretrained multimodal large model, achieving unified control of multi-tasks such as perception, navigation, and whole-body manipulation.
See and Think: Embodied Agent in Virtual Environment: This paper proposes STEVE, an open-world embodied agent in Minecraft based on three major components: visual perception, language instruction, and code action. By fine-tuning LLaMA-2 on the STEVE-21K dataset and combining it with a visual encoder and a skill database, STEVE significantly outperforms existing methods in tech tree unlocking and block search tasks.
Prioritized Semantic Learning for Zero-shot Instance Navigation: This paper proposes the Prioritized Semantic Learning (PSL) method. Through a semantic-augmented agent architecture, a prioritized semantic training strategy, and a semantic expansion inference scheme, it significantly improves the agent's semantic perception capabilities in zero-shot object/instance navigation, achieving SOTA performance on both ObjectNav and the newly proposed InstanceNav tasks.
SemGrasp: Semantic Grasp Generation via Language Aligned Discretization: This work proposes SemGrasp, which designs a hierarchical VQ-VAE to discretize grasp poses into three semantic tokens ("orientation-manner-refinement"). It then fine-tunes a Multimodal Large Language Model (MLLM) to align objects, grasps, and language within a unified semantic space, enabling the generation of physically plausible and semantically consistent human grasp poses from natural language instructions.
An Economic Framework for 6-DoF Grasp Detection: This paper proposes the EconomicGrasp framework. By identifying that the ambiguity problem in dense supervision is the root cause of the conflict between performance and resource consumption, it designs an economic supervision paradigm (retaining all view perspectives but cropping rotation angles and depths) and a focus representation module (an interactive grasp head with composite scoring). It outperforms the SOTA by approximately 3 AP on GraspNet-1Billion with only 1/4 of the training time and 1/8 of the memory cost.
ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environments: Introduces the ReALFRED benchmark, which replaces ALFRED's synthetic single-room scenes with 150 real-world 3D scanned, interactive multi-room environments, providing 30,696 free-form language instructions and revealing a significant performance drop of existing embodied instruction-following methods in real environments.
LLM as Copilot for Coarse-Grained Vision-and-Language Navigation: This paper proposes the VLN-Copilot framework, where a vision-and-language navigation agent actively seeks help from an LLM when confused under coarse-grained (short and ambiguous) instructions. Acting as a copilot, the LLM generates real-time, fine-grained navigation guidance, significantly improving navigation success rates on two coarse-grained VLN datasets.
AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipation: Proposes the STAformer architecture and two affordance-based modules (an environment affordance database + interaction hotspots), improving the relative performance of Short-Term object Interaction Anticipation (STA) in egocentric videos by 30-45% on Ego4D and EPIC-Kitchens.
Learning Cross-Hand Policies of High-DOF Reaching and Grasping: A two-stage hierarchical framework is proposed, which uses semantic keypoints and the Interaction Bisector Surface (IBS) as hand-agnostic state representations. Combined with a Transformer policy network and hand-specific adaptation models, it achieves zero-shot transfer of dexterous grasping policies across different high-DOF robotic hands.

Browse all 14 Robotics & Embodied AI papers →

🎮 Reinforcement Learning (3)¶

Octopus: Embodied Vision-Language Programmer from Environmental Feedback: This paper proposes Octopus, an embodied vision-language programming model that bridges high-level planning and low-level manipulation by generating executable code. It introduces a Reinforcement Learning with Environmental Feedback (RLEF) training scheme to enhance decision-making quality.
Visual Grounding for Object-Level Generalization in Reinforcement Learning: This paper leverages the visual grounding capability of a vision-language model (MineCLIP) to generate confidence maps of target objects. VLM knowledge is transferred to reinforcement learning through two pathways—reward design and task representation—enabling zero-shot generalization to unseen objects and instructions.
AdaGlimpse: Active Visual Exploration with Arbitrary Glimpse Position and Scale: This paper proposes AdaGlimpse, which utilizes Soft Actor-Critic (SAC) reinforcement learning to select glimpses of arbitrary positions and scales from a continuous action space. Combined with a ViT encoder equipped with elastic positional encoding, it achieves multi-task active visual exploration (reconstruction, classification, and segmentation), outperforming state-of-the-art methods that use 18% of pixels, while requiring only 6% of pixels.

🎁 Recommender Systems (1)¶

AID-AppEAL: Automatic Image Dataset and Algorithm for Content Appeal Enhancement and Assessment Labeling: This paper proposes the Image Content Appeal Assessment (ICAA) task for the first time, distinguishing it from traditional Image Aesthetics Assessment (IAA). It designs a complete pipeline integrating automatic dataset generation, appeal estimation, and appeal enhancement, achieving large-scale dataset creation with zero human annotation using Stable Diffusion and Textual Inversion.

🔄 Self-Supervised Learning (19)¶

Exemplar-Free Continual Representation Learning via Learnable Drift Compensation: Proposes Learnable Drift Compensation (LDC), which trains a forward projector to map the old feature space to the new feature space. This effectively compensates for the semantic drift of class prototypes without needing to store old exemplars, achieving exemplar-free semi-supervised continual learning for the first time.
MarineInst: A Foundation Model for Marine Image Analysis with Instance Visual Description: This paper proposes MarineInst, a foundation model for marine image analysis that simultaneously outputs instance masks and semantic descriptions. Additionally, it constructs MarineInst20M—the largest marine image dataset to date (20 million images), supporting multi-level marine visual analysis tasks from image-level scene understanding to region-level instance understanding.
InfMAE: A Foundation Model in the Infrared Modality: This paper proposes InfMAE, the first foundation model designed specifically for the infrared modality. By constructing the Inf30 dataset with 300,000 infrared images, and designing an information-aware masking strategy along with a multi-scale encoder, the proposed method outperforms existing state-of-the-art approaches in three downstream tasks: infrared semantic segmentation, object detection, and small object detection.
PosFormer: Recognizing Complex Handwritten Mathematical Expression with Position Forest Transformer: The Position Forest Transformer (PosFormer) is proposed. By encoding the LaTeX sequence of mathematical expressions into a position forest structure, it explicitly models the hierarchical and spatial relationships among symbols. An implicit attention correction module is designed to comprehensively outperform SOTA methods on single-line, multi-line, and complex expression datasets, without introducing additional inference overhead.
Efficient Image Pre-Training with Siamese Cropped Masked Autoencoders: CropMAE is proposed: a siamese masked autoencoder trained by replacing video frame pairs with two randomly cropped views of the same image. With an extremely high masking ratio of 98.5%, it learns object boundary-aware representations using only 2 visible patches. This accelerates training by up to 23.8× compared to SiamMAE while achieving competitive performance on video propagation tasks.
ViC-MAE: Self-Supervised Representation Learning from Images and Video with Contrastive Masked Autoencoders: ViC-MAE unifies contrastive learning and masked autoencoders into a single framework. By treating short video clips as augmented views (rather than duplicating images into video frames), it achieves excellent performance on both image and video downstream tasks—reaching 87.1% top-1 on ImageNet-1K (+2.4% over OmniMAE) and 75.9% on SSv2.
PromptCCD: Learning Gaussian Mixture Prompt Pool for Continual Category Discovery: This paper proposes the PromptCCD framework, which utilizes a Gaussian Mixture Model (GMM) as a prompt pool to achieve continual discovery of novel categories in unlabeled data streams while mitigating catastrophic forgetting.
COHO: Context-Sensitive City-Scale Hierarchical Urban Layout Generation: This work proposes a city-scale 2.5D urban layout generation method based on Graph Masked Autoencoders (GMAE). By capturing multi-level semantic context across buildings, blocks, and communities through a canonical graph representation and combining it with priority-scheduled iterative sampling, the method achieves realistic, semantically consistent, and topologically correct large-scale urban layout generation across 330 US cities.
Adaptive Multi-head Contrastive Learning: This paper proposes Adaptive Multi-head Contrastive Learning (AMCL), which generates different feature perspectives through multiple projection heads and independently weights each sample pair using an adaptive temperature mechanism derived from Maximum Likelihood Estimation (MLE). This effectively resolves the overlap in similarity distributions of positive and negative samples under diverse data augmentations, consistently improving the performance of SimCLR, MoCo, and Barlow Twins.
SCPNet: Unsupervised Cross-modal Homography Estimation via Intra-modal Self-supervised Learning: SCPNet is proposed, which of first achieves effective unsupervised cross-modal homography estimation on datasets with large modal discrepancy (e.g., satellite-to-map) through the synergy of three key components: intra-modal self-supervised learning, correlation networks, and consistent feature map projection, achieving an MACE that is 14% lower than the supervised method MHN.

Browse all 19 Self-Supervised Learning papers →

📐 Optimization & Theory (2)¶

Fine-Grained Scene Graph Generation via Sample-Level Bias Prediction: This paper proposes a sample-level bias prediction method named SBP. By leveraging a Bias-Oriented GAN, it utilizes the contextual information of the union region of object pairs to predict sample-specific bias correction vectors, reforming coarse-grained relationships into fine-grained ones. SBP outperforms dataset-level bias correction methods by an average of 5.6%/3.9%/3.2% in Average@K on VG/GQA/VG-1800 datasets, respectively.
Handling the Non-smooth Challenge in Tensor SVD: A Multi-objective Tensor Recovery Framework: A multi-objective tensor recovery framework (MOTC) based on learnable tensor nuclear norm is proposed. By introducing learnable unitary matrices in place of fixed transforms, this approach addresses the performance degradation of t-SVD methods on non-smooth tensor data, while effectively exploiting the low-rankness of tensors across all dimensions through multi-objective optimization.

🔗 Causal Inference (4)¶

Understanding Physical Dynamics with Counterfactual World Modeling: This paper proposes Counterfactual World Modeling (CWM), which trains a masked video predictor using a temporally-factored masking policy and designs a "counterfactual prompting" mechanism to extract multiple visual structures (e.g., optical flow, segmentation, keypoints) from a single pre-trained model without fine-tuning, achieving state-of-the-art performance on the Physion benchmark for physical dynamics understanding.
Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillation: This paper proposes a bi-level data pruning strategy, BiLP, which combines static pruning based on empirical loss and dynamic pruning based on individual treatment effect (ITE) to efficiently select the most valuable real samples for dataset distillation. It consistently improves the performance of existing distillation methods in a plug-and-play manner while reducing computational overhead.
Integrating Markov Blanket Discovery into Causal Representation Learning for Domain Generalization: This work proposes the CMBRL framework to discover Markov Blanket (MB) features—the minimal sufficient statistics of the target variable—within the latent space. This replaces the convention of selecting only causal or anti-causal variables in existing methods, constructing an invariant prediction mechanism to achieve cross-domain generalization.
Learning Chain of Counterfactual Thought for Bias-Robust Vision-Language Reasoning: This paper proposes the Counterfactual Bias-Robust Reasoning dataset (CoBRa) and the Chain of Counterfactual Thought (CoCT) method. By constructing edited knowledge graphs and image content, the study evaluates and mitigates knowledge bias in large vision-language models (LVLMs), enabling models to perform step-by-step reasoning rather than relying on biased knowledge. This approach significantly outperforms existing methods on tasks requiring reasoning under knowledge bias.

🔬 Interpretability (6)¶

EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understanding: This paper proposes the EgoExo-Fitness dataset, which contains synchronized egocentric and exocentric fitness videos. It provides two-level temporal boundary annotations and innovative explainable action assessment labels (technical keypoint verification, natural language commentary, and quality scoring) and establishes five benchmark tasks.
PLOT: Text-based Person Search with Part Slot Attention for Corresponding Part Discovery: This paper proposes the PLOT framework, which utilizes a Part Discovery Module based on Slot Attention to automatically discover corresponding human body parts across modalities (image-text). Combined with Text-based Dynamic Part Attention (TDPA) to dynamically adjust the importance of each part, it thoroughly outperforms state-of-the-art (SOTA) methods on three benchmarks without requiring part-level annotations.
Improving Intervention Efficacy via Concept Realignment in Concept Bottleneck Models: This paper identifies that the low efficiency of human intervention in Concept Bottleneck Models (CBMs) stems from the independent processing of concepts during intervention, which neglects inter-concept correlations. It proposes a lightweight Concept Intervention Realignment Module (CIRM) that automatically realigns the predictions of related concepts post-intervention, reducing the number of interventions required to reach target performance by up to 70%.
DetailSemNet: Elevating Signature Verification through Detail-Semantic Integration: DetailSemNet is proposed for offline signature verification, which decouples features into detail and semantic branches via a Detail-Semantics Integrator, and introduces EMD-based local structural matching to achieve SOTA performance on multiple multilingual signature datasets.
POA: Pre-training Once for Models of All Sizes: POA proposes introducing an elastic student branch into the self-supervised self-distillation framework. Through parameter sharing and random sub-network sampling, hundreds of pre-trained models of different sizes can be produced simultaneously with a single pre-training run (e.g., directly extracting ViT-S/B from ViT-L). Each sub-network achieves SOTA performance on k-NN, linear probing, and downstream tasks.
SINDER: Repairing the Singular Defects of DINOv2: Reveals that the root cause of high-norm defect tokens in DINOv2 feature maps is the principal left singular vector of network weights (singular defect), and proposes SINDER—which repairs the defects by fine-tuning singular values on a small dataset while preserving feature quality.

📦 Model Compression (24)¶

Isomorphic Pruning for Vision Models: This paper proposes Isomorphic Pruning, which models network sub-structures as graphs and groups them by graph isomorphism. By ranking and pruning independently within each isomorphic group, it addresses the problem of incomparable importance among heterogeneous sub-structures, outperforming specially designed pruning methods on both ViTs and CNNs.
AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer: This paper proposes AdaLog, an adaptive logarithmic base quantizer that addresses the power-law distribution of post-Softmax and post-GELU activations in ViTs by replacing fixed \(\log_2\)/\(\log_{\sqrt{2}}\) quantizers with a searchable logarithmic base. Additionally, a Fast Progressive Combinatorial Search (FPCS) strategy is designed to efficiently determine quantization hyperparameters, which significantly outperforms existing ViT PTQ methods under ultra-low bit (3/4-bit) configurations.
Is Retain Set All You Need in Machine Unlearning? Restoring Performance of Unlearned Models with Out-Of-Distribution Images: Proposes SCAR (Selective-distillation for Class and Architecture-agnostic unleaRning), a retain-set-free approximate unlearning algorithm that guides the feature vectors of forgotten samples toward the nearest incorrect class distribution via Mahalanobis distance, and utilizes OOD image distillation to maintain model performance.
UNIC: Universal Classification Models via Multi-teacher Distillation: This paper proposes the UNIC framework, which integrates knowledge from multiple complementary pre-trained models into a single student model through improved multi-teacher distillation strategies (including a ladder of projectors and teacher dropping techniques), achieving cross-task universal classification.
Improving Knowledge Distillation via Regularizing Feature Direction and Norm: A novel ND loss function is proposed. By simultaneously aligning the feature direction of the student to the class mean direction of the teacher and encouraging the student to generate large-norm features, it significantly improves the performance of existing knowledge distillation methods on ImageNet, CIFAR100, and COCO.
Adaptive Compressed Sensing with Diffusion-Based Posterior Sampling: This paper proposes AdaSense, which leverages the zero-shot posterior sampling capability of pre-trained diffusion models to quantify reconstruction uncertainty, thereby adaptively selecting the optimal measurement matrix. It achieves training-free adaptive compressed sensation across multiple domains including face images, MRI, and CT, outperforming non-adaptive methods and even the optimal PCA-based non-adaptive scheme.
Adversarially Robust Distillation by Reducing the Student-Teacher Variance Gap: This paper proposes an adversarially robust knowledge distillation method based on feature distribution statistical alignment. By reducing the feature variance gap between adversarial and clean examples in the student and teacher models, the adversarial robustness of the student model is enhanced. It is discovered that robust accuracy exhibits a strong negative linear correlation with the variance gap.
Auto-DAS: Automated Proxy Discovery for Training-free Distillation-aware Architecture Search: This paper proposes Auto-DAS, an automated proxy discovery framework based on evolutionary algorithms for training-free distillation-aware architecture search (DAS). By automatically discovering optimal proxy metrics within a search space composed of student intrinsic statistics and teacher-student interaction statistics, it bypasses the limitations of hand-crafted proxies. Auto-DAS achieves SOTA ranking correlations and search accuracies across various architectures and search spaces, including ResNet, ViT, and NAS-Bench-101/201.
Bidirectional Stereo Image Compression with Cross-Dimensional Entropy Model: Proposes a bidirectionally symmetric stereo image compression framework, BiSIC, using a 3D convolutional joint codec and a cross-dimensional entropy model. It outperforms both traditional standards and existing learned methods on PSNR and MS-SSIM, while eliminating the reconstruction quality imbalance between the left and right views inherent in unidirectional approaches.
Improving Zero-Shot Generalization for CLIP with Variational Adapter: A Prompt-based Variational Adapter (PVA) is proposed to separate base and novel category samples in the latent space through a variational adapter. A divide-and-conquer strategy is adopted to process them separately, combined with residual connections to enhance the transfer capability of novel categories, achieving state-of-the-art performance on generalized zero-shot learning and cross-dataset transfer learning benchmarks.

Browse all 24 Model Compression papers →

🕸️ Graph Learning (4)¶

Synchronous Diffusion for Unsupervised Smooth Non-Rigid 3D Shape Matching: A synchronous diffusion regularization method is proposed for unsupervised non-rigid 3D shape matching. The core idea is that "synchronously diffusing the same function on two shapes should yield consistent outputs." Through this simple yet efficient regularization, the matching smoothness of existing deep functional map methods is significantly improved, achieving SOTA performance on several datasets including FAUST, SCAPE, and TOPKIDS.
Confidence Self-Calibration for Multi-Label Class-Incremental Learning: To address the overconfident predictions and false-positive errors caused by partial labels in Multi-Label Class-Incremental Learning (MLCIL), a Confidence Self-Calibration (CSC) framework is proposed. It calibrates label relationships using a Class-Incremental Graph Convolutional Network (CI-GCN) and calibrates confidence via max-entropy regularization, significantly outperforming SOTA methods on MS-COCO and VOC.
GKGNet: Group K-Nearest Neighbor Based Graph Convolutional Network for Multi-Label Image Recognition: Proposes GKGNet, the first fully graph convolutional multi-label recognition model, which dynamically constructs graph structures between labels and image regions utilizing a Group KNN mechanism, achieving SOTA performance on MS-COCO and VOC2007 with lower computational cost.
SENC: Handling Self-collision in Neural Cloth Simulation: This paper proposes SENC, which effectively addresses the cloth self-collision problem in self-supervised neural cloth simulation for the first time, using a self-collision loss based on Global Intersection Analysis (GIA) and a self-collision-aware graph neural network.

📈 Time Series (3)¶

OmniSat: Self-Supervised Modality Fusion for Earth Observation: This paper proposes OmniSat, a unified framework that fuses heterogeneous remote sensing data—including multi-spectral time-series (S2), SAR time-series (S1), and high-resolution single-temporal images (SPOT/Aerial)—into a unified representation using modality-specific encoders and cross-modal contrastive self-supervised pre-training. It outperforms all unimodal and multimodal baselines on semantic segmentation and crop classification.
Multi-person Pose Forecasting with Individual Interaction Perceptron and Prior Learning: This paper proposes IAFormer (Interaction-Aware Pose Forecasting Transformer). By designing the Interaction Perceptron Module (IPM) to evaluate the level of individual interaction with events, and introducing the Interaction Prior Learning Module (IPLM) to accumulate prior knowledge of high-frequency interaction patterns, it achieves semantic-level multi-person pose forecasting, significantly outperforming existing methods on multiple multi-person scene datasets.
Semantically Guided Representation Learning For Action Anticipation: The S-GEAR framework is proposed, which learns visual action prototypes and utilizes the semantic associations of language models to guide the geometric relationships among these prototypes. This enables the model to comprehend the semantic interconnectedness among actions, thereby enhancing action anticipation performance. S-GEAR achieves SOTA or highly competitive results across four benchmarks: Epic-Kitchens 55/100, EGTEA Gaze+, and 50 Salads.

🏥 Medical Imaging (28)¶

Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis: This paper proposes X-Gaussian, the first framework to apply 3D Gaussian Splatting (3DGS) to X-ray novel view synthesis. By designing a radiative Gaussian point cloud model (replacing spherical harmonics) and an angle-pose cuboid uniform initialization strategy (replacing SfM), it outperforms SOTA NeRF methods by 6.5 dB while achieving 73× inference acceleration with only 15% of the training time.
I-MedSAM: Implicit Medical Image Segmentation with Segment Anything: Proposed is I-MedSAM, which integrates the powerful generalization capability of SAM with the continuous space prediction advantages of Implicit Neural Representations (INR). By enhancing high-frequency boundary information with a frequency adapter and refining segmentation through uncertainty-guided sampling, it outperforms existing discrete and implicit methods with only 1.6M trainable parameters.
Alternate Diverse Teaching for Semi-supervised Medical Image Segmentation: Proposes AD-MT (Alternate Diverse Mean Teacher), which addresses the confirmation bias problem in semi-supervised medical image segmentation through random periodic alternate updating of two teacher models and an entropy-based conflict-combating strategy, comprehensively outperforming SOTA methods on ACDC, LA, and Pancreas datasets.
GTP-4o: Modality-Prompted Heterogeneous Graph Learning for Omni-Modal Biomedical Representation: The authors propose GTP-4o, an omni-modal biomedical representation learning framework based on heterogeneous graphs. It explicitly models cross-modal relationships through heterogeneous graph embeddings, utilizes a graph prompting mechanism to complete missing modalities, and designs knowledge-guided hierarchical cross-modal aggregation. It achieves SOTA on glioma grading and survival prediction tasks.
TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Data: Proposed the TIP framework, which jointly pre-trains on tabular data and images through three self-supervised tasks: masked tabular reconstruction, image-tabular matching, and contrastive learning, to learn robust multimodal representations against incomplete tabular data for downstream classification tasks.
UMBRAE: Unified Multimodal Brain Decoding: This paper proposes UMBRAE, which aligns fMRI signals with image features using a universal brain encoder and feeds them into a frozen MLLM to achieve multimodal brain decoding (description, grounding, retrieval, visual reconstruction). It innovatively introduces a cross-subject training strategy, enabling a single model to serve multiple subjects and outperform single-subject models.
OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding: Constructs OphNet, currently the largest video benchmark dataset for ophthalmic surgery (2,278 videos, 285 hours, 66 surgery types, 102 surgical phases, and 150 fine-grained steps). It supports four main tasks: surgery type recognition, phase recognition, temporal localization, and phase prediction, with a scale approximately 20 times larger than the largest prior benchmark for surgical workflow analysis.
RadEdit: Stress-Testing Biomedical Vision Models via Diffusion Image Editing: Proposes RadEdit, a diffusion-based medical image editing method that introduces a dual-mask mechanism (edit mask and keep mask) to break spurious correlations in data, generating high-quality synthetic test suites to stress-test the robustness of biomedical vision models against dataset shift.
Improving Medical Multi-modal Contrastive Learning with Expert Annotations: Proposes eCLIP, which enhances the representation quality of medical multi-modal contrastive learning without modifying the core CLIP architecture by integrating radiologists' eye-tracking gaze heatmaps as an auxiliary supervisory signal, combined with mixup augmentation and curriculum learning strategies.
Unleashing the Power of Prompt-driven Nucleus Instance Segmentation: The PromptNucSeg framework is proposed, which automatically generates nucleus center point prompts by training a prompter and fine-tunes SAM for nucleus-by-nucleus segmentation. It solves the overlapping nucleus segmentation problem by introducing neighboring nuclei as negative prompts, achieving SOTA performance on three benchmarks without complex post-processing.

Browse all 28 Medical Imaging papers →

🧬 Computational Biology (1)¶

SparseSSP: 3D Subcellular Structure Prediction from Sparse-View Transmitted Light Images: Proposes SparseSSP, an efficient framework with a mixed-dimension topology that converts 3D subcellular structure prediction into a 2D network task via a Z-axis depth-to-channel transformation, reducing imaging frequency by up to 87.5% while maintaining state-of-the-art accuracy.

⚛️ Physics & Scientific Computing (1)¶

Robust Fitting on a Gate Quantum Computer: Robust fitting is implemented on a real gate quantum computer (IonQ Aria) for the first time: a quantum circuit is proposed for 1D \(\ell_\infty\) feasibility testing, filling a critical gap in using Bernstein-Vazirani (BV) circuits to compute Boolean influence, and demonstrating how to accumulate 1D influence for high-dimensional non-linear models (e.g., fundamental matrix estimation).

🌍 Earth Science (1)¶

Semi-supervised Video Desnowing Network via Temporal Decoupling Experts and Distribution-Driven Contrastive Regularization: This paper proposes SemiVDN, the first semi-supervised video desnowing framework. By incorporating a physics-prior-guided temporal decoupling expert module and distribution-driven contrastive regularization, SemiVDN utilizes unlabeled real-world snowy videos to narrow the synthetic-to-real domain gap, outperforming existing methods on both synthetic and real-world datasets.

📡 Signal & Communications (6)¶

Defect Spectrum: A Granular Look of Large-Scale Defect Datasets with Rich Semantics: This paper constructs the Defect Spectrum dataset, providing fine-grained, semantic-rich, and large-scale multi-class defect annotations (125 defect classes, 3,518 + 1,920 images) across four industrial benchmarks. It also proposes Defect-Gen, a two-stage diffusion generator, to synthesize high-quality, diverse defect images under few-shot conditions, improving defect segmentation mIoU by up to 9.85.
RAW-Adapter: Adapting Pre-trained Visual Model to Camera RAW Images: This paper proposes RAW-Adapter, which efficiently adapts sRGB pre-trained models to camera RAW images with extremely small parameter overhead (0.2–0.8M) via an input-level adapter (learnable ISP stages) and a model-level adapter (injecting ISP intermediate features into the backbone). It achieves SOTA performance on detection and segmentation tasks under various lighting conditions, including normal, low-light, and overexposure.
PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation: This paper proposes PYRA, which generates decoupled adaptive modulation weights in parallel and modulates features of tokens to be merged using a re-activation strategy. This approach enables Vision Transformers to achieve both training efficiency (tuning only 0.4% parameters) and inference efficiency (approx. 1.7x-3.2x speedup) during downstream task adaptation, achieving comparable or superior performance to uncompressed PEFT methods.
Optimizing Illuminant Estimation in Dual-Exposure HDR Imaging: This paper proposes extracting a compact Dual-Exposure Feature (DEF) from dual-exposure HDR image pairs, based on which two ultra-lightweight illuminant estimators, EMLP and ECCC, are constructed. They achieve or exceed the performance of prior methods requiring hundreds of thousands of parameters, while using only a few hundred to a few thousand parameters.
QueryCDR: Query-based Controllable Distortion Rectification Network for Fisheye Images: The proposed QueryCDR network, utilizing a distortion-aware learnable query mechanism (DLQM) and two controllable modulation modules (CCMB/CAMB), achieves high-quality controllable rectification for fisheye images with various distortion degrees without retraining for the first time.
Unsupervised Exposure Correction: This paper proposes the first unsupervised exposure correction (UEC) method, which leverages multi-exposure sequences generated freely by ISP pipelines to train images as mutual ground truths. It designs a pixel-level transformation function with only 19K parameters to preserve image details, outperforming supervised SOTA on exposure correction and downstream edge detection.

👥 Social Computing (2)¶

Distribution-Aware Robust Learning from Long-Tailed Data with Noisy Labels: Proposes the DaSC framework, which simultaneously addresses the joint problem of long-tailed distribution and noisy labels through distribution-aware class centroid estimation (DaCC) and confidence-aware contrastive learning (SBCL + MIDL), achieving SOTA results on CIFAR and real-world noisy datasets.
GRACE: Graph-Based Contextual Debiasing for Fair Visual Question Answering: Proposes GRACE (GRAph-based Contextual DEbiasing), a graph-based contextual debiasing method. Through unsupervised context graph learning and graph-based diverse in-context example selection, it addresses the data bias inherited by large language models in knowledge-enhanced VQA systems.

📂 Others (46)¶

FisherRF: Active View Selection and Mapping with Radiance Fields Using Fisher Information: This paper proposes FisherRF, which utilizes Fisher Information to directly quantify the observed information of radiance fields model parameters. By maximizing the Expected Information Gain (EIG), it selects the optimal view. The proposed method achieves state-of-the-art (SOTA) performance across three tasks: view selection, active mapping, and uncertainty quantification. Furthermore, by leveraging sparsity and custom CUDA kernels, it achieves a view evaluation speed of 70 fps.
Improving Point-based Crowd Counting and Localization Based on Auxiliary Point Guidance: Proposes an Auxiliary Point Guidance (APG) strategy and an Implicit Feature Interpolation (IFI) module to stabilize the proposal-target matching instability in point-based crowd counting methods by explicitly generating auxiliary positive and negative samples near ground-truth points, achieving state-of-the-art results on multiple datasets.
Docling Technical Report: Docling is an open-source PDF document conversion tool that integrates DocLayNet-based layout analysis models and TableFormer table structure recognition models. It efficiently converts PDFs into structured JSON or Markdown formats on standard hardware.
An Incremental Unified Framework for Small Defect Inspection: This work proposes an Incremental Unified Framework (IUF), integrating incremental learning into unified reconstruction-based defect detection for the first time. By establishing semantic boundaries via Object-Aware Self-Attention (OASA), compressing non-primary semantic spaces through Semantic Compression Loss (SCL), and protecting old object features using an SVD-based weight update strategy, IUF achieves state-of-the-art incremental defect detection performance at both image and pixel levels on MVTec-AD and VisA.
ET: The Exceptional Trajectories - Text-to-Camera-Trajectory Generation with Character Awareness: Proposes the first camera-character trajectory dataset, E.T. (115K samples, 11M frames), extracted from real movies, alongside Director, a diffusion-based camera trajectory generation method that generates complex camera trajectories based on text descriptions and character trajectories. Additionally, designs CLaTr, a contrastive language-trajectory embedding, for trajectory generation quality evaluation.
Foster Adaptivity and Balance in Learning with Noisy Labels: Proposes the SED method, which addresses the label noise problem through an adaptive and class-balanced sample selection and re-weighting mechanism. It achieves SOTA performance on both synthetic and real-world noisy datasets without requiring prior knowledge such as predefined thresholds.
ADMap: Anti-disturbance Framework for Vectorized HD Map Construction: This paper proposes the ADMap framework, which cascadedly monitors the point sequence prediction process from both inter-instance and intra-instance levels using three modules: Multi-Scale Perception Neck (MPN), Instance Interactive Attention (IIA), and Vector Direction Difference Loss (VDDL). This effectively alleviates the point sequence jitter/jaggedness issues in vectorized HD map construction and achieves SOTA performance on nuScenes and Argoverse2.
AttnZero: Efficient Attention Discovery for Vision Transformers: This paper proposes AttnZero, the first framework to automatically discover efficient attention modules. By constructing a structured search space consisting of six computation graph types and a rich set of operators, and leveraging an evolutionary algorithm for multi-objective search, it automatically discovers linear attention formulations applicable to various ViTs. It achieves ImageNet top-1 accuracies of 74.9%/78.1%/82.1%/82.9% on DeiT/PVT/Swin/CSwin, respectively, and constructs the Attn-Bench-101 benchmark containing 2000 attention variants.
Auto-GAS: Automated Proxy Discovery for Training-Free Generative Architecture Search: This paper proposes Auto-GAS, the first training-free architecture search framework for generative adversarial networks (GANs). By automatically discovering and optimizing zero-cost proxy metrics to replace traditional training-based evaluations, it achieves a \(110\times\) search speedup while maintaining comparable generation quality with training-based methods.
PartCraft: Crafting Creative Objects by Parts: This work proposes PartCraft, which achieves part-selection-based control for text-to-image generation for the first time. Users can "pick" different parts (such as a bird's head, wings, and body) from various objects, and the model naturally combines them into a novel and structurally coherent creative object.

Browse all 46 Others papers →