Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMs

Conference: ECCV 2026
arXiv: 2606.20177
Code: https://hhc1997.github.io/RS-Neg-and-NeFo/
Area: Multimodal VLM
Keywords: Negation Comprehension, Remote Sensing Multimodal Large Language Models, Test-Time Adaptation, Benchmark Dataset, Truth-Value Inversion

TL;DR

Addressing the severe performance degradation of remote sensing MLLMs under negative queries, this paper constructs the first remote sensing negation comprehension benchmark, RS-Neg (22K samples covering four tasks from regional to scene levels), and proposes a test-time adaptation method, NeFo. By explicitly modeling the logical role of negation (truth-value inversion operator) as an optimization objective, NeFo significantly enhances the negation comprehension capabilities of multiple base models using only around 5% of unlabeled test samples, demonstrating strong generalization on unseen tasks.

Background & Motivation

Multimodal large language models (MLLMs) have made remarkable progress in remote sensing tasks, but their negation comprehension capability remains largely unexplored. Negation is a fundamental component of natural language—human infants at 18 months can understand new objects through negative sentences—yet even 70B parameter MLLMs suffer from severe performance degradation when processing negation. This issue is particularly critical in remote sensing scenarios: disaster monitoring requires answering "how many buildings are not flooded," and transport planning requires locating "routes not covered by ice." Misunderstanding negation can directly lead to outputs contrary to user intentions, posing severe safety hazards.

Existing negation evaluation works (such as NegVQA, GaslightingBench, NegBench, etc.) focus on general-domain natural images. Their data generation pipelines cannot directly adapt to the small-sized objects (typically smaller than 32x32 pixels) and fine-grained attribute/state-level reasoning demands in remote sensing images (e.g., "roads not covered by ice" involves state-level negation rather than simple object absence). This raises two core problems: (1) How to comprehensively evaluate the negation comprehension ability of remote sensing MLLMs? (2) How to enhance the robustness of MLLMs under negative queries?

Core Idea: To explicitly encode the logical role of negation in linguistics—a unary operator that flips the truth value of a proposition—as a test-time optimization objective. By maximizing the difference in outputs between a negative sentence and its de-negated variant, the model is driven to focus on negative semantics, while using the original model as a teacher to apply knowledge distillation only to affirmative queries to avoid consolidating errors.

Method

Overall Architecture

This paper contains two core contributions. The first part is the automated construction pipeline of the RS-Neg benchmark: starting from existing remote sensing image-text datasets, an LLM is used to extract objects, attributes, and states from the descriptive texts to generate candidate negative concepts. Then, these negative elements are verified as indeed absent in the images through Monte Carlo Tree Search (MCTS)-based Dynamic Focus (DyFo). Finally, the filtered negative descriptions are converted by the LLM into four types of task data: VQA, MCQ, Visual Grounding, and Scene Classification, totaling 22,464 samples. The second part is the NeFo test-time adaptation method: given a test query containing negation, a lexical rule mask is first used to mask out negation words to obtain a de-negated variant. Then, the LoRA parameters are jointly optimized through a truth-value inversion loss (maximizing the KL divergence of outputs between the two variants) and a knowledge retention loss (performing KL distillation and entropy minimization on the de-negated variant), enhancing the model's sensitivity to negation in a completely self-supervised manner during test time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Test Sample (v, q)<br/>q contains negation"] --> B["Negation Masking<br/>q → q^mask"]
    B --> C["Original MLLM Forward<br/>P(y|v,q) and P(y|v,q^mask)"]
    C --> D["Truth-Value Inversion Loss<br/>Maximize KL(P_q || P_qmask)"]
    C --> E["Knowledge Retention Loss<br/>KL Distillation + Entropy Minimization"]
    D --> F["Joint Loss L = L_inv + L_ret"]
    E --> F
    F --> G["LoRA Update Attention Layers<br/>Parameter-Efficient Adaptation"]

Key Designs

1. RS-Neg automated negation data pipeline: Solving the negation verification challenge of small targets in remote sensing

Existing negation data generation methods rely on global image perception, which almost completely fails when facing targets smaller than 32x32 pixels in remote sensing images—the paper's Appendix B shows that even Claude 4.5-Opus makes mistakes in such scenarios. The pipeline in this paper solves this problem in three steps. First, the LLM parses the description texts of image-text pairs, extracts the mentioned objects, attributes (color/shape), and states (action/condition), and then generates candidate negative concepts that are "semantically related but do not appear in the descriptions". Second, DyFo (Dynamic Focus) is used for visual verification: for object-level negation, Grounding DINO is directly used for existence detection; for attribute/state-level negation, they are combined with the corresponding object to form the root node of the MCTS search, alternately performing semantic focusing (visual experts locate and crop target regions) and spatial zooming (gradually zooming in on cropped regions for instance-level inspection). Qwen2.5-VL-7B scores each search node and guides the search direction. Third, the LLM converts the filtered negative concepts into task-specific formats: VQA constructs yes/no questions (e.g., "Does this image show the {pos_object} without {neg_object}? Yes"), MCQ consists of one correct description and three distractors (with at least one containing negation), Grounding keeps the original bbox but modifies the description to introduce negation, and the classification task uses rules to add negative labels as distractors. LLM rephrasing increases linguistic diversity, preventing the model from overestimating fixed negation templates.

2. Truth-value inversion loss: Encoding logical semantics of negation as optimization signals

From a linguistic logic perspective, negation is a unary operator that flips the truth value of a proposition—the ideal outputs of "the area is flooded" and "the area is not flooded" should be completely opposite. NeFo formalizes this intuition into the truth-value inversion loss: given a negative query \(q\), lexical rules are first used to mask out negation words (English negation words are limited: no, not, without, etc.) to obtain the de-negated variant \(q^{mask}\), and then the KL divergence of the output distributions between the two is maximized:

\[\mathcal{L}_{inv} = -\max\left(\mathbb{D}_{\text{KL}}[\mathcal{P}_{\Theta}(y|v,q) \| \mathcal{P}_{\Theta}(y|v,q^{mask})], \alpha\right)\]

where \(\alpha\) is a truncation parameter (set to 0.4) to ensure training stability. This loss amplifies the output difference arising solely from the presence or absence of negation words, forcing the model to concentrate its attention on negation-related semantic information. Unlike methods requiring manual annotation or teacher models, this signal is completely self-supervised—only the positions of negation words in the query need to be known.

3. Knowledge retention loss: Preventing catastrophic forgetting during test-time adaptation

The core risk of TTA is that the model may lose pre-trained knowledge while adapting to the test distribution. NeFo employs a dual strategy to circumvent this issue. First, the original model (with frozen parameters) is used as a teacher to apply KL distillation only to the de-negated variant \(q^{mask}\), rather than the query containing negation—because the original model itself might produce noisy outputs on negative queries, and distilling its negation predictions would instead consolidate errors. Second, an entropy minimization auxiliary objective is added to make the model's predictions on affirmative inputs more confident, while providing a more stable control anchor for \(\mathcal{L}_{inv}\):"

\[\mathcal{L}_{ret} = \beta \mathbb{D}_{\text{KL}}[\mathcal{P}_{\Theta^o}(y|v,q^{mask}) \| \mathcal{P}_{\Theta}(y|v,q^{mask})] + \gamma \mathcal{H}(\mathcal{P}_{\Theta}(y|v,q^{mask}))\]

where \(\beta\) and \(\gamma\) are both balancing coefficients (default 0.4). The final joint optimization \(\min_{\tilde{\Theta}} \mathcal{L} = \mathcal{L}_{inv} + \mathcal{L}_{ret}\) only updates the LoRA parameters (rank=8, scale=16) of the attention layers in the LLM module. The number of trainable parameters is only about 2.5M, which is extremely lightweight.

4. Lightweight online adaptation strategy: Practical deployment solution under minimal data requirements

To address the rapid adaptation needs in real-world remote sensing deployment scenarios, NeFo adopts an online learning mode: model parameters are fine-tuned via LoRA using only a minimal number of test samples (150 for VQA, 300 for MCQ, accounting for about 5% of the entire test set), and then the performance on the entire test set is reported. Training utilizes the AdamW optimizer with a batch size of 1. The practical value of this design lies in: labeling data in remote sensing scenarios is extremely expensive, whereas NeFo does not rely on any manual annotations—it accomplishes adaptation using only the unlabeled test queries themselves, making it highly suitable for resource-constrained scenarios such as edge devices or drone remote sensing. Cross-scale experiments further validate its universality, showing consistent improvements across the Qwen3-VL series from 2B to 32B.

A Complete Example

To illustrate the workflow of NeFo, we refer to a specific sample from RS-Neg VQA. The input is a remote sensing image \(v\) after a flood disaster and the query \(q\): "Is there any building not flooded?"

Step 1: Lexical Masking. Identify and remove the negation word "not" to obtain the de-negated variant \(q^{mask}\): "Is there any building flooded?".

Step 2: The original MLLM (such as Qwen2.5-VL-7B) performs forward inference on both variants. Suppose the output distribution for the negative query \(q\) is "Yes" with 0.35 probability, "No" with 0.65 probability; the distribution for the de-negated query \(q^{mask}\) is "Yes" with 0.82 probability, "No" with 0.18 probability. Ideally, they should be opposite—if buildings are indeed flooded, \(q^{mask}\) should output "Yes" and \(q\) (which asks "not flooded") should output "No".

Step 3: Loss Calculation. \(\mathcal{L}_{inv}\) calculates the KL divergence between the two distributions and takes its negative to maximize the discrepancy (protected by the \(\alpha=0.4\) truncation); \(\mathcal{L}_{ret}\) aligns the current model's prediction for \(q^{mask}\) with the frozen original model's prediction distribution while minimizing entropy to sharpen predictions.

Step 4: Gradients are backpropagated solely through LoRA low-rank matrices, and the adapter parameters of the attention layers are updated. After around a few iterations on 150 VQA samples, the model learns to systematically flip its output logic when encountering negation words, without forgetting the correct response to affirmative queries.

Loss & Training

The total loss is the sum of two terms: \(\mathcal{L} = \mathcal{L}_{inv} + \mathcal{L}_{ret}\). Within this, \(\mathcal{L}_{inv}\) includes the truncation parameter \(\alpha=0.4\) to prevent excessively large KL divergence from causing unstable training; in \(\mathcal{L}_{ret}\), \(\beta=0.4\) controls the strength of knowledge retention, and \(\gamma=0.4\) controls the strength of entropy minimization. Key training strategies: (1) Completely self-supervised with zero manual annotations; (2) Original model parameters are frozen during TTA, and only LoRA parameters are updated (rank = 8, scale = 16, ~2.5M); (3) For MCQ data, each option is converted to a yes/no format of "Does this caption describe the image?", and only samples containing negation words participate in training; (4) The learning rate and \(\gamma\) tuning settings for each base model are detailed in Appendix E.

Key Experimental Results

Main Results

In RS-Neg VQA and RS-Neg MCQ, NeFo is compared with three SOTA TTA methods (Tent, SAR, TLM). The table below displays the overall accuracy of four representative base models under negative queries.

Base Model	Method	RS-Neg VQA (total)	RS-Neg MCQ (total)
Qwen2.5-VL-7B	Baseline	74.96	54.29
	+ Tent	73.09	55.77
	+ SAR	65.21	55.52
	+ TLM	69.30	49.43
	+ NeFo	79.52	65.46
Qwen3-VL-7B	Baseline	71.47	57.30
	+ NeFo	75.42	64.55
RS-LLaVA-7B	Baseline	73.15	23.72
	+ NeFo	74.75	24.46
GeoReason-7B	Baseline	67.30	45.59
	+ NeFo	69.76	53.39

Key Findings: (1) Existing TTA methods generally lead to performance degradation under negative queries because they self-reinforce erroneous predictions; (2) NeFo achieves consistent improvements across all base models, with the MCQ improvement of Qwen2.5-VL reaching up to 11.17%; (3) By negation type, object-level negation achieves the largest improvement (e.g., Qwen2.5-VL VQA object +6.07), while state-level negation shows more modest gains (+1.22) because state-level negation requires contextual reasoning and is inherently more difficult.

Zero-shot generalization results further prove that NeFo learns transferable negation comprehension capabilities. Taking Qwen2.5-VL as an example, the model TTA-tuned on MCQ data improves from 60.37% to 66.63% (+6.26%) on the unseen RS-Neg classification task, from 54.03 to 63.37 (+9.34%) in terms of [email protected] on RS-Neg Grounding, and from 37.32 to 57.47 (+20.15%) on the real-world flood disaster dataset FloodNet VQA. This huge leap directly reflects the practical value of negation comprehension in real-world disaster relief scenarios.

Ablation Study

Using Qwen2.5-VL-7B as the base, the three core components of NeFo are abated one by one: truth-value inversion loss (TI), knowledge retention loss (KR), and entropy minimization (EM).

Configuration	RS-Neg VQA (total)	RS-Neg MCQ (total)	Key Findings
Base (Qwen2.5-VL)	74.96	54.29	Unadapted baseline
NeFo w/o TI	74.05	57.78	Removing TI yields almost no gain in VQA; TI is the core driver for VQA
NeFo w/o EM	76.84	29.06	Removing EM is catastrophic for MCQ (-36.40); entropy minimization provides a stable anchor for affirmative predictions
NeFo w/o KR	77.82	22.71	Removing KR is the most fatal for MCQ (-42.75); the model overfits and suffers catastrophic forgetting
NeFo (full)	79.52	65.46	Three components work synergistically

Key Findings: (1) Different tasks exhibit distinctly different sensitivities to each component—TI is most critical in VQA (almost no improvement after removing it), while KR and EM are indispensable in MCQ; (2) MCQ involves mixed discrimination of affirmative and negative descriptions. After removing KR, the model loses its ability to discriminate affirmative descriptions during TTA, and performance collapses to 22.71%, far below the baseline of 54.29%; (3) Even when keeping KR and TI, removing EM still causes MCQ to plummet from 65.46% to 29.06%, because the model's predictions on affirmative inputs become blurred, making \(\mathcal{L}_{inv}\) lose a stable control anchor.

Hyperparameter analysis shows that NeFo is insensitive to \(\beta\) (VQA accuracy fluctuates within a narrow range of 73.76%-74.75%) because a very small \(\beta\) is sufficient to prevent forgetting—the knowledge retention loss remains close to zero throughout TTA. Conversely, it is more sensitive to \(\gamma\): MCQ decreases by 7.28% when \(\gamma=0.1\), but performs stably in the 0.7-1.0 range. Cross-scale experiments show that the smallest model Qwen3-VL-2B equipped with NeFo improves VQA from 64.61 to 74.51 (+9.9%), approaching the 32B baseline (77.70), demonstrating that NeFo has extremely high practical value for resource-constrained scenarios.

Key Findings

• TI is the core driver of VQA: Removing TI drops VQA almost back to the baseline level (74.05 vs 74.96), indicating that simple distribution adaptation cannot replace explicit negation logical modeling.
• KR is essential for the survival of MCQ: In MCQ, the model needs to handle both affirmative and negative descriptions simultaneously. Without KR, the model overfits to negative data and completely loses its discriminative power on affirmative descriptions, validating the paper's core thesis on catastrophic forgetting.
• Smaller models benefit more significantly: Qwen3-VL-2B jumps from 64.61 to 74.51 (+9.9%) on VQA via NeFo, approaching the baseline performance of the 32B model, proving NeFo is highly attractive for edge device deployment.
• Optimal range exists for TTA data volume: On MCQ, performance continuously improves (+17.4%) when training samples increase from 50 to 600, but collapses due to overfitting when increased to 700, requiring a smaller learning rate to mitigate.

Highlights & Insights

• Directly translating linguistic definitions into optimization objectives is the most ingenious design of this paper. The assertion that negation is a truth-value inversion operator arises from classic linguistic theory by Horn (2001), but it has never been encoded as a loss function for MLLMs. This "borrowing concepts for losses" idea can be transferred to enhance the comprehension of other logical semantics (such as quantifiers all/some, modals must/may).
• Performing distillation only on the de-negated variant rather than the original query is a seemingly subtle but crucial design choice. The original model itself is prone to error on negative queries; distilling its negation predictions is equivalent to consolidating errors. This insight is simple yet easily overlooked, and it is the root cause for MCQ collapsing to 22.71% in the w/o KR ablation.
• Minimalism of lexical masking: Using only a negation word dictionary for masking without complex NLP parsers proves that, in languages with limited negation words (such as English), a minimalist solution is sufficient to support effective self-supervised signals.
• The combination of LoRA + TTA provides a practical path for remote sensing deployment: The settings of 2.5M parameters, 150 unlabeled samples, and a batch size of 1 allow the method to complete adaptation almost in real-time on drones or edge devices, without needing to upload data to the cloud.

Limitations & Future Work

• Lexical masking relies on an English negation vocabulary (no, not, without, etc.) and cannot handle implicit negation (such as rarely, hardly, few) or complex negation morphologies across languages. More complex masking strategies might be needed in languages like Chinese with more diverse negative expressions.
• The improvements in state-level negation are consistently smaller than those in object-level negation (VQA average +2.46% vs +6.07% for objects), exposing the method's inadequate modeling of negation types that require contextual reasoning.
• Experiments are mainly performed through LoRA fine-tuning on 7B-8B parameter scale models (still effective but with narrowed gains when extended to 32B), and behavior under 70B+ models or full parameter tuning is yet to be validated.
• The RS-Neg benchmark data are entirely automatically generated by LLMs, lacking manual annotation quality verification. Negation queries generated by LLMs might contain unnatural expressions or logical flaws, and the authors do not report quality data from manual spot checks.
• NeFo requires online learning during testing, meaning it cannot take effect in a zero-shot manner, posing a latency constraint for real-time applications requiring millisecond responses. Future work can explore consolidating the LoRA parameters learned in TTA into the base model to achieve "learn once, benefit permanently".
• The combined effect of TTA and training-stage data augmentation (such as adding negative samples in pre-training or SFT stages) remains unexplored, though the two may be complementary.

Related Work & Insights

• vs NegVQA (Zhang et al., 2025): NegVQA constructs binary-choice questions to evaluate negation comprehension in general domains, with data generation relying on global image perception. This paper introduces DyFo to handle small targets specifically in remote sensing scenarios and covers four categories of tasks, providing a more comprehensive evaluation dimension.
• vs NegCLIP / CoNCLIP / NegBench: These works improve the negation comprehension of CLIP-style embedding models through post-training on large-scale negative image-text pairs. NeFo target generative MLLMs and requires no training annotations, using the TTA paradigm to avoid the expensive costs of remote sensing data annotation.
• vs NEAT (Han et al., 2025): NEAT also adopts the TTA concept but is only applicable to alignment correction of vision-language embedding models. NeFo extends TTA to autoregressive generative MLLMs and is the first to explicitly encode the logical semantics of negation as an optimization objective rather than just doing distribution alignment.
• vs DyFo (Li et al., 2025): The MCTS visual search of DyFo is directly used in the RS-Neg construction pipeline for negative concept verification. This is a neat case of leveraging existing tools, reflecting the engineering wisdom of "combining existing tools to solve new problems".

Rating

• Novelty: ⭐⭐⭐⭐ It systematizes the remote sensing negation comprehension problem for the first time (benchmark + method). The design of the truth-value inversion loss is supported by linguistic theory and is practical and effective, though it represents a combinational innovation overall ("known components (TTA + LoRA + KL) + new scenario + new loss formula").
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive and credible, covering four base models, three TTA baselines, full ablations (component-by-component + hyperparameters), cross-model scale scaling (2B-32B), data scale scaling analysis, and three categories of zero-shot generalization tests.
• Writing Quality: ⭐⭐⭐⭐ The motivation is clear (the 18-month human infant negation entry point is compelling), the structure is reasonable, and the core formulas are clearly defined. Some method details (such as the specific branch strategy of DyFo search) are not fully elaborate in the main text and require checking the appendix.
• Value: ⭐⭐⭐⭐⭐ Accurate comprehension of negation is a safety-critical bottleneck for deploying remote sensing MLLMs in real-world scenarios (disaster response, military reconnaissance, urban planning). This paper provides actionable benchmarking tools and practical lightweight solutions, making the TTA scheme with 5% data and 2.5M parameters highly attractive for resource-constrained scenes.

Related Papers

• [ECCV 2026] RSICCLLM: A Multimodal Large Language Model for Remote Sensing Image Change Captioning
• [CVPR 2026] VLM4RSDet: Collaborative Optimization with Vision-Language Model for Enhancing Remote Sensing Object Detection
• [ICLR 2026] PhysLLM: Harnessing Large Language Models for Cross-Modal Remote Physiological Sensing
• [ICLR 2026] CityLens: Evaluating Large Vision-Language Models for Urban Socioeconomic Sensing
• [NeurIPS 2025] CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Models