WeatherReasonSeg: A Benchmark for Weather-Aware Reasoning Segmentation in Visual Language Models¶

Conference: ECCV 2026
Paper: ECCV Official Page
Area: Multimodal VLM / Reasoning Segmentation
Keywords: adverse-weather robustness, reasoning segmentation, mask-guided queries, visual grounding, evaluation benchmark

TL;DR¶

WeatherReasonSeg combines controlled synthetic weather with five-dimensional reasoning queries for real adverse-weather images into 44,721 image-query pairs, revealing failures in both mask prediction and language-conditioned target grounding, with the best evaluated reasoning model reaching only 28.9 gIoU on real nighttime scenes.

Background & Motivation¶

Reasoning segmentation goes beyond recognizing a category such as a bus: it infers the intended object from its function, context, or a user's requirements, then predicts a pixel mask. ReasonSeg and LISA advanced this task, but success on clear images does not establish reliability under rain or snow occlusion, fog-induced contrast loss, or nighttime illumination. Weather can damage object boundaries and remove the structural or contextual cues needed for reasoning, and either failure ultimately produces an incorrect mask.

Existing weather datasets usually provide semantic segmentation labels but lack language questions requiring target inference; existing reasoning segmentation benchmarks lack controlled weather variation. Real-weather testing alone makes severity effects difficult to isolate, while synthesis alone cannot represent nonuniform scattering, occlusion, and illumination changes. The paper therefore combines two data sources rather than proposing a new restoration network or segmentation training algorithm.

It further organizes questions about the same target into five semantic dimensions to examine whether failures are closer to missing visible structure or failing to map a requirement to an object. Core Idea: measure degradation sensitivity with synthetic weather while retaining the original queries, then assess practical pixel-grounded reasoning using real-weather images, target masks, and five-dimensional queries.

Method¶

Overall Architecture¶

WeatherReasonSeg is a dataset and evaluation protocol, not a newly trained segmentation model. An evaluated system receives a degraded image and a natural-language question and predicts a target mask; additional diagnostics evaluate bounding boxes without SAM and a reference upper bound obtained by giving SAM2 ground-truth spatial locations.

Dataset construction follows two independent routes. The synthetic route starts from ReasonSeg images, depth maps, and original queries and creates rain, snow, and fog at different severities. The real route extracts targets from ACDC images and pixel annotations, generates five types of mask-guided questions, and filters them through human-model verification. These routes are not sequential inference modules, so the dataset structure is not presented as a neural-network flowchart.

The dataset contains 44,721 image-query pairs, not 44,721 independent images. Its synthetic component contains 8,811 pairs, with 2,937 each for rain, snow, and fog. Its real component contains 35,910 pairs: 8,680 rainy, 9,545 snowy, 7,465 foggy, and 10,220 nighttime pairs. Multiple questions can refer to the same object, so the pair count should not be equated with visual-scene diversity.

Key Designs¶

1. Controlled weather synthesis: change visual visibility while retaining the language task

The synthetic component uses ReasonSeg and pairs degraded images with the original queries rather than regenerating questions. Comparisons between clear and corrupted images therefore preserve the language task, making changes more attributable to visual input degradation. Depth maps are also supplied to support physically meaningful weather generation instead of simply overlaying unstructured noise.

The three generators model rain-streak accumulation, snowflake scattering and occlusion, and atmospheric light scattering in fog. Parameters such as rain density, snow-particle distribution, and visibility distance control degradation; expert knowledge and VLM-assisted calibration inform severity settings. Although the generation mechanism allows continuous control, the experiments report only Light, Moderate, and Heavy levels, not a fully measured continuous severity curve.

This design supports stability analysis across severity levels within a task, but synthetic images still do not reproduce all real-weather effects. The main paper delegates generator implementations and parameter settings to supplementary material, which is absent from the supplied cache. Specific scattering equations, parameter ranges, or depth estimators therefore cannot be supplied from this source.

2. Mask-guided five-dimensional queries: fix the target before varying the reasoning perspective

The real component uses ACDC images and pixel annotations covering rain, fog, snow, and nighttime. A target mask is extracted and supplied alongside the image to a language model, which constructs questions about that known target. The mask constrains question-answer alignment during dataset creation; it is not an answer prompt routinely given to the evaluated VLM. Figure 2 labels the generator as Qwen, but the main text does not specify its version.

The five dimensions are not arbitrary paraphrases. Function asks about an object's intrinsic purpose, such as a truck transporting goods. Application asks about its usage context, such as urban delivery. Structure concerns visible features, such as a cargo compartment. Relationship concerns spatial or interactive relations with surrounding entities. Requirement starts from a user's goal and asks which object can satisfy it, such as transporting bulky goods.

Function is object-centric purpose recognition, whereas Requirement is goal-centric target selection: they are related but not interchangeable. Holding the object fixed reduces confounding from changing target categories and probes reliance on observable attributes versus abstract context. Nevertheless, these dimensions are not a strict difficulty hierarchy, and a Structure question is not necessarily easier than a Requirement question.

3. Single-target consistency filtering: prevent mismatches between questions and pixel answers

Human annotators and the language model jointly verify initial queries. The rules remove questions that merely enumerate objects or components without functional or contextual meaning; reject groups whose five questions do not identify the same physical object; retain groups with an explicit, visually identifiable target entity; and discard questions referring to absent, visually unclear, or semantically ambiguous objects.

The essential constraint is not linguistic complexity but agreement that every question in a group should yield the same target mask. Otherwise, a model could answer a reasonable question and still be penalized because the annotation refers to another object. This matters especially in real adverse weather: an annotator's knowledge of a category does not guarantee that the degraded image retains enough evidence to identify it.

Filtering also defines the benchmark's scope. It emphasizes answerable, localizable targets rather than abstention or uncertainty about completely invisible objects. The main text does not report retention rates, inter-annotator agreement, or full prompts, so the stated rules should not be mistaken for a quantified guarantee of annotation quality.

4. Layered evaluation: distinguish final masks, grounding diagnostics, and a prompted upper bound

The paper evaluates six frameworks: GLaMM, Grounded-SAM, PixelLM, LISA, Seg-R1, and Seg-Zero. Full systems measure question-to-mask performance. Seg-Zero* removes SAM and evaluates bounding boxes to examine grounding degradation before mask decoding. SAM2 receives ground-truth spatial prompts such as bounding boxes, measuring segmentation when the target location is already known. These settings differ in both available information and output space and should not be treated as a fully controlled model ranking.

The metrics are gIoU and cIoU. Here gIoU means the arithmetic mean of per-image IoU, not Generalized IoU from object detection; cIoU divides the accumulated intersection by the accumulated union. For predicted and ground-truth regions \(P_i\) and \(G_i\) in evaluation sample \(i\), the textual definitions can be expressed as:

\[ \mathrm{gIoU}=\frac{1}{N}\sum_{i=1}^{N}\frac{|P_i\cap G_i|}{|P_i\cup G_i|},\qquad \mathrm{cIoU}=\frac{\sum_{i=1}^{N}|P_i\cap G_i|}{\sum_{i=1}^{N}|P_i\cup G_i|}. \]

Tables report these metrics on a percentage scale. gIoU weights samples equally, whereas cIoU is more influenced by samples with larger union areas, so their trends need not coincide. The bounding-box diagnostic uses box regions; subtracting its score from a full model's mask score does not isolate SAM's contribution.

Loss & Training¶

The paper introduces no new loss function and does not describe a complete procedure for training a weather-adaptive model on WeatherReasonSeg. Its focus is dataset construction and evaluation of existing frameworks. Discussions of reinforcement learning or decoupled segmentation in related work should not be interpreted as an additional RL training method implemented by this paper.

Implementation details primarily list models and metrics, without fully specifying checkpoints, decoding hyperparameters, compute budgets, or training-overlap checks. The synthetic table distinguishes val and test, but the supplied cache lacks enough detail to reconstruct every split and sample mapping. Reproduction requires checking the official data release rather than guessing these settings.

Key Experimental Results¶

Main Results¶

The following excerpt from the paper's Table 3 reports real-weather gIoU, where higher is better. SAM2 is explicitly identified as a ground-truth-location reference, not a competitor evaluated with identical input information.

Method	Fog	Rain	Snow	Night
SAM2, ground-truth-location reference	82.2	80.1	81.5	74.8
PixelLM	2.8	2.0	1.9	2.6
GLaMM	13.1	9.3	11.5	9.8
Grounded-SAM	23.7	18.8	15.3	11.4
LISA-7B	18.9	11.5	14.9	14.7
Seg-R1	30.9	26.7	29.4	18.5
Seg-Zero-7B	40.6	35.1	36.8	28.9

Seg-Zero-7B is the strongest full reasoning segmentation model in this table, yet its nighttime score is 45.9 percentage points below the prompted SAM2 reference. This gap highlights the importance of obtaining the correct target location, but it does not attribute all error strictly to reasoning: the comparison also changes prompt information and model architecture.

Ablation Study¶

The paper does not introduce new modules and remove them in architectural ablations. The following controlled severity analysis comes from Table 2 and consistently uses Seg-Zero-7B test mask gIoU, without mixing validation or bounding-box scores.

Synthetic weather	Clean	Light	Moderate	Heavy	Clean-to-Heavy drop
Fog	57.5	53.9	50.9	49.9	7.6
Rain	57.5	54.0	51.5	47.8	9.7
Snow	57.5	53.1	45.7	45.7	11.8

The final column gives percentage-point differences calculated from these entries. Snow produces the largest final drop in this slice, rain drops more than fog, and Moderate and Heavy snow tie. The appropriate conclusion is an overall degradation trend, not a universal weather ordering or strictly monotonic decline for every model.

The next excerpt from Table 4 analyzes semantic dimensions under real rain; all entries are cIoU:

Query dimension	Seg-Zero	LISA	Seg-R1
Function	41.2	18.7	38.8
Application	31.7	14.3	36.2
Structure	35.5	17.1	34.5
Relationship	36.8	16.7	36.7
Requirement	31.8	15.6	34.2

Key Findings¶

Seg-Zero's rain cIoU differs by 9.4 percentage points between Function and Requirement. This compares query types under the same weather, not degradation relative to clear images.
In Table 2, Seg-Zero drops from 65.3 on Clean to 47.7 on Heavy Snow in validation bounding-box* evaluation, a 17.6-point decline. Grounding is already affected, but box IoU remains a behavioral proxy rather than a direct measure of internal chain-of-thought correctness.
Monotonic decline does not hold entry by entry: Seg-R1 fog test gIoU rises from 46.9 at Light to 47.8 at Moderate before falling to 43.4 at Heavy. This exception does not overturn the broad severe-weather degradation, but it limits the strength of the claim.
The prose and tables disagree in places: the prose gives Grounded-SAM Heavy Fog cIoU as 10.8, while Table 2 reports 12.3 for fog-test and 10.8 for snow-test. Seg-Zero snow cIoU is 39.72 in the prose and 39.7 in Table 2. This note retains the table values and explicitly flags the discrepancies.

Highlights & Insights¶

Synthetic and real data play different roles: the former holds queries fixed to probe severity sensitivity; the latter exposes failures arising from real imaging and urban scenes. Their combination is more diagnostic than one weather-averaged score.
Mask-first query generation connects language questions to pixel answers. Five-dimensional questions attach different semantic demands to the same target, supporting studies of how visibility and query abstraction jointly affect grounding.
The ground-truth-location reference suggests a reusable evaluation practice: report both end-to-end masks and segmentation when target locations are known. This is useful only when the extra information is explicit, rather than used to declare the segmentation module faultless.

Limitations & Future Work¶

The authors acknowledge an urban-driving bias with many transportation-related objects. Claims about natural environments, nontraffic objects, and outdoor work settings need additional data.
The main text omits weather parameters, complete query prompts, filtering statistics, and sufficient reproduction settings; the supplied cache lacks supplementary material. These gaps constrain independent reproduction and should not be filled with speculation.
Table 3 reports Seg-Zero rain gIoU of 35.1, whereas all five rain dimensions in Table 4 lie between 31.5 and 33.8. If these dimensions exhaustively partition the same sample set, per-sample gIoU cannot aggregate to 35.1. The main text does not explain whether subsets or protocols differ, so the two tables are not forcibly reconciled.
Dimension comparisons lack matching clear real-image query controls, and SAM2 receives ground-truth locations. Future evaluation could hold models and targets fixed, replace ground-truth versus predicted location prompts stage by stage, and introduce clear-degraded pairs to better separate visual information loss, grounding errors, and mask errors.
Removing unclear targets supports answerability but does not evaluate safe abstention under extreme weather. Visibility labels, confidence calibration, and selective-prediction metrics are possible extensions, not results already demonstrated by this paper.

Compared with ReasonSeg / LISA: the task still maps implicit language intent to pixel masks, but adds controlled visual degradation and structured queries for real adverse weather. Its contribution is evaluation coverage and data construction, not a new segmentation-token interface.
Compared with ACDC: it retains real adverse-condition images and pixel annotations while adding queries requiring target inference. This extends category perception to language-conditioned target selection while retaining the driving-domain bias.
Relation to Seg-Zero / SAM2: Seg-Zero is an evaluated reasoning segmentation baseline; SAM2 provides a ground-truth-location reference. Their gap motivates separate grounding diagnostics, not a claim that this paper introduces a stronger model.
Resources: the paper links the WeatherReasonSeg dataset as a reproduction entry point; the supplied cache provides no identifiable code-repository link for this work.

Rating¶

Novelty: 4/5. The intersection of adverse weather and pixel-grounded reasoning is a clear gap, and dual-source data with five-dimensional queries is useful.
Experimental Thoroughness: 3/5. Six frameworks, multiple weather types, and severity levels are covered, but reproduction detail, controlled attribution, and statistical consistency remain insufficient.
Writing Quality: 3/5. Dataset construction is clear, but monotonicity claims, some table references, and aggregate results require clarification.
Value: 4/5. A practical stress test for reasoning segmentation in real environments, especially for studying grounding robustness.