Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels¶
Conference: ECCV 2026
Paper: Official page · Paper PDF
Code: https://github.com/venkanna37/align-and-segment
Area: Remote Sensing
Keywords: Building segmentation, label misalignment, spatial registration, self-supervised consistency, geometric augmentation
TL;DR¶
Align and Segment (AnS) jointly learns building segmentation and geometric label correction, using self-supervised consistency and paired geometric augmentation to prevent memorized offsets, raising segmentation IoU from 0.39 for naive joint training to 0.78 on systematically misaligned Las Vegas data without clean aligned training labels.
Background & Motivation¶
Satellite imagery and building polygons are often collected independently: an image may come from a satellite platform, while its annotations come from OpenStreetMap (OSM) or an earlier mapping effort. Their disagreement is not necessarily isolated pixel noise; an entire neighborhood can be shifted or rotated relative to the image. This structured error is especially problematic when its direction is consistent across training examples. A sufficiently capable segmentation network can learn to place every building beside its visible roof and still achieve a low training loss.
Adding a registration network sounds like a natural remedy: estimate the offset, correct the labels, and train segmentation against the corrected targets. Yet joint optimization does not automatically separate these responsibilities. Early in training, registration is near the identity transformation, so the segmentation network learns the original misaligned targets; once it does so, correcting the offset is no longer necessary to reduce the loss. Earlier approaches often require some clean labels or assume approximately unbiased misalignment, neither of which is guaranteed for real map data.
AnS therefore focuses less on a new segmentation backbone than on what forces each network to learn its intended task. It creates registration exercises with known transformations from the existing masks and changes the direction of dataset offsets through paired geometric augmentation. Core idea: give registration an independent geometric training signal while disrupting the fixed offsets that segmentation could memorize, enabling the two networks to recover alignment without clean aligned training labels.
Method¶
Overall Architecture¶
Training takes a remote-sensing image and its corresponding misaligned binary building mask, and produces both a segmentation prediction and a geometrically corrected version of the supplied labels. Paired geometric augmentation processes the input pair; joint segmentation–registration learning couples a segmentation network, SNet, with a transformation network, TNet; absolute transformation consistency supplies an additional self-supervised constraint on TNet. The consistency branch is a parallel training exercise, not an inference-time post-processing stage.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Image and misaligned mask"] --> B["Paired geometric<br/>augmentation"]
B --> C["Joint segmentation–registration<br/>learning: SNet and TNet"]
B --> D["Absolute transformation<br/>consistency"]
D -->|Constrains TNet| C
C --> E["Building prediction<br/>and corrected labels"]
For segmentation of a new image, only SNet is required; OSM labels are not an inference input. For correction of an existing map, TNet additionally takes the supplied mask and the SNet prediction to estimate the correction. These outputs serve different purposes: segmentation predicts buildings from imagery, whereas geometric correction preserves the supplied shapes and topology but cannot itself invent missing buildings.
Key Designs¶
1. Paired geometric augmentation: disrupt a predictable offset direction
The image and its misaligned mask undergo the same horizontal flips, vertical flips, and rotations of 90°, 180°, or 270°. Applying a common transformation does not remove their misalignment; it changes the direction of that misalignment in the augmented coordinate system. For example, a 50-pixel horizontal displacement becomes a displacement of the opposite sign after jointly flipping the pair horizontally. A segmentation network can therefore no longer rely as easily on placing every prediction at the same offset from the visible building.
The purpose is specifically to counter the statistical structure of systematic misalignment, rather than merely to increase visual diversity. Random, unbiased transformations do not offer the same simple shortcut, so augmentation need not have the same effect there. Indeed, augmentation alone improves the systematically shifted case substantially but reduces random-misalignment segmentation IoU from 0.71 to 0.66. This is an important qualification: augmentation is not a complete registration method, and its benefit depends on the other training constraints.
2. Joint segmentation–registration learning: correct supervision in mask space
SNet maps an image to a probabilistic building mask using a conventional semantic segmentation architecture. TNet receives that predicted mask together with the original misaligned mask, rather than solving registration directly between RGB imagery and polygons. It predicts a transformation for the entire label patch, and a spatial transformation operation resamples the labels into a corrected target. Cross-entropy between this target and SNet's prediction trains the coupled system. The experimental SNet uses a U-Net-style decoder with a ConvNeXt-Tiny encoder, while TNet uses ViT-Small. The spatial transformer operation should not itself be confused with an attention-based Transformer; ViT is the chosen transformation-prediction network.
Although the paper calls the transformation affine, its implemented parameters are one rotation angle and two translations, not a general six-parameter affine map. A scaled tanh activation bounds the predicted angle and displacements, preventing arbitrarily large transformations. Border regions without common image support after transformation are excluded from loss evaluation rather than treated as valid filled-in supervision. Unlike an offline label-repair pipeline, TNet keeps adapting as SNet's predictions change. However, this architecture alone leaves a degenerate solution: the segmentation network can reproduce the offset labels while TNet predicts almost no correction.
3. Absolute transformation consistency: anchor registration independently of segmentation quality
AnS applies an additional, randomly sampled transformation to an existing misaligned mask, then feeds the transformed mask and the original mask to the same TNet. The original mask need not be correctly located in the satellite image: the auxiliary task only asks TNet to undo the newly introduced transformation, whose inverse is known. This converts a lack of clean image-aligned annotations into a geometric learning task with automatically available targets. It also gives TNet a useful signal before SNet can produce reliable building predictions.
The consistency objective combines mean squared error between transformation-matrix elements and an IoU loss between the restored and original masks. The matrix term supplies a geometric error signal even when the masks barely overlap, while the IoU term rewards agreement of the actual building regions. The matrix-error term is weighted by 100 to put the two components on a comparable numerical scale. Because the desired inverse transformation is known, the objective is stronger than simply demanding agreement between predictions that might all share a common incorrect offset. A fresh random transformation is sampled each epoch rather than fixing the auxiliary pairing for an entire training round.
A Worked Example¶
Consider an image whose building labels are shifted 50 pixels to the right, matching the paper's systematic-noise construction. A joint horizontal flip turns this into a 50-pixel leftward displacement relative to the flipped buildings. SNet predicts buildings in the current image, and TNet compares that prediction with the label mask to estimate a correction. Meanwhile, the auxiliary branch applies another known random transformation to a label mask and asks TNet to undo it, discouraging a persistent identity-only solution. After training, a new image requires only SNet for segmentation; TNet is additionally used when existing polygons need correction. This example illustrates the training mechanism, not a guarantee of exact recovery for an individual test case.
Loss & Training¶
The full objective adds segmentation cross-entropy and the consistency loss, with both networks trained jointly while SNet's DINOv3-pretrained encoder remains frozen. Training uses AdamW at a learning rate of \(10^{-5}\) for 300 epochs with a batch size of 48. The output activation scale is 0.35; for the standard 320-pixel patches, the paper reports translation bounds of ±112 pixels and rotation bounds of approximately ±20.06°. The best checkpoint is selected using validation \(\mathrm{IoU}_{learn}\), not a metric requiring clean reference masks. “Unsupervised” here means that clean aligned labels are unnecessary for learning correction; it does not mean that building annotations or pretrained weights are absent.
Equations (1)–(6) are damaged in the locally extracted text, so their coordinate matrices and detailed loss formulas are not reconstructed here. The mechanisms above follow the readable explanations in Section 3; the separate supplement is not included in the cache, leaving alternative-encoder results and supplementary implementation details outside this note's evidence base.
Key Experimental Results¶
Main Results¶
The synthetic datasets use SpaceNet 2: \(D_{uni}\) independently samples horizontal and vertical shifts within ±50 pixels and rotations within ±4.5°, while \(D_{bias}\) applies a fixed horizontal shift of 50 pixels. Each synthetic city's patches are randomly divided into 80% training, 10% validation, and 10% test data. For real misalignment, ReBO uses its released test set and manually corrected roof polygons as reference labels; these roof scores should not be treated as interchangeable with ground-footprint accuracy.
All three reported IoUs are higher-is-better, but they measure different properties: - \(\mathrm{IoU}_{seg}\) compares SNet's predicted mask with the clean reference and measures actual segmentation quality. - \(\mathrm{IoU}_{align}\) tests TNet with a clean mask and a version subjected to a known transformation, measuring registration under ideal segmentation rather than directly measuring end-to-end map repair. - \(\mathrm{IoU}_{learn}\) compares SNet's prediction with the corrected training mask; it measures internal agreement, which does not by itself establish correct positioning.
The selected comparisons below come from Table 3 and use each dataset's corresponding test set. Map Alignment (MA) is a useful comparator because, like AnS, it does not require clean labels.
| Dataset / misalignment | AnS segmentation IoU ↑ | MA segmentation IoU ↑ | AnS alignment IoU ↑ | MA alignment IoU ↑ |
|---|---|---|---|---|
| Las Vegas / random | 0.79 | 0.55 | 0.84 | 0.70 |
| Las Vegas / systematic | 0.78 | 0.49 | 0.88 | 0.47 |
| Paris / random | 0.56 | 0.49 | 0.67 | 0.51 |
| ReBO / real | 0.62 | 0.55 | 0.74 | 0.57 |
These results do not establish superiority over every method in Table 3. Supervised ACN reaches segmentation IoU 0.94 on random-misalignment Las Vegas versus AnS's 0.79, but uses clean training labels and requires misaligned labels as an additional inference input. Spatial Correction (SC), although grouped with unsupervised baselines in parts of the discussion, uses clean validation labels according to Section 5 and is not a strictly clean-label-free comparator.
Ablation Study¶
The following selection is from Table 2 on the Las Vegas test sets, with the same frozen DINOv3 ConvNeXt-Tiny encoder. Every configuration includes cross-entropy, and every IoU is higher-is-better.
| Consistency loss | Geometric augmentation | Random: segmentation IoU | Systematic: segmentation IoU | Systematic: alignment IoU | Systematic: learning IoU |
|---|---|---|---|---|---|
| No | No | 0.71 | 0.39 | 0.41 | 0.79 |
| Yes | No | 0.74 | 0.41 | 0.43 | 0.78 |
| No | Yes | 0.66 | 0.72 | 0.76 | 0.82 |
| Yes | Yes | 0.79 | 0.78 | 0.88 | 0.82 |
Key Findings¶
- Under systematic misalignment, the full system gains 0.39 segmentation IoU and 0.47 alignment IoU over naive joint training. Augmentation supplies the largest isolated improvement, with consistency providing additional gains.
- Internal agreement can conceal failure: without either countermeasure, learning IoU is already 0.79 while segmentation IoU is only 0.39. The two networks can agree at the wrong location.
- Section 6 reports that pretraining and freezing TNet before learning segmentation achieves only 0.48, 0.44, and 0.38 segmentation IoU on random, systematic, and ReBO data, respectively. This supports ongoing joint adaptation rather than a fixed registration module.
- San Juan has real OSM misalignment but no clean reference masks. Its reported learning IoU of 0.70 in Section 6 is not a ground-truth segmentation score comparable to the main table.
Highlights & Insights¶
- Separate the source of supervision from the final task. An unknown building location in an image does not prevent generating a known mask transformation, giving registration supervision independent of immature segmentation predictions.
- Augmentation targets the error distribution. A joint flip leaves the displacement magnitude unchanged but alters its direction; the ablation distinguishes this effect from a generic generalization benefit.
- Measure agreement and correctness separately. The three-IoU protocol exposes a joint model that fits its training objective without recovering the correct geometry, a useful diagnostic pattern for coupled learning systems.
Limitations & Future Work¶
- Authors' limitation: the generic decoder is not optimized for building boundaries and has no post-processing. Predicted masks can remain blurry, whereas transformed existing labels better preserve their original shapes.
- Authors' limitation: missing annotations and other non-geometric label errors are untreated. A single patch-level transformation also cannot independently fix different offsets for buildings within the same patch.
- Reader assessment: the synthetic-city experiments train and test within each city, not by training on one city and deploying on another. Random patch splits also differ from a strict spatial-extrapolation evaluation.
- Reader assessment: checkpoint selection still depends on learning IoU, which can be inflated by shortcut solutions. The method reduces this failure empirically but does not prove that all incorrect solutions can be identified without clean reference data.
Related Work & Insights¶
- Versus AutoCorrect: AutoCorrect performs image–mask registration with transformation consistency. AnS feeds two masks to TNet and anchors its auxiliary task to a known inverse transformation.
- Versus Map Alignment: MA corrects labels across multiple training rounds and fixes transformations within a round. AnS resamples each epoch and continuously learns segmentation together with correction.
- Versus ACN / MapRepair: the stronger supervised results show the continuing value of clean annotations. AnS primarily lowers the alignment requirements of training data rather than replacing all specialized models under equal supervision.
Rating¶
- Novelty: 4/5. Combining absolute mask-space transformation supervision with systematic-offset countermeasures is well motivated, although the networks and transformation tools are established.
- Experimental Thoroughness: 4/5. Synthetic and real misalignment, ablations, and sequential-training comparisons are covered; spatial extrapolation, non-rigid errors, and missing labels remain open.
- Writing Quality: 4/5. The shortcut explanation and diagnostic metrics are persuasive, but metric definitions and supervision differences require careful reading.
- Value: 4/5. The method is practically relevant for learning from existing building maps, with boundary fidelity and complex local misalignment still limiting deployment.