Satellite Image Segmentation: Datasets, Models and Annotation for Remote Sensing AI

Satellite image segmentation assigns a class label to every pixel of a satellite or aerial image, such as building, road, water, crop or flooded area, so imagery can be turned into maps and measurements. This guide covers the main segmentation types, 11 verified public benchmark datasets, the models teams use from U-Net to geospatial foundation models, the tooling, and the annotation rules that decide final quality.

Semantic, instance and panoptic segmentation, and change detection

"Segmentation" covers several distinct tasks in remote sensing. They need different labels, different models and different metrics, so pick the task before you pick a dataset.

Most land cover and flood work is semantic segmentation. Building and infrastructure inventories usually need instance-level outputs, because the business question is "how many" and "how big", not only "where". If counts and approximate locations are enough, bounding boxes are cheaper to produce than masks. Change detection is often implemented as segmentation on a stacked pair of images, so the same annotation tooling applies, but the labels describe a difference rather than a state. For the wider picture of satellite labeling tasks, see our introduction to satellite image annotation techniques and the general guide on how image segmentation works.

How a satellite image segmentation pipeline works

1. Acquisition. Choose sensor and ground sampling distance (GSD) for the smallest object you must detect: 10 m Sentinel-2 for land cover and floods, 0.3 to 0.5 m commercial imagery for buildings and roads, 5 to 10 cm aerial orthophotos for cars and roof details.
2. Pre-processing. Radiometric and atmospheric correction, co-registration of multi-date images, reprojection to one coordinate reference system, and cloud masking. Keep the band order and scaling identical between training and inference.
3. Tiling. Cut large scenes into fixed-size chips (often 256 to 1024 pixels) with overlap, and keep the geotransform of each chip so predictions can be stitched back to map coordinates.
4. Training and inference. An encoder-decoder network predicts a class per pixel. Multispectral inputs (near infrared, red edge, SWIR) or SAR bands are fed as extra channels.
5. Post-processing. Stitch tiles, smooth seams, remove slivers below a minimum mapping unit, and vectorize masks into polygons when the output goes to a GIS.

Public benchmark datasets for satellite image segmentation

The table lists widely used public datasets, with figures taken from each dataset's paper or official page. Check the licence column before you train a commercial model: several of the best-known sets are restricted to academic or non-commercial use.

A few notes from the source pages. DeepGlobe's "unknown" class covers clouds and is excluded from scoring. ISPRS withholds ground truth for part of the tiles and scores submissions on them. OpenEarthMap labels inherit the licence of their source imagery, and its xBD-derived images must be downloaded separately. Dynamic World's training labels also include a cloud class, and the dataset was labeled by both expert and non-expert annotators. The SpaceNet series as a whole is hosted on the AWS Registry of Open Data, with licences that vary by dataset.

The earlier version of this article pointed readers to Radiant MLHub. That platform no longer operates: Radiant Earth announced in 2023 that it would be replaced by Source Cooperative, and in October 2023 confirmed that all data previously on Radiant MLHub had been migrated there. Look for former MLHub datasets on Source Cooperative.

The practical takeaway for data leads: public sets are excellent for benchmarking and pretraining, but the licences, class definitions and regions rarely match a production use case. Most commercial projects end up fine-tuning on their own labeled imagery. Related reading: land cover classification annotation challenges and crop classification datasets.

Models for satellite image segmentation

U-Net and DeepLabv3+: the CNN baselines

U-Net (Ronneberger et al., 2015) pairs a contracting path that captures context with a symmetric expanding path for precise localization. It was designed for biomedical images, but its ability to train on small labeled sets made it the default remote sensing baseline. DeepLabv3+ (Chen et al., 2018) combines Atrous Spatial Pyramid Pooling, which probes features at several rates for multi-scale context, with a decoder that refines object boundaries. Both accept any number of input channels once the first convolution is adapted, which is how teams feed near infrared, SWIR or SAR bands.

SegFormer and transformer models

SegFormer (Xie et al., 2021) uses a hierarchical transformer encoder that outputs multi-scale features, needs no positional encoding, and uses a lightweight all-MLP decoder. The absence of positional encoding matters for satellite work, where inference tiles are often a different size than training tiles.

Geospatial foundation models (2024 to 2026)

Prithvi-EO-2.0, from IBM and NASA (arXiv, December 2024), comes in 300M and 600M parameter versions based on ViT-L and ViT-H. It was pretrained on about 4.2 million global time series samples from NASA's Harmonized Landsat and Sentinel-2 (HLS) archive at 30 m, and uses temporal and location embeddings. Its paper evaluates segmentation tasks including Sen1Floods11 flood mapping, burn scars, landslides and crop types. The weights are on Hugging Face under Apache 2.0.

Foundation models are not an automatic upgrade. The PANGAEA benchmark evaluated popular geospatial foundation models (including CROMA, DOFA, Prithvi, SatlasNet, Scale-MAE and SpectralGPT) on 11 datasets, and found they "do not consistently outperform supervised models". With full labels, a supervised U-Net placed in the top two on the most datasets and was best on HLS Burn Scars, Sen1Floods11 and AI4SmallFarms. Foundation models gained ground as labels became scarce (10% of the training labels). In practice: if you have a well-labeled training set, benchmark a U-Net before adopting a large backbone; if labels are scarce, a foundation model is worth testing.

Segment Anything for remote sensing

Meta's Segment Anything (SAM) was trained on SA-1B, about 11 million images and over 1 billion masks, and is prompted with points, boxes or masks. SAM 2 extends this to video with a streaming memory and is released under Apache 2.0. SAM produces class-agnostic masks, so on its own it does not output land cover classes. The open-source segment-geospatial (samgeo) package adapts SAM, SAM 2 and SAM 3 to GeoTIFFs, supports text prompts through Grounding DINO, and exports results to GeoPackage, Shapefile or GeoJSON. Its most common use is annotation assistance: generating candidate polygons that annotators accept, correct or reject.

Metrics: mIoU, per-class IoU and F1

• IoU per class is the overlap between predicted and reference pixels divided by their union. Report it for every class, because small classes (roads, cars, buildings in rural scenes) are hidden by a good average.
• mIoU averages per-class IoU and is the standard headline number for semantic segmentation benchmarks, including PANGAEA.
• Object-level F1 is used for footprints: DeepGlobe's building track and SpaceNet count a predicted building as correct when it overlaps a reference footprint at IoU 0.5 or more.
• Exclude no-data and cloud pixels from scoring the way DeepGlobe excludes its unknown class, and say so in your evaluation report.

SAR image segmentation with AI

Synthetic aperture radar sees through clouds and at night, which is why it dominates flood mapping. It is also harder to label and to model: speckle noise, layover and the side-looking geometry make buildings look nothing like their optical footprint. The SpaceNet 6 baseline shows the gap: building extraction from SAR alone reached an F1 of about 0.12 to 0.14, while pretraining on optical imagery before SAR fine-tuning raised it to 0.21. Common practice is to label on a co-registered optical image or a pre-event reference when available, then transfer the mask to the SAR grid, and to keep VV and VH as separate input channels. Our article on SAR annotation challenges and best practices covers labeling details, and the flood zone annotation guide covers disaster mapping.

Practical tooling

• TorchGeo: PyTorch library with loaders for common benchmark datasets, geospatial samplers, multispectral transforms and pretrained weights, including models that use all Sentinel-2 bands.
• TerraTorch: IBM's Apache 2.0 toolkit, built on PyTorch Lightning and TorchGeo, for fine-tuning geospatial foundation models such as Prithvi-EO-2.0, Clay v1, DOFA and Satlas from a configuration file.
• rasterio and GDAL: raster input and output, reprojection, windowed reads for tiling, and mask vectorization. Rasterio is a BSD-licensed Python layer over GDAL, which is MIT licensed.
• QGIS: GPL-licensed desktop GIS for inspecting imagery, reviewing predictions against basemaps and editing polygons. samgeo is also available as a QGIS plugin.

How to annotate satellite imagery for segmentation

Model choice moves results by a few points; inconsistent labels can cost far more. These are the rules a geospatial annotation guideline should fix before the first image is labeled. The thresholds below are examples to adapt, not universal standards.

Tiling and GSD

• Label at the native GSD, never on a resampled preview. At 0.5 m a typical 4.5 m car spans about 9 pixels; at 10 m it is sub-pixel and should not be a class at all.
• Annotate at full scene or large-tile level, then cut training chips. Reference datasets use fixed chips, such as 512 × 512 px in Sen1Floods11 and 510 × 510 px in Dynamic World.
• Define a minimum mapping unit, for example "ignore isolated patches under 4 × 4 pixels", so annotators do not spend time on noise.

Class hierarchy and definitions

• Write a two-level hierarchy (vegetation → tree, low vegetation, cropland) so a class can be merged later without relabeling.
• Give each class a measurable definition: "tree = woody canopy taller than low vegetation, visible as a crown", "impervious surface = paved ground that is not a building roof".
• Add an explicit catch-all class, like ISPRS clutter/background, rather than forcing ambiguous pixels into the nearest class.

Clouds, shadows and no-data

• Use separate cloud, cloud shadow and no-data classes. Do not guess the land cover under a thick cloud.
• Decide whether building shadows belong to the ground class they fall on or to a shadow class, and keep the rule identical across the project.

Boundary rules

• Building footprints follow the roof outline in nadir images; for off-nadir imagery, state whether to trace the roof or the base.
• Roads: label the paved surface, not the full right of way; bridges belong to road, the water underneath stays water.
• Mixed boundary pixels go to the class covering most of the pixel. Water bodies are traced at the waterline on the acquisition date, not from a map layer.

QA by inter-annotator IoU

• Have a sample of tiles (for example 5 to 10%) labeled independently by two annotators and compute per-class IoU between them. Dynamic World's validation tiles were each labeled independently by three experts.
• Set a threshold per class (for example IoU of 0.85 for buildings, lower for fuzzy classes such as rangeland), and send tiles that fall below it to a reviewer.
• Low agreement on one class usually signals an unclear definition, so fix the guideline before relabeling.
• Run automated checks: no unlabeled pixels, no invalid class values, no self-intersecting polygons, masks aligned with the image geotransform.

Pre-labeling with SAM-based tools or an earlier model speeds up polygon work, but every pre-label still needs human review, especially along boundaries. Multispectral false-color composites help annotators separate vegetation from bare soil; see multispectral vs RGB labeling strategies.

Failure modes to plan for

• Class imbalance. Roads and buildings cover a small share of pixels; use class-weighted or Dice-style losses and report per-class IoU.
• Domain shift between regions. LoveDA splits its scenes into urban and rural domains because, as its authors note, they can show completely different landscapes. Hold out whole regions, not random tiles, for validation.
• Seasonal and sensor shift. Vegetation classes change with phenology, and band definitions differ between satellites. Include several seasons and normalize per sensor.
• Spatial leakage. Overlapping chips from the same scene in train and test inflate scores; split before tiling.

Segmentation outputs feed land cover mapping, crop monitoring, flood response, building inventories and infrastructure planning. For the GIS side of these applications, see our guide to geospatial mapping.

Get satellite segmentation data labeled by specialists

DataVLab is a Paris-based data annotation company, founded in 2019, that builds pixel-accurate training data for geospatial AI. Our EU-based managed annotators work in custom annotation tooling with multi-step QA, under ISO/IEC 27001 certified and GDPR-compliant workflows. Explore our satellite image annotation services, semantic segmentation services, geospatial data annotation or aerial image annotation, or contact us to scope a pilot on your imagery.