SR-Forge

Classification Metrics¶

Metrics for models that answer yes or no about every pixel — semantic segmentation, change detection, anomaly and defect detection, cloud or shadow masking, water and road extraction, any foreground/background split.

They are called classification metrics because that is what they are: F1, precision, recall and MCC don't know they're looking at an image, and would behave identically on a flat array of yes/no decisions. SR-Forge just applies them per pixel.

Nothing here is tied to a task. It's tied to the shape of the question: a prediction map, a target map, and a decision per element.

metrics:
  - _target: srforge.metrics.classification.F1Score
    params:
      aggregate: micro
  - _target: srforge.metrics.classification.MCC
  - _target: srforge.metrics.classification.FalsePositives


Counts	`TruePositives` · `FalsePositives` · `FalseNegatives` · `TrueNegatives`
Ratios	`Precision` · `Recall` · `F1Score` · `IoU` · `MCC` · `TverskyIndex`
Losses	`BinaryCrossEntropy` · `DiceLoss` · `JaccardLoss` · `TverskyLoss` · `CrossEntropy`, and `soft: true` on any ratio — training with them

x is a score or probability map, thresholded at threshold (default 0.5); y is the target, positive where >= 0.5. If your model emits multi-channel logits, apply the softmax or argmax in a transform — that belongs to the model's output convention, not to the metric.

Micro or macro — read this before choosing¶

Every ratio takes aggregate — the option every SR-Forge metric takes (see Using Metrics → Micro or macro); the ratios default to micro:

"micro" (default) pools every pixel in the epoch and computes the ratio once. This is what the literature usually reports.
"macro" scores each image and averages the scores.

They are not two spellings of the same number. Take two images:

	tp	fp	fn	F1
A — plenty of positives, mostly found	90	10	10	0.900
B — no positives at all, a few false alarms	0	5	0	0.000

micro:  pool everything, divide once   ->  0.878
macro:  average the two scores         ->  0.450

Image B is the whole story. It contains nothing to find, so the model cannot score a single true positive — its F1 is 0 no matter how well it behaves. Under macro that zero is averaged in at full weight and halves your result. Under micro its five false alarms are five pixels among hundreds.

This is the normal case whenever the positive class is sparse, which covers change detection, defect and anomaly detection, and segmentation of small or thin structures. Neither averaging is wrong; they answer different questions:

micro — across all the pixels I looked at, how did I do?
macro — on a typical image, how did I do?

Pick deliberately. The name appears in your logs either way, so a reader can tell which you used.

Not every ratio diverges

Precision, Recall, F1Score and IoU all suffer the empty-image problem. A specificity-style metric (TN / (TN + FP)) barely moves, because true negatives dominate every image and are never zero.

Counts¶

TruePositives, FalsePositives, FalseNegatives and TrueNegatives report totals over the epoch, not averages — the actual number of pixels in each cell.

They are the only per-pixel quantity here: every pixel falls into exactly one of the four, independently of its neighbours. That makes them Level 2, so masking is applied for you, and it makes them additive, so the epoch total is exact however the images were sized or batched.

Worth logging alongside the ratios. When F1 moves you usually want to know whether it was recall or false alarms that changed, and the counts say so directly.

Which ratio¶

	formula	notes
`Precision`	TP / (TP + FP)	of what you flagged, how much was right
`Recall`	TP / (TP + FN)	of the positives, how much you found
`F1Score`	2TP / (2TP + FP + FN)	harmonic mean of the two
`IoU`	TP / (TP + FP + FN)	Jaccard index
`MCC`	uses all four cells	correlation, in [-1, 1]
`TverskyIndex`	TP / (TP + α·FP + β·FN)	F1 (α = β = 0.5) or IoU (α = β = 1) with the two errors priced apart

Three relations worth knowing before you report several of these:

F1Score is Dice. 2|A∩B|/(|A|+|B|) is the same expression. Reporting both is reporting one number twice.

IoU is a function of F1 — exactly F1 / (2 - F1). The two rank models identically. Report IoU because it is conventional, not as corroboration.

MCC is the only one that sees true negatives. F1, precision, recall and IoU are all blind to TN, so piling up correctly-negative pixels does not move them. That makes MCC the honest choice on heavily imbalanced masks, and its zero means no better than chance rather than no overlap.

Options¶

Option	Default	Meaning
`threshold`	`0.5`	Score above which a pixel counts as predicted-positive
`aggregate`	`"micro"`	See above
`smooth`	`0.0` (`1.0` when soft)	Added to numerator and denominator
`soft`	`false`	Count with probabilities, to train on it

`smooth`¶

An image with no positives, where the model predicted none, gives 0/0:

F1 = 2·0 / (2·0 + 0 + 0)     undefined

	result	says
tiny epsilon	`0.0`	"total failure" — punishes a correct prediction
`smooth=1`	`1.0`	"perfect" — nothing to find, nothing found

smooth=1 is the usual choice in segmentation code, and it's safe at any value here because the counts are summed before the ratio is taken.

Masking¶

All of these honour y_mask, which matters for invalid regions, cloud masks, no-data borders and any region that should not be scored:

- _target: srforge.metrics.classification.F1Score
  io:
    inputs: {x: prediction, y: target, y_mask: valid_mask}

Masked pixels are excluded from every count, so they affect neither the numerator nor the denominator.

Two mistakes that produce plausible numbers¶

Both are refused as of this version, but worth understanding:

A multi-channel prediction against a single-channel target. Torch broadcasts [B,2,H,W] against [B,1,H,W] and roughly doubles every count. Reduce logits to one score per pixel first.

A target without a channel axis. [B,1,H,W] against [B,H,W] looks harmless and is worse — broadcasting aligns from the right, so the target's batch dimension lands on the prediction's channel dimension. Pass y.unsqueeze(1).

Two more cannot be detected, so they are yours to avoid:

Binding y to an image instead of a mask. The metric does y >= 0.5, so "bright pixel" silently becomes "positive".

Passing logits where probabilities are expected. threshold=0.5 on a logit map is a cut at sigmoid(0.5) = 0.62 — you report a 62% confidence operating point believing it is 50%.

Training with them¶

The metrics above can't be a training loss. Each one first decides yes/no per pixel (x >= threshold), and no gradient passes through that decision. Put F1Score in a criterion and the model learns nothing from it.

These can:

Loss	What it is	Better	Use it for
`BinaryCrossEntropy`	per-pixel `−[y·log p + (1−y)·log(1−p)]`	lower	the steady per-pixel signal
`DiceLoss`	`F1Score(soft=True)`	higher	overlap, robust to a rare positive class
`JaccardLoss`	`IoU(soft=True)`	higher	the same idea, scored as IoU
`TverskyLoss`	`TverskyIndex(soft=True)`	higher	Dice with missed positives and false alarms priced apart
any ratio with `soft: true`	the ratio on soft counts	as the ratio	soft precision, soft MCC, your own
`CrossEntropy`	per-pixel `−Σ_c y_c·log p_c`	lower	several classes, one channel each

The soft ratios keep their direction — soft F1 is higher-is-better, like F1 — and the training total negates them, as it negates every higher-is-better metric (how the total is formed). So a negative weighted Dice in your logs is the sign that Dice is being maximised. There is no 1 − Dice hiding in the number: had there been, a value like 0.2 could not tell you which way it was being pushed.

The usual recipe for change detection or any sparse mask is BCE + Dice: cross-entropy gives every pixel a gradient from the first step, Dice keeps a rare positive class from being drowned by the background.

criterion:
  _target: srforge.metrics.LossCombiner
  params:
    losses:
      - _target: srforge.metrics.classification.BinaryCrossEntropy
      - _target: srforge.metrics.classification.DiceLoss

metrics:                       # what you report and pick the best model by
  - _target: srforge.metrics.classification.F1Score

x is a probability (a sigmoid output), as everywhere on this page.

How can F1 become a loss?

Replace the yes/no with the probability itself when counting:

	hard (the metric)	soft (the loss)
TP	pixels with `p ≥ 0.5` and `y = 1`	`Σ p·y`
FP	pixels with `p ≥ 0.5` and `y = 0`	`Σ p·(1−y)`
FN	pixels with `p < 0.5` and `y = 1`	`Σ (1−p)·y`

With 0/1 predictions the two columns are equal; in between, the soft one changes smoothly, so there is a gradient. The formula is the metric's own ratio(), unchanged — DiceLoss is F1Score(soft=True), a subclass with nothing of its own. That is why a ratio you write yourself can be trained on with no extra code: Specificity(soft=True).

Keep the hard metric for reporting. The soft F1 of a probability map is not the F1 of the decisions the model will make.

Options: smooth, aggregate, names, Tversky's alpha / beta

smooth defaults to 1 whenever soft is on — the named losses included — and to 0 for the hard metrics. Without it a batch with no positives gives Dice 0 / FP: a constant, so nothing pushes the false alarms down. An explicit value always wins.

aggregate: micro (default) pools the counts over the batch — "batch Dice", stable when some images have nothing to find. macro computes one value per image and averages, dominated by those empty images just as macro F1 is.

Names. A soft ratio is logged as Soft<Class> — F1Score(soft=True) as SoftF1Score — so a soft value never appears under the hard metric's name. The named losses keep their own (DiceLoss), and a name: you set is used as given.

threshold has no meaning when soft — there is no decision to make — so passing both is refused.

TverskyLoss(alpha, beta) prices false alarms (alpha) and missed positives (beta) apart. 0.5 / 0.5 is Dice, 1 / 1 is Jaccard, and 0.3 / 0.7 is a common choice when missing a change is worse than a false alarm. TverskyIndex is the same formula as a metric.

Class weights

Both cross-entropies take class_weights, a constant of your data — work it out once from the class balance and write it into the config.

- _target: srforge.metrics.classification.BinaryCrossEntropy
  params:
    class_weights: [1.0, 12.0]        # (negative, positive)
- _target: srforge.metrics.classification.CrossEntropy
  params:
    class_weights: [0.3, 1.0, 4.0]    # one per channel

Or let SR-Forge count the training set once, when the config is built, and pass the result on:

class_weights:
  _target: srforge.dataset.stats.get_class_weights
  params:
    dataset: ${ref:dataset.training}
    field: label             # the entry key your loss's y reads
    num_classes: 2
    scheme: inverse_freq     # rare classes weigh more; or proportional

criterion:
  _target: srforge.metrics.classification.BinaryCrossEntropy
  params:
    class_weights: ${ref:class_weights}

The labels must already be class indices (a 0/1 mask counts as two classes); turn a raw label into classes with a transform on the dataset, which runs before counting. ignore_index and mask_field leave elements out of the counts. See srforge.dataset.stats.

Each pixel's term is multiplied by its class's weight, then pixels are averaged as usual. torch's F.cross_entropy(weight=...) divides by the total weight of the targets instead, so its value differs by that factor; the gradients point the same way.

CrossEntropy is for several classes, not a yes/no map

CrossEntropy expects one channel per class (a softmax output, C ≥ 2), and its target is either the same shape or a map of class indices ([B, H, W] or [B, 1, H, W], integer). On a single channel −y·log p has no term for the negative class — predicting "yes" everywhere would score a perfect 0 — so it refuses one and points to BinaryCrossEntropy.

Writing your own¶

One method. Counts, thresholding, masking and both averagings are inherited:

from srforge.metrics.classification import ConfusionRatio, Counts
from srforge.registry import register_class

@register_class
class Specificity(ConfusionRatio):
    def ratio(self, c: Counts):
        return c.tn / (c.tn + c.fp + 1e-12)

ratio receives a Counts with .tp, .fp, .fn and .tn — per image under "macro", already totalled over the epoch under "micro". The same method serves both, so there is no second code path to keep in step.

Attributes rather than dictionary keys on purpose: c.tpp raises immediately and names the attribute, where a mistyped string key would fail somewhere downstream or select the wrong counts. Counts also carries predicted_positive, actual_positive and total.

A new count is just as short:

@register_class
class PredictedPositives(ConfusionCount):
    @property
    def best_min(self) -> bool:
        return False

    def indicator(self, x, y):
        return self.predicted(x)

Why these are native¶

SR-Forge normally adapts external metrics rather than reimplementing them, and torchmetrics has all of these. These are written out because they need two things no general metric library offers: they honour y_mask, and they share a pipeline with every other SR-Forge metric, so a masked F1 can sit beside the training loss it is judging, with the same IO binding and the same logging.

Classification Metrics¶

Micro or macro — read this before choosing¶

Counts¶

Which ratio¶

Options¶

smooth¶

Masking¶

Two mistakes that produce plausible numbers¶

Training with them¶

Writing your own¶

Why these are native¶

`smooth`¶