Skip to content
SR-Forge

Scores

MetricEntry dataclass

Holds all variants of a single metric
  • raw: output of calculate_score (no sign, no weight)
  • best_min: True if smaller is better; False if larger is better
  • weight: optimization weight (can be 0)
  • finalize: optional rule for turning accumulated raw into one number, replacing the default "average the per-sample values"

finalize exists because some metrics are not averages of per-sample scores. A micro-averaged F1 is 2*ΣTP / (2*ΣTP + ΣFP + ΣFN) over every pixel in the epoch, which is not the mean of per-image F1 — on change-detection data the two differ by tens of percent, because images containing no change score 0 and drag a mean down. Such a metric reports its parts ({"tp": [B], "fp": [B], "fn": [B]}) and supplies the rule that combines them, so the aggregation travels with the value instead of living in something that has to find it again by name later.

A finalized entry is not a per-sample score, so it never appears in the per-sample views. What it contributes to total_raw and total_weighted depends on aggregate:

  • "micro" — its pooled value over whatever the entry currently holds: one training batch when the runner backpropagates, the whole epoch when a checkpoint compares totals. That is batch-level micro for training, and it is differentiable wherever the metric is.
  • "custom" — nothing. A count or a worst case is not a loss term.

The rule must be a function of per-key means. In a single process raw holds every per-sample value; in a distributed run TrainingStrategy.reduce_scores has already collapsed each key to one global mean. A rule written with .mean(), or as a ratio of sums (where the sample count cancels), gives the same answer in both cases. F1, precision, recall, IoU, MCC and a pooled PSNR all satisfy this.

A rule that does not — a bare .sum(), or an order statistic such as a median or a 95th percentile — is correct on one process and silently wrong on N, because the values it needs no longer exist by the time it runs. Such a metric does not belong here.

is_finalized: bool property

Whether this entry carries parts + a rule rather than scores.

counts_toward_total: bool property

Whether this entry is a loss term. Per-image scores are; a pooled (micro) value is the same quantity weighted differently, so it is too; a custom epoch rule (a total, a worst case) is not.

total_batch(weighted: bool) -> torch.Tensor

This entry's [B] contribution to a loss total.

A pooled entry has one value for the whole batch rather than one per image, so it is repeated across the batch: the runner's .mean() then recovers exactly that value, and its gradient, with no rescaling.

reduction_for(key: str = None) -> str

The declared reduction for key, defaulting to "cat".

reduced() -> TensorOrDict

raw with each key collapsed as the metric declared.

This is what :meth:finalize receives: values already totalled, minimised or averaged according to what they are, so the formula does not have to compensate for an aggregation it did not choose. Without a declaration the raw values pass through unchanged, which is the behaviour of every metric that predates this. With bands, each band is collapsed on its own: {band: reduced value}.

band_values(weighted: bool = False) -> Dict[str, torch.Tensor]

Each band's own value, by the metric's own rule.

For a finalized entry with bands: the metric's :meth:finalize run on each band's totals, so F1.red is the micro F1 of the red band alone. weighted signs and weights micro values the way :meth:final_weighted does.

final_value() -> torch.Tensor

The single number this entry represents.

With bands, the bands combine the way the metric's images do:

  • macro and micro — the average of the band values, each band counting equally (bands often differ in resolution, and pooling would let the finest one dominate);
  • custom — the band totals combined by the metric's own declared rules, then its finalize (a count sums, a worst case takes the minimum). Where a rule cannot run across bands, the average of the band values.

final_weighted() -> torch.Tensor

:meth:final_value as it enters the weighted total.

Sign and weight apply to a pooled (micro) value exactly as to a per-image score, so the weighted numbers of a MetricScores add up to its total_weighted(). A custom rule's value — a count, a worst case — is not a loss term and has no weighted form, so it is returned as is.

MetricScores dataclass

__setitem__(name: str, entry: MetricEntry) -> None

In-place add or replace a metric. Used by hooks that inject auxiliary loss terms from on_post_forward — drop a :class:MetricEntry here and the runner's total_weighted() picks it up before backward, and LossLogger emits it as a per-name series alongside the criterion's metrics.

add_scores(other: MetricScores) -> None

In-place accumulation: concatenates batch dimension when names match.