SR-Forge

Writing a Metric¶

This page shows how to write your own metric — and so your own loss, since a loss is simply a metric you train on.

First, which methods to write — a short decision flow.
Then one metric written every possible way, so you can see what each method adds and what you lose without it.
Last, the details: inputs, masks, epoch rules, configs.

It assumes you have read Using Metrics up to §8 — in particular what weight, masks and micro/macro are.

1. The three methods¶

A metric answers up to three questions. Each method answers one of them:

Method	Answers	Returns
`pointwise`	How wrong is each pixel?	a map, same layout as the image: `[B, C, H, W]`
`calculate_score`	How wrong is each image?	one number per image: `[B]`
`finalize`	How wrong is the whole batch or epoch?	one number

  prediction, target
        │
        │  pointwise        how wrong is each pixel?
        ▼
  pixel map   [B, C, H, W]
        │
        │  calculate_score  how wrong is each image?
        ▼
  image scores   [B]
        │
        │  finalize         how wrong is everything seen so far?
        ▼
  one number

You never have to write all three. SR-Forge fills in what you leave out:

No calculate_score? SR-Forge writes it for you: it calls your pointwise, applies the mask, and averages each image's map.
No finalize? SR-Forge averages the image scores.

Why split it into three at all? Because different things happen at each stage, and the framework can only help at a stage it can see. The mask is applied to pixels. Image scores are added across metrics and can be inspected one by one. Pooling across images happens at the end. When you write only the stage your metric really needs, SR-Forge handles the others correctly — and the less you write, the more it can do for you.

2. Which methods should I write?¶

Answer three questions, in order.

1. Can you say how wrong each pixel is, on its own?

For example: the difference between the predicted and the true brightness of that pixel.

No — the error only makes sense for a region or for the whole image. SSIM compares small windows; NIQE judges the whole picture. → write calculate_score.
Yes → go to question 2.

2. Is the image's score just the average of those pixel errors?

Yes — like L1, the average absolute difference. → write only pointwise. You are done. Multiplying by a fixed number still counts as yes: put the number inside pointwise.
No — you do something more to the average, like taking its logarithm (PSNR). → write calculate_score, and build the pixel errors inside it.

3. Should the epoch's number be the average of the image scores?

Yes → nothing to write. This is the default.
You want it pooled over all pixels (micro) → nothing to write if you wrote only pointwise — users just set aggregate: micro. After calculate_score you have to offer it yourself (variant F below).
You want a total or a worst case — say, all pixels flagged during the epoch → also write finalize.

If your metric answers yes/no for every pixel — segmentation, change detection, cloud masks — skip all this and subclass one of the classification metrics: you write one formula and the rest is done.

Your answers	You write	Built-in examples
per-pixel error, score = its average	`pointwise`	L1, MSE, Charbonnier, CrossEntropy, LPIPS, SAM, UncertaintyLoss
per-pixel error, something done after averaging	`calculate_score`	PSNR
no per-pixel error	`calculate_score`	SSIM, MGE, TBE, TotalVariation
epoch number is not an average	`pointwise` or `calculate_score` (as above), plus `finalize`	TruePositives, FalsePositives, … (`pointwise` + `finalize`)

Why not always write pointwise when I have per-pixel errors?

Because SR-Forge assumes the average of your pointwise map is your metric — and so do wrappers such as UncertaintyLoss. PSNR has a per-pixel squared error, but its average is MSE, not PSNR. When PSNR used to expose that map as its pointwise, UncertaintyLoss(PSNR()) quietly trained on MSE — the logarithm simply vanished. Variant C below shows exactly this.

The exact rule, as equations

For one image, let e(p) be the error at pixel p, N the number of valid pixels, and S the image's score.

If `S` is	Write
`(1/N) · Σ e(p)`	`pointwise`, returning `e(p)`
`c · (1/N) · Σ e(p)`, `c` a fixed number	`pointwise`, returning `c · e(p)`
`g((1/N) · Σ e(p))`, `g` anything but a fixed multiplication	`calculate_score`
not expressible through a per-pixel `e(p)`	`calculate_score`

For the epoch, over images i:

If the epoch value `E` is	Write
`(1/n) · Σᵢ Sᵢ`	nothing (default)
`g(Σᵢ Σₚ eᵢ(p) / Σᵢ Nᵢ)`	nothing for `pointwise` (`aggregate: micro`); for `calculate_score`, `aggregations` + `finalize`
`h(Σᵢ aᵢ)` from totals `aᵢ` you choose	`finalize` + `reductions`

3. One metric, written every way¶

The decision flow tells you what to write. This section shows why, by taking one familiar metric and writing it with every combination of the three methods.

The metric. We want to measure how far a reconstruction is from its target:

per pixel, the squared error (x − y)²;
per image, its average — the MSE — or, in decibels, the PSNR 10 · log10(1 / MSE);
over a whole set, either the average of per-image PSNRs (macro), or the PSNR of the pooled error (micro).

Those last two are different numbers. Take two images of the same size:

	MSE	PSNR
A (almost perfect)	0.0001	40 dB
B (poor)	0.01	20 dB

Average of the two PSNRs (macro): (40 + 20) / 2 = 30 dB.
PSNR of the pooled error (micro): the average MSE is 0.00505, so 10 · log10(1 / 0.00505) ≈ 23 dB.

The logarithm turns A's tiny error into a large number, and in a plain average that one image lifts the result by 7 dB. Images A and B come back below.

All variants below start with:

import torch
from srforge.metrics import Metric

and every one declares best_min — True if lower is better (MSE), False if higher is better (PSNR). It sets the sign in the weighted total.

A. Only `pointwise`¶

class MyMSE(Metric):
    @property
    def best_min(self) -> bool:
        return True

    def pointwise(self, x: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
        return (x - y) ** 2                      # one error per pixel

How it runs. Your pointwise makes the pixel map. SR-Forge's own calculate_score applies y_mask to it and averages each image → MSE per image. SR-Forge's own finalize averages the images.

What you get, without writing it:

masking, applied correctly (including one-channel masks on multi-channel data);
reduction (mean / sum) and aggregate: micro — with micro, the epoch value is the MSE over every valid pixel seen, not the mean of per-image MSEs;
dict-of-bands inputs, scored band by band;
wrappers: UncertaintyLoss(MyMSE()) works, because it can reuse your map.

What you can't do. The image score is always the average of your map. There is no way to take the logarithm afterwards, so this cannot be PSNR.

Use it when your per-image score is the average of a per-pixel error. That is most metrics — and it is the least code with the most help.

B. Only `calculate_score`¶

class MyPSNR(Metric):
    @property
    def best_min(self) -> bool:
        return False

    def calculate_score(self, x: torch.Tensor, y: torch.Tensor,
                        y_mask: torch.Tensor = None) -> torch.Tensor:
        mse = self.reduce((x - y) ** 2, mask=y_mask)       # per-image MSE
        return 10 * torch.log10(1.0 / mse.clamp_min(1e-12))

How it runs. You compute each image's score yourself. SR-Forge's finalize averages them — the macro PSNR.

What you get. Full control of the per-image number: here, the log of the mean. self.reduce still does the masked averaging for you, if you pass it y_mask.

reduction needs one extra line here. This variant calls self.reduce(...), so users could choose reduction: sum, but only once the class says so: uses_reduction = True.

What you give up:

masking is your job. Nothing checks that you used y_mask; forget to pass it to reduce and you report an unmasked score as if it were masked;
only macro. aggregate: micro is refused — SR-Forge cannot pool a number it only sees after your logarithm (variant F adds it back);
no wrappers. UncertaintyLoss(MyPSNR()) is refused: there is no pixel map to reuse.

Use it when there is no per-pixel error, or something non-linear happens after the average.

C. `pointwise` and `calculate_score`¶

class MyPSNRWithMap(Metric):
    @property
    def best_min(self) -> bool:
        return False

    def pointwise(self, x: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
        return (x - y) ** 2

    def calculate_score(self, x: torch.Tensor, y: torch.Tensor,
                        y_mask: torch.Tensor = None) -> torch.Tensor:
        mse = self.reduce(self.pointwise(x, y), mask=y_mask)
        return 10 * torch.log10(1.0 / mse.clamp_min(1e-12))

How it runs. Your calculate_score wins: it is what scores each image, and it happens to call your pointwise. The result is identical to B.

What you get. The same as B, plus a pixel map that wrappers can use.

The trap. Wrappers use the map, and only the map. UncertaintyLoss(MyPSNRWithMap()) multiplies the squared error by the uncertainty and averages it: it trains on MSE, and your logarithm is never applied. With log_var set to 0 (no uncertainty at all), the wrapped metric returns exactly each image's MSE. Nothing warns you — the number looks perfectly reasonable.

Use it when the average of your map is a meaningful metric on its own, and you still need your own calculate_score for something else. That is rare; if the map's average is not your metric, don't expose a pointwise (choose B).

D. `pointwise` and `finalize`¶

class PooledPSNR(Metric):
    @property
    def best_min(self) -> bool:
        return False

    def pointwise(self, x: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
        return (x - y) ** 2

    def finalize(self, raw):
        # raw is {"sum", "count"}, already totalled over the epoch
        mse = raw["sum"] / raw["count"]
        return 10 * torch.log10(1.0 / mse.clamp_min(1e-12))

How it runs. Because you override finalize, SR-Forge stops averaging images. For each image it stores two numbers that add up — the sum of the (masked) squared errors and the count of valid pixels — and totals them over the epoch (and across GPUs). Your finalize turns the totals into one PSNR: the pooled PSNR — 23 dB for images A and B.

What you get. An exact pooled value over the whole epoch, with masking still done for you.

What you give up:

no per-image scores. The stored values are ingredients, not scores; asking for per-image views raises an error;
no aggregate choice. The metric is custom: your finalize is the rule;
not part of the training loss. A custom value is left out of the weighted total — it is a report-only metric;
no wrappers. UncertaintyLoss refuses it, since it needs per-image scores.

Use it when you want one specific epoch number and nothing else. If you want PSNR that can be both macro and micro, and trained on, use F.

E. `calculate_score` and `finalize`¶

class WorstPSNR(Metric):
    """The worst image's PSNR over the epoch."""

    @property
    def best_min(self) -> bool:
        return False

    @property
    def reductions(self):
        return "min"                         # combine image scores by minimum

    def calculate_score(self, x: torch.Tensor, y: torch.Tensor,
                        y_mask: torch.Tensor = None) -> torch.Tensor:
        mse = self.reduce((x - y) ** 2, mask=y_mask)
        return 10 * torch.log10(1.0 / mse.clamp_min(1e-12))

    def finalize(self, raw):
        return raw                           # already the minimum

How it runs. You score each image as in B. reductions tells SR-Forge how to combine those scores across batches and GPUs — here, keep the minimum. finalize receives the combined result and returns the number.

What you get. Any epoch rule built from per-image quantities you choose: a worst case, a total, a ratio of totals. Valid rules are sum, mean, min, max and cat.

What you give up. Everything D gives up (no per-image views, no aggregate, not in the training loss), plus everything B gives up (masking is your job, no wrappers).

Use it when the epoch number is something other than an average. Details in epoch rules.

F. `calculate_score` and `finalize`, offering macro and micro¶

This is how SR-Forge's own PSNR is written.

class FlexiblePSNR(Metric):
    aggregations = ("macro", "micro")        # what users may choose; first = default

    @property
    def best_min(self) -> bool:
        return False

    def calculate_score(self, x: torch.Tensor, y: torch.Tensor,
                        y_mask: torch.Tensor = None):
        squared = (x - y) ** 2
        if self.aggregate == "micro":
            return self._sum_and_count(squared, y_mask)    # ingredients
        return self._psnr(self.reduce(squared, mask=y_mask))

    def finalize(self, raw):                 # only used under micro
        return self._psnr(raw["sum"] / raw["count"])

    @staticmethod
    def _psnr(mse):
        return 10 * torch.log10(1.0 / mse.clamp_min(1e-12))

How it runs. Declaring aggregations lets users pick. Under macro it behaves exactly like B and finalize is never called. Under micro, calculate_score returns {sum, count} (via _sum_and_count), SR-Forge totals them, and finalize computes the pooled PSNR — over the batch during training, over the epoch for reporting.

What you get. Both answers from one class — 30 dB with macro, 23 dB with micro for images A and B — and both can be trained on: a micro value enters the training loss as the pooled value over the batch.

What you give up. A little more code than B. Masking is still your job, and there are still no wrappers. Declaring "micro" without a finalize fails as soon as the class is defined.

Use it when you need a per-image transform (as in B) and users should be able to choose micro.

G. All three¶

class FullPSNR(FlexiblePSNR):
    def pointwise(self, x: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
        return (x - y) ** 2

    def calculate_score(self, x: torch.Tensor, y: torch.Tensor,
                        y_mask: torch.Tensor = None):
        squared = self.pointwise(x, y)
        if self.aggregate == "micro":
            return self._sum_and_count(squared, y_mask)
        return self._psnr(self.reduce(squared, mask=y_mask))

How it runs. Exactly like F. The only difference is that the pixel map is also available to wrappers — so the trap from C applies again: a wrapper would train on the squared error, not on PSNR.

Use it when you would choose C, but also need micro. Rare.

H. None of them¶

A metric that writes neither pointwise nor calculate_score has nothing to compute. It is refused the first time it is called, with a message saying to write one of the two.

Side by side¶

	You write	Per-image scores	Masking	`aggregate`	In the training loss	`UncertaintyLoss` can wrap it
A	`pointwise`	yes	automatic	macro, micro	yes	yes
B	`calculate_score`	yes	pass `y_mask` to `reduce`	macro	yes	no
C	`pointwise` + `calculate_score`	yes	pass `y_mask` to `reduce`	macro	yes	yes — but it uses the map, not your score
D	`pointwise` + `finalize`	no	automatic	custom	no	no
E	`calculate_score` + `finalize` + `reductions`	no	pass `y_mask` to `reduce`	custom	no	no
F	`calculate_score` + `finalize` + `aggregations`	yes, under macro	pass `y_mask` to `reduce`	macro, micro	yes	no
G	all three + `aggregations`	yes, under macro	pass `y_mask` to `reduce`	macro, micro	yes	yes — but it uses the map, not your score

Read it this way: every method you write takes a job away from SR-Forge. Writing only pointwise (A) leaves SR-Forge the most to do for you. Each extra method buys control, and costs some of that help.

4. The required pieces¶

Whatever you write, every metric needs:

best_min — True if lower is better.
Type annotations on every input parameter. The parameter names become the names you bind in io, and the annotations tell SR-Forge how to treat dict inputs. A missing annotation is an error when the metric is built.
**kwargs forwarded to super().__init__, if you write your own __init__ — so the options every metric shares (weight, name, reduction, aggregate) keep working:

class Huber(Metric):
    def __init__(self, delta: float = 1.0, **kwargs):
        super().__init__(**kwargs)           # weight, name, aggregate, ...
        self.delta = delta

    @property
    def best_min(self) -> bool:
        return True

    def pointwise(self, x: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
        err = (x - y).abs()
        quadratic = err.clamp(max=self.delta)
        return 0.5 * quadratic ** 2 + self.delta * (err - quadratic)

Unknown options are rejected rather than silently ignored.

5. Choosing the inputs¶

The x / y / y_mask names of the built-in metrics are a convention, not a rule. A metric's inputs are the parameters of the method you write:

pointwise (and no calculate_score): pointwise's parameters, plus y_mask, which SR-Forge adds and applies for you.
calculate_score: its parameters, exactly as written.

So a metric can take one tensor, three, or anything it needs, named however reads best. A per-pixel weighted L1, whose weight map comes from the model:

class WeightedL1(Metric):
    """L1 where each pixel counts as much as the model's weight map says."""

    @property
    def best_min(self) -> bool:
        return True

    def pointwise(self, pred: torch.Tensor, target: torch.Tensor,
                  weight: torch.Tensor) -> torch.Tensor:
        return weight * (pred - target).abs()

- _target: my_project.metrics.WeightedL1
  io:
    inputs: {pred: sr, target: hr, weight: importance, y_mask: valid_mask}

A metric with no reference at all takes one input — pointwise(self, x) — and is bound with {x: sr} (plus y_mask if you mask it).

What to keep in mind:

y_mask is reserved for pointwise metrics. SR-Forge adds it and applies it to the map, so a pointwise that declares its own y_mask is refused when the class is defined. To handle the mask yourself, write calculate_score.
A parameter without a default is required. If its field is missing, the call fails naming it. Give optional inputs a default of None.
Some wrappers expect the convention. CorrectedLoss hands its base metric fields named x, y, y_mask, so a base with other names needs its own inner io mapping (Using Metrics §10). UncertaintyLoss has no such limit: it takes whatever inputs its base's pointwise does, plus log_var.

Following the convention where it fits is still worthwhile: a metric that takes x, y, y_mask drops into every wrapper and every config written for the built-ins without extra mapping.

6. Masks in your own `calculate_score`¶

When you write calculate_score, SR-Forge no longer applies the mask for you. Either honour it or refuse it — never accept it and ignore it, because the number will still look reasonable.

Per-pixel work inside calculate_score (like PSNR): pass the mask to self.reduce(map, mask=y_mask). It applies the same shape rules as the automatic path.
Windowed work (SSIM, gradients, anything built on a convolution): compute the map on the real inputs, then average it over self.erode_mask(mask, window_size) — the positions whose whole window is valid. Do not zero the masked inputs first: the zeros end up inside the neighbouring windows and are scored.
No sensible masked version: refuse it — raise NotImplementedError("masking is not supported") when y_mask is given.

To receive a whole dict of bands in one call instead of band by band — SAM does, because its angle needs every band at once — annotate the parameters (of pointwise or calculate_score) as a dict: Dict[str, torch.Tensor], or a Union that includes one.

7. `reduction` in your own `calculate_score`¶

reduction (mean, sum, none) decides how a per-pixel map becomes one value per image. It acts inside self.reduce(...), so it means something only for a metric whose score goes through reduce(). SR-Forge accepts it where it does and refuses it where it doesn't — an option that is accepted and silently ignored gives a plausible, wrong number.

When you write only pointwise, SR-Forge calls reduce() itself and knows reduction applies. When you write your own calculate_score, it can't see inside, so it assumes you don't call reduce(). If you do, you tell it with one line on the class:

class LogL1(Metric):
    uses_reduction = True                               # calculate_score calls self.reduce()

    @property
    def best_min(self) -> bool:
        return True

    def calculate_score(self, x: torch.Tensor, y: torch.Tensor,
                        y_mask: torch.Tensor = None) -> torch.Tensor:
        l1 = self.reduce((x - y).abs(), mask=y_mask)    # map -> one value per image
        return torch.log1p(l1)                          # something after the reduction

Users can now pick LogL1(reduction="sum") — the log of the summed error — or keep mean, the default.

When to write it. Only when both are true:

your metric has its own calculate_score (not just pointwise), and
inside it, you call self.reduce(...) on a per-pixel map.

How your metric is written	Example	`reduction`	You write
only `pointwise` — the score is the average of a per-pixel error	L1, MSE, SAM	works automatically	nothing
own `calculate_score` that calls `self.reduce(...)`, then does something with the result — a log, a sqrt, a ratio	PSNR, `LogL1` above	works once declared	`uses_reduction = True`
own `calculate_score` with no per-pixel map to reduce — windowed, counts, a whole-image network	SSIM, F1, TBE	refused automatically	nothing

You never write uses_reduction = False: it is already what SR-Forge assumes for your own calculate_score.

Forgetting it in the middle row breaks nothing by default: the metric works with reduction="mean". Only someone asking for "sum" or "none" gets an error, and it tells you the line to add. Adding it without calling reduce() is the real mistake: users could then set reduction: sum and it would be silently ignored — exactly what the refusal exists to prevent. Add it only when the reduce() call is really there.

8. Epoch rules in more depth¶

Variants D, E and F write finalize. What it receives depends on the rest of the metric:

Your metric	`finalize` receives
`pointwise` + `finalize` (D)	`{"sum", "count"}`, totalled over the epoch
`calculate_score` returning a tensor, with `reductions` (E)	that tensor, combined by the rule
`calculate_score` returning a dict of parts, with `reductions`	a dict of combined parts
micro via `_sum_and_count` (F)	`{"sum", "count"}`, totalled

reductions declares what each part is — sum, mean, min, max or cat — as a single string, or a dict per part. SR-Forge then combines each part correctly, across batches and across GPUs.

Store what adds up. Keep counts and sums, never ratios or means: a mean has already thrown away how many values it was taken over, so averaging means silently gives large and small images the same weight. If your quantity has a denominator, carry the numerator and the denominator as two parts, and divide in finalize.

A total over the epoch, as parts:

class FlaggedPixels(Metric):
    """Total number of pixels predicted positive, over the epoch."""

    @property
    def best_min(self) -> bool:
        return False

    @property
    def reductions(self):
        return {"flagged": "sum"}

    def calculate_score(self, x: torch.Tensor) -> dict:
        return {"flagged": (x >= 0.5).flatten(1).sum(1).float()}   # per image

    def finalize(self, raw):
        return raw["flagged"]                 # already summed

If what you want is a pooled ratio such as F1 or IoU, use or subclass the classification metrics — they already do this properly.

9. Using your metric from a config¶

Reference it by its full import path, or register it for a short name — see Making classes available in YAML:

- _target: my_project.metrics.MyMSE
  params: {weight: 0.5, aggregate: micro}
  io:
    inputs: {x: sr, y: hr, y_mask: valid_mask}

Next: back to Using Metrics for the wrappers (CorrectedLoss, UncertaintyLoss, LossScheduler), or on to Writing Scripts.