SR-Forge

Metrics from other libraries¶

SR-Forge does not reimplement metrics that already exist elsewhere. Name one from torchmetrics or pyiqa directly in your config and it arrives in training as an ordinary Metric:

metrics:
  - _target: torchmetrics.image.PeakSignalNoiseRatio
    params:
      data_range: 1.0

No wrapper class, no adapter mentioned anywhere. That's the whole feature.

Neither library is installed by SR-Forge

torchmetrics and pyiqa are optional and undeclared — SR-Forge never imports them itself. Install whichever you want: pip install torchmetrics, pip install pyiqa. Name one you don't have and the error tells you which, how to install it, and what else is supported.

Why adapt instead of reimplement¶

A metric is never just a formula — it carries an assumption about what an image is.

NIQE is the clearest case. It scores an image by how far its statistics sit from a model fitted to roughly 125 natural photographs, and pyiqa's pipeline converts RGB to luma before measuring. Neither assumption survives contact with multispectral, hyperspectral, SAR or Bayer data. On SAR in particular, speckle reads as maximally unnatural, so the score behaves as a speckle meter — and a denoiser that destroys genuine texture will appear to improve it.

If SR-Forge shipped its own NIQE, it would be asserting that prior in its own source code, for every user. Adapting instead keeps the judgement where it belongs: you pick the metric, and you own whether it suits your data.

No-reference metrics¶

Most metrics compare against a reference. Some don't — and for real data without ground truth, those are the only ones you have.

pyiqa declares which is which, so SR-Forge binds each metric to exactly the inputs it consumes:

metrics:
  - _target: pyiqa.create_metric
    params:
      metric_name: niqe        # no-reference: takes one image

A no-reference metric binds only x. It is never asked for a reference that does not exist, and binding one in io: is an error rather than a silently ignored key.

Configuring the adapter¶

Usually you don't. An external metric takes the same options as every native metric, with the same defaults, so what you have to set applies unchanged. The only thing to add is what the library itself requires, under params: — torchmetrics' PSNR won't build without data_range.

Option	Default	What it does
`weight`	`1.0`	How much the metric counts in the weighted total — the loss in training, and the number that picks the best checkpoint in validation. `0.0` logs it without counting it.
`aggregate`	`macro`	`micro` pools the metric's own counts and reports what its `compute()` makes of the total. torchmetrics only — pyiqa refuses it, having nothing to pool. See below.
`name`	metric's own	The label used in logs and charts.

Every image gets its own score whatever the library returns; there is nothing to set for that.

To change an option, put it in a sibling adapt: block. params: belongs to the library:

_target: torchmetrics.image.PeakSignalNoiseRatio
params:                  # -> torchmetrics' constructor
  data_range: 1.0
adapt:                   # -> SR-Forge's adapter
  weight: 0.0
  name: psnr

Keeping the two apart means neither vocabulary can shadow the other — a library is free to ship a metric with its own name or weight argument. A mistyped option is rejected, not ignored.

Or name the adapter yourself¶

If you'd rather see the adapter in the config — and have your editor and the API reference list its options — name it as the _target and put the library's metric under inner:

_target: TorchMetricAdapter          # or PyIQAAdapter for a pyiqa metric
params:
  inner:                             # -> the library's metric, as usual
    _target: torchmetrics.image.PeakSignalNoiseRatio
    params: {data_range: 1.0}
  weight: 0.0                        # -> the adapter, like any metric's options
  name: psnr

This builds exactly the same metric as the adapt: example above. The options now sit next to inner, in the adapter's own params:, as they would for any native metric. Two mistakes are caught when the config loads:

options in both places — an adapt: block on inner and options on the adapter. One set would be dropped, so neither is guessed.
the wrong adapter — a torchmetrics metric under PyIQAAdapter, or the reverse. The error names the right library. The plain _target form picks the right adapter by itself.

Why SR-Forge's reduction isn't in the table

SR-Forge's reduction collapses a per-pixel map into each image's score. The library hands over finished scores, never a map, so an adapted metric refuses the option, like every metric without a map. A library's own argument that happens to share the name is unrelated. It goes under params:, and for torchmetrics it only changes speed.

Two things worth understanding¶

Per image or pooled over the epoch¶

aggregate decides what the epoch value means, exactly as for a native metric:

	what you get	matches
default	one score per image, averaged	macro — a native metric's `aggregate="macro"`
`aggregate: micro`	the metric's own counts totalled over every batch and process, `compute()` run once	micro — what torchmetrics would say if it saw the whole epoch at once

For image metrics like PSNR or SSIM the two barely differ. For ratios like F1 or MCC they do, whenever positives are sparse — see micro or macro.

- _target: torchmetrics.classification.BinaryF1Score
  adapt:
    aggregate: micro        # pooled F1 over the epoch

aggregate: micro works with any torchmetrics metric whose state is summed, averaged or min/maxed — which covers the classification scores, PSNR and SSIM. Metrics that store raw predictions (AUROC, FID) are refused with a message naming the state, because those cannot be totalled across processes.

pyiqa's metrics are per-image quality scores with nothing behind them to pool, so they are always macro.

Masking is refused, not ignored¶

Native metrics accept a y_mask. No external library has a mask concept, so adapted metrics deliberately do not accept one:

io:
  inputs:
    x: sr
    y_mask: mask      # error at config resolution, before training starts

Accepting it would compute an unmasked score over masked data and report it as masked. Failing early is the point.

Making a torchmetrics metric faster¶

Skip this unless a torchmetrics metric is slowing your validation down. It changes speed only — the scores are the same either way.

Every image always gets its own score. Most torchmetrics metrics return one number for a whole batch, which is no image's score, so the adapter calls them once per image. pyiqa returns one value per image, so it is called once per batch. Some torchmetrics metrics can be told to return one value per image too. Then the adapter notices, and calls them once per batch:

torchmetrics metric	add under `params:`
SSIM	`reduction: none`
PSNR	`reduction: none` and `dim: [1, 2, 3]` — `reduction` alone still gives one number

- _target: torchmetrics.image.PeakSignalNoiseRatio
  params:
    data_range: 1.0
    reduction: none         # torchmetrics' own argument, not SR-Forge's
    dim: [1, 2, 3]          # PSNR needs this too, to score each image

Nothing changes on the SR-Forge side. The adapter works out which kind of metric it has from the first batch of more than one image.

In Python¶

Code-first scripts never touch the config resolver, so adapt explicitly:

import srforge
from torchmetrics.image import PeakSignalNoiseRatio

metric = srforge.adapt(PeakSignalNoiseRatio(data_range=1.0), aggregate="micro")

Hand a raw metric to anything that holds SR-Forge metrics and it tells you so:

from srforge.metrics.wrappers import CorrectedLoss

CorrectedLoss(PeakSignalNoiseRatio(data_range=1.0))
# TypeError: CorrectedLoss expects a Metric for 'base_metric', but got
#            PeakSignalNoiseRatio, which is a torchmetrics metric.
#            Wrap it first:
#                srforge.adapt(PeakSignalNoiseRatio(...))
#            Configs do this automatically; a Python script has to ask.

SR-Forge deliberately does not wrap it for you. Converting silently would save you one call, at the price of the object you passed not being the object stored — invisible at the call site and in the class's own source. The error names the fix instead, so srforge.adapt(...) stays where a reader can see it.

srforge.adapt returns anything it doesn't recognise unchanged, so it is safe to call on a mixed list. Passing adapter options for something that needs no adapting is an error: they would be dropped, and silently accepting them would leave you believing they took effect. A native Metric takes the same options in its own constructor.

Writing your own metric wrapper¶

You get this check for free. No decorator, no call, no knowledge of adapters — just annotate the parameter as a Metric, which you would write anyway:

class MyWrapper(Metric):
    def __init__(self, base: Metric, **kwargs):
        super().__init__(**kwargs)
        self.base = base

Anyone handing MyWrapper a torchmetrics metric now gets the same message, naming your class and your parameter. The check is installed on every Metric subclass whose __init__ annotates a parameter as holding metrics; omit the annotation and you simply opt out.

Adding another library¶

One adapter class, one registry entry, no changes to the resolver:

from srforge.metrics.adapters import ExternalMetric, register_adapter

@register_adapter(library="piq", module="piq", install="pip install piq")
class PIQAdapter(ExternalMetric):
    @staticmethod
    def detects(obj): ...        # imports inside the body
    @staticmethod
    def _direction(inner): ...   # True when lower is better
    @staticmethod
    def _arity(inner): ...       # "FR" or "NR"
    def _call(self, x, y): ...

The same registry drives dispatch and the "not installed" message, so the set of libraries SR-Forge claims to support and the set it can actually adapt cannot drift apart.

Design notes

The reasoning, the rejected alternatives, and the measurements behind the reduction policy are in docs/design/external-metric-adapters-rfc.md.