generic.Metric

class maite.protocols.generic.Metric(*args, **kwargs)

Generic version of a protocol that specifies a model/data alignment calculation behavior on data batches.

Implementers must provide update, compute, and reset methods as specified in below “Methods” section. Briefly, these methods are designed to update a cache based on a batch of model inference predictions and their intended targets, compute a metric based on current cache contents, and clear the current cache, respectively.

Note: In practice, this class is specialized within AI-problem specific submodules using structural types for TargetType and DatumMetadataType. Implementing this class directly (i.e., without specializing on concrete types) is not recommended. Static type checkers will effectively consider all non-specified type parameters as Any-type, effectively masking potential type incompatibilities.

Attributes:

metadataMetricMetadata

A typed dictionary containing at least an ‘id’ field of type str.

Methods

update(pred_batch: Sequence[InputType], target_batch: Sequence[TargetType], metadata_batch: Sequence[DatumMetadataType]) -> None	Add predictions and targets (and metadata if applicable) to metric’s cache for later calculation.
compute() -> dict[str, Any]	Compute metric value(s) for currently cached predictions and targets, returned as a dictionary.
reset() -> None	Clear contents of current metric’s cache of predictions and targets.