evaluate

maite.tasks.evaluate(*, model, metric=None, dataloader=None, dataset=None, batch_size=1, augmentation=None, return_augmented_data=False, return_preds=False, collate_fn=<function default_collate_fn>)

Evaluate a model’s performance on data according to some metric with optional augmentation.

The types handled by all passed components must be compatible to avoid static type checking errors. For example, if the __getitem__ method of a passed dataset returns some InputType in the first element of the return tuple then the model.__call__ argument must be type hinted such that Sequence[InputType] can be assigned to it.

Parameters:

modelSomeModel

Maite Model object.

metricSomeMetric | None, (default=None)

Compatible maite Metric.

dataloaderSomeDataloader | None, (default=None)

Compatible maite dataloader.

datasetSomeDataset | None, (default=None)

Compatible maite dataset.

batch_sizeint, (default=1)

Batch size for use with dataset (ignored if dataset=None).

augmentationSomeAugmentation | None, (default=None)

Compatible maite augmentation.

return_augmented_databool, (default=False)

Set to True to return post-augmentation data as a function output.

return_predsbool, (default=False)

Set to True to return raw predictions as a function output.

collate_fnCallable[[Iterable[tuple[T_input, T_target, T_metadata]]], tuple[Sequence[T_input], Sequence[T_target], Sequence[T_metadata]] ], (default=None)

Callable responsible for transforming an iterable of 3-tuples where each encodes a single datapoint in some batch into a tuple of 3 sequences that each represent a batch of collated inputs, collated targets, and collated metadata, respectively. Defaults to naively push elements from input iterable onto sequences in their order of iteration.

Returns:

tuple[dict[str, Any], Sequence[TargetType], Sequence[tuple[Sequence[InputType], Sequence[TargetType], Sequence[DatumMetadataType]]]]

Tuple of returned metric value, sequence of model predictions, and sequence of data batch tuples fed to the model during inference. The actual types represented by InputType, TargetType, and DatumMetadataType will vary by the AI task of the components provided as input arguments (e.g., image classification or object detection.) Note that the second and third return arguments will be empty if return_augmented_data is False or return_preds is False, respectively.

Raises:

InvalidArgument

If neither a dataloader nor a dataset is provided