​
Metrics#

Available metrics include:

AccuracyMetricPerplexityMetric
DiceCoefficientMetricMeanIOUMetric
FBetaScoreMetric

​
metrics.Metric

​
class cerebras.pytorch.metrics.Metric**(*args, **kwargs)**

[source]#

Base class for implementing metrics compatible with Cerebras WSC.

This class is designed to be used as a base class for implementing metrics compatible with Cerebras Wafer-Scale Cluster, but they also work with CPU and GPU backends.

To implement a new metric, subclass Metric and implement the following: - reset: This is to initialize the metric state. - update: This is to update the metric state at every iteration. - compute: This is to compute the final metric value based on the state.

To use metrics, instantiate them and call them with the appropriate inputs. For example:

>>> metric = MyMetric()
>>> metric(input_1, input_2)  # Calls update and compute
>>> metric.compute()  # Returns the final (cached) metric value

Constructs a Metric instance.

Parameters: name (str) – The name of the metric. This is used to reference the metric and does not have to be unique.

_registry_** = **#

_property_ num_updates_: int#

Returns the number of times the metric was updated.

​
abstract reset()

[source]#

Resets the metric state.

​
abstract update(*args, _kwargs_)**[source]#

Updates the metric state.

​
abstract compute()[source]#

Computes and returns the current metric value.

​
register_state(name, tensor, persistent=False)[source]#

Registers a state variable to the module.

By default, metric state variables are non-persistent buffers that are not included in the module’s state dictionary. To have them as part of the state dictionary, set persistent=True.

Once registered, the state variable can be accessed as an attribute on the module by the given name.

Parameters:

  • name (str) – The name of the state variable.

  • tensor (torch.Tensor) – The tensor to register.

  • persistent (bool) – Whether this state is part of the module’s state_dict.

forward(*args, _kwargs_)**[source]#

Updates and computes the metric value.

​
metrics.AccuracyMetric#

class cerebras.pytorch.metrics.AccuracyMetric(*args, _kwargs_)**[source]#

Computes the accuracy of the model’s predictions

Parameters: name – Name of the metric

Constructs a Metric instance.

Parameters: name – The name of the metric. This is used to reference the metric and does not have to be unique.

reset()[source]#

update(labels, predictions, weights=None, dtype=None)[source]#

compute()[source]#

​
metrics.PerplexityMetric#

class cerebras.pytorch.metrics.PerplexityMetric(*args, _kwargs_)**[source]#

Computes the perplexity of the model’s predictions

Parameters: name – Name of the metric

Constructs a Metric instance.

Parameters: name – The name of the metric. This is used to reference the metric and does not have to be unique.

reset()[source]#

update(labels, loss, weights=None, dtype=None)[source]#

compute()[source]#

​
metrics.DiceCoefficientMetric#

class cerebras.pytorch.metrics.DiceCoefficientMetric(*args, _kwargs_)**[source]#

Dice Coefficient is a common evaluation metric for semantic image segmentation.

Dice Coefficient is defined as follows: Dice = 2 * true_positive / (2 * true_positive + false_positive + false_negative).

The predictions are accumulated in a confusion matrix, weighted by weights, and dice coefficient is then calculated from it.

Parameters:

  • num_classes – The possible number of labels the prediction task can have. This value must be provided, since a confusion matrix of dimension = [num_classes, num_classes] will be allocated.

  • name (Optional_[str]_) – Optional string which indicates name of the metric. If None or empty string, it defaults to the name of the class.

reset()[source]#

update(labels, predictions, weights=None, dtype=None)[source]#

Updates the dice coefficient metric.

Parameters:

  • labels – A Tensor of ground truth labels of type int32 or int64.

  • predictions – A Tensor of prediction results for semantic labels, of type int32 or int64.

  • weights – Optional Tensor whose rank is either 0, or the same rank as labels, and must be broadcastable to labels (i.e., all dimensions must be either 1, or the same as the corresponding labels dimension). If weights is None, weights default to 1. Use weights of 0 to mask values.

Raises: ValueError – If predictions and labels have mismatched shapes, or if weights is not None and its shape doesn’t match predictions

compute()[source]#

​
metrics.MeanIOUMetric#

class cerebras.pytorch.metrics.MeanIOUMetric(*args, _kwargs_)**[source]#

Mean Intersection-Over-Union is a common evaluation metric for semantic image segmentation, which first computes the IOU for each semantic class and then computes the average over classes. iou is defined as follows: IOU = true_positive / (true_positive + false_positive + false_negative). The predictions are accumulated in a confusion matrix, weighted by weights, and mIOU is then calculated from it.

For estimation of the metric over a stream of data, the function creates an update_op operation that updates these variables and returns the mean_iou.

If weights is None, weights default to 1. Use weights of 0 to mask values.

Parameters:

  • num_classes – The possible number of labels the prediction task can have. This value must be provided, since a confusion matrix of dimension = [num_classes, num_classes] will be allocated.

  • name (Optional_[str]_) – Optional string which indicates name of the metric. If None or empty string, it defaults to the name of the class.

reset()[source]#

update(labels, predictions, weights=None, dtype=None)[source]#

Updates the mean IOU metric.

Parameters:

  • labels – A Tensor of ground truth labels of type int32 or int64.

  • predictions – A Tensor of prediction results for semantic labels, of type int32 or int64.

  • weights – Optional Tensor whose rank is either 0, or the same rank as labels, and must be broadcastable to labels (i.e., all dimensions must be either 1, or the same as the corresponding labels dimension).

Raises: ValueError – If predictions and labels have mismatched shapes, or if weights is not None and its shape doesn’t match predictions

compute()[source]#

​
metrics.FBetaScoreMetric#

class cerebras.pytorch.metrics.FBetaScoreMetric(*args, _kwargs_)**[source]#

Calculates F Score from labels and predictions.

fbeta = (1 + beta^2) * (precision * recall) / ((beta^2 * precision) + recall)

Where beta is some positive real factor. :param num_classes: Number of classes. :param beta: Beta coefficient in the F measure. :param average_type: Defines the reduction that is applied. Should be one

of the following: - ‘micro’ [default]: Calculate the metric globally, across all

samples and classes.

  • ‘macro’: Calculate the metric for each class separately, and

    average the metrics across classes (with equal weights for each class). This does not take label imbalance into account.

Parameters:

  • ignore_labels (Optional_[List]_) – Integer specifying a target classes to ignore.

  • name (Optional_[str]_) – Name of the metric

reset()[source]#

update(labels, predictions, dtype=None)[*source]#

compute()[source]#

​
metrics.get_all_metrics#

cerebras.pytorch.metrics.get_all_metrics()[source]#

Get all registered metrics.