> ## Documentation Index
> Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# cerebras.pytorch Package

> Top-level functions for creating and retrieving the Cerebras backend

## Backend related functions

Top-level functions for creating and retrieving the Cerebras backend being used in the current run

### `backend`

#### cerebras.pytorch.backend(*backend\_type=None*, *\*args*, *\*\*kwargs*)

[source](../_modules/cerebras/pytorch/backend.html#backend)[#](#cerebras.pytorch.backend "Permalink to this definition")

Instantiates a backend with the given type.

**Parameters:**

* **backend\_type** (*Optional\[**str**]*) - The type of backend to instantiate. One of “CSX”, “CPU”, “GPU” If no backend\_type is provided, returns the current backend if it exists.

* **args** - Positional arguments to pass to the backend implementation

* **kwargs** - Keyword arguments to pass to the backend implementation

Supported backend types include:

`CSX`

**Parameters:**

* **artifact\_dir** (*str*) – The directory at which to store any Cerebras specific artifacts generated by the backend. Default: `$cwd/cerebras_logs`

* **compile\_dir** (*str*) – The directory at which to store any compile related artifacts. These compile artifacts are used to cache the compile to avoid recompilation. Default: `/opt/cerebras/cached_compile`

* **compile\_only** (*bool*) – If `True`, then configure the `CSX` backend only for compilation. This means that all parameter data is immediately dropped as it isn’t required for compilation. Further, the data executor will not send an execution request to the wafer scale cluster. This mode is intended to verify that the model is able to be compiled. As such, no system is required in this mode. Default: `False`

* **validate\_only** (*bool*) – If `True`, then configure the `CSX` backend only for validation. This means that all parameter data is immediately dropped as it isn’t required for validation. Further, the data executor will not send compile and execute requests to the wafer scale cluster. This mode is intended to verify that the model is able to be traced. As such, no system is required in this mode. Default: `False`

* **drop\_data** (*bool*) – If `True`, all parameter data is immediately dropped even if in a non-compile-only run. In this case, a checkpoint containing values for all stateful tensors must be loaded in order to be able to run. Default: `False`

* **max\_checkpoints** (*int*) – If provided, `cstorch.save` will automatically only keep the newest `max_checkpoints` checkpoints, removing the oldest when the number of checkpoints exceeds the specified number. Default: `None`

* **log\_initialization** (*bool*) – If `True`, print logs during weight initialization to keep the users updated on the current progress. Default: `True`

* **retrace\_every\_iteration** (*bool*) – If `True`, retrace the entire training/evaluation step every iteration. This provides the benefit of being able to check whether the graph changes or not every iteration. But tracing overhead can hurt performance for certain models. Default: `False`

`CPU`

**Parameters:**

* **artifact\_dir** (*str*) – The directory at which to store any Cerebras specific artifacts generated by the backend. Default: `$cwd/cerebras_logs`

* **max\_checkpoints** (*int*) – If provided, `cstorch.save` will automatically only keep the newest `max_checkpoints` checkpoints, removing the oldest when the number of checkpoints exceeds the specified number. Default: `None`

* **mixed\_precision** (*bool*) – If `True`, use the `autocast` context manager during the forward pass. Default: `None`

`GPU`

**Parameters:**

* **artifact\_dir** (*str*) – The directory at which to store any Cerebras specific artifacts generated by the backend. Default: `$cwd/cerebras_logs`

* **enable\_distributed** (*bool*) – If `True`, configure the run to use `torch.distributed`. Note, the run must have been triggered using [torchrun](https://pytorch.org/docs/stable/elastic/run.html_). Default: `False`

* **main\_process\_id** (*int*) - The rank of the main process. Default: `0`

* **dist\_backend** (*str*) - The distributed backend to use to [initialize the process group](https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group). Default: `"nccl"`

* **init\_method** (*str*) - The method to use to [initialize the process group](https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group). Default: `None`

* **sync\_batchnorm** (*bool*) - If True, wraps the model with [convert\_sync\_batchnorm](https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#torch.nn.SyncBatchNorm.convert_sync_batchnorm). Default: `False`

### `current_backend`

cerebras.pytorch.current\_backend(*raise\_exception=True*, *raise\_warning=True*)[\[source\]](../../../_modules/cerebras/pytorch/backend.html#current_backend)[#](#cerebras.pytorch.current_backend "Permalink to this definition")

DEPRECATED: Use cstorch.backend() instead. Gets instance of the current backend.

\*\* Parameters \*\*

**raise\_exception** (*bool*) – If True, raise an exception if no backend has been instantiated. Otherwise return None

### `current_torch_device`

cerebras.pytorch.`current_torch_device`()[\[source\]](../../../_modules/cerebras/pytorch/backend.html#current_torch_device)[#](#cerebras.pytorch.current_torch_device "Permalink to this definition")

Gets the torch device of the current backend.

Returns torch.device(‘cpu’) if no backend has been initialized yet

### `use_cs`

cerebras.pytorch.use\_cs()[\[source\]](../../../_modules/cerebras/pytorch/backend.html#use_cs)[#](#cerebras.pytorch.use_cs "Permalink to this definition")

Returns True if the active device is a CSX device.

## Performance/Debug Flags

`cerebras.pytorch.backends` controls the behavior of specific backends that Cerebras PyTorch API supports.

As of right now, the only backend for which there are configurable options include

* `cerebras.pytorch.backends.csx`

### `cerebras.pytorch.backends.csx`

CSX related performance/debug flags

cerebras.pytorch.backends.csx.precision.optimization\_level:  *int*[#](#cerebras.pytorch.backends.csx.precision.optimization_level "Permalink to this definition")

The precision optimization level (POL) to use when compiling the model. The POL determines the level of precision to use for the model’s weights and activations and can thus affect the model’s accuracy and performance.

The value must be an integer in range `[0, 3)`. Default: `1`

cerebras.pytorch.backends.csx.performance.micro`_batch_size`:  *Union\[None,  int,  Literal\['auto',  'explore'],  Dict\[str,  Dict\[str,  int]]]*[#](#cerebras.pytorch.backends.csx.performance.micro_batch_size "Permalink to this definition")

The micro-batch size to use when compiling the model. The micro-batch size can affect the model’s performance and memory usage.

Valid values include:

* “auto”: Automatically choose an optimal micro batch size.

* “explore”: Search for an optimal micro batch size and return.

* ```
    {
      "explore": {
          "min": Optional[<positive_int>],
          "max": Optional[<positive_int>]
      }
    }
  ```
  Search for an optimal micro batch size within the min and max bounds and return.

* Positive `int`: Use this micro batch size.

* None: Disable micro batch tiling.

The default value is `"auto"`.

**cerebras.pytorch.backends.csx.performance.**`transfer_processes`**:**  ***int***[#](#cerebras.pytorch.backends.csx.performance.transfer_processes "Permalink to this definition")

The number of processes to use for transferring data to and from the Wafer Scale Cluster.

The default value is `5`.

**cerebras.pytorch.backends.csx.debug.**`retrace_every_iteration`**:**  ***bool*\_**[#](#cerebras.pytorch.backends.csx.debug.retrace_every_iteration "Permalink to this definition")

Whether to retrace the training/validation graph every iteration. Default: `False`.

**cerebras.pytorch.backends.csx.debug.**`lazy_initialization`**:** ***bool***[#](#cerebras.pytorch.backends.csx.debug.lazy_initialization "Permalink to this definition")

Whether to use lazy weight initialization. Default: `True`.

**cerebras.pytorch.backends.csx.debug.**`debug_args`**:** ***DebugArgs***[#](#cerebras.pytorch.backends.csx.debug.debug_args "Permalink to this definition")

Arguments to pass to the cluster for cluster debugging purposes only. Default: `None`.

**cerebras.pytorch.backends.csx.debug.**`ini`**:** ***DebugArgs***[#](#cerebras.pytorch.backends.csx.debug.ini "Permalink to this definition")

INI configuration flags for cluster debugging purposes only. Default: `None`.

\***cerebras.pytorch.backends.csx.debug.**`compile_crd_memory_gi`**:**  ***Optional\[int]***[#](#cerebras.pytorch.backends.csx.debug.compile_crd_memory_gi "Permalink to this definition")

The memory limit for the compile coordinator.

**cerebras.pytorch.backends.csx.debug.**`execute_crd_memory_gi`**:**  ***Optional\[int]***[#](#cerebras.pytorch.backends.csx.debug.execute_crd_memory_gi "Permalink to this definition")

The memory limit for the execute coordinator.

**cerebras.pytorch.backends.csx.debug.**`wrk_memory_gi`**:** ***Optional\[int]***[#](#cerebras.pytorch.backends.csx.debug.wrk_memory_gi "Permalink to this definition")

**cerebras.pytorch.backends.csx.debug.**`wrk_memory_gi`**:** ***Optional\[int]***[#](#cerebras.pytorch.backends.csx.debug.wrk_memory_gi "Permalink to this definition")

The memory limit for the workers.

**cerebras.pytorch.backends.csx.debug.**`act_memory_gi`**:** ***Optional\[int]***[#](#cerebras.pytorch.backends.csx.debug.act_memory_gi "Permalink to this definition")

The memory limit for the activation hosts.

**cerebras.pytorch.backends.csx.debug.**`cmd_memory_gi`**:** ***Optional\[int]***[#](#cerebras.pytorch.backends.csx.debug.cmd_memory_gi "Permalink to this definition")

The memory limit for the command hosts.

**cerebras.pytorch.backends.csx.debug.**`wgt_memory_gi`**:** ***Optional\[int]***[#](#cerebras.pytorch.backends.csx.debug.wgt_memory_gi "Permalink to this definition")

The memory limit for the weight hosts.

## Compile related functions

### `compile`

#### **cerebras.pytorch**.`compile`**(model, backend=None)**

[\[source\]](../../../_modules/cerebras/pytorch/core/compile.html#compile)[#](#cerebras.pytorch.compile "Permalink to this definition")

Prepares the PyTorch module for tracing.

This method prepares the module by moving it to the device so that it can be compiled after the first trace. Note that parameter initialization must be done before calling this method since post this call, the parameters are moved to the device.

**Parameters:**

* **model** ([*torch.nn.Module*](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.4)")) – The PyTorch module to be compiled.

* **backend** (*Optional*\[**Union**\[**str**,*cerebras.pytorch.backend.Backend*]]) – The Cerebras backend to use to compile. If None, the current backend is used. If not current backend is set, the CPU backend is initialized and used. Defaults to None.

**Returns:**

A pseudo-module that almost acts like the original module but does not have any of the property accessor or private methods of the original module. It can be called module(\*args, \*\*kwargs) to run the forward pass, similar to the original module.

### `trace`

cerebras.pytorch.trace(*step\_fn*)[\[source\]](../../../_modules/cerebras/pytorch/core/compile.html#trace)[#](#cerebras.pytorch.trace "Permalink to this definition")

A decorator that wraps the training/evaluation step function for tracing.

Any operation that is meant to be executed on the Cerebras Wafer-Scale Cluster must be wrapped with this decorator. This includes the forward pass, backward pass, optimizer steps, and more.

For example, the following code snippet shows how to wrap a training step that does the forward and backward pass and optimizer step:

```
@cstorch.trace
def training_step(batch, model, optimizer, loss_fn):
    features, labels = batch
    outputs = model(features)
    loss = loss_fn(outputs, labels)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    return loss
```

`Parameters:` **step\_fn** (*callable*) – The training/evaluation step function to be wrapped.

\*\*Returns:\*\*The wrapped training/evaluation step function.

\*\*Return type:\*\*callable

In addition, no tensor value may be eagerly evaluated at any point inside this training step. This means, no tensor is allowed to be printed, fetched via a debugger, or used as part of a python conditional. Any operation that requires knowing the value of tensor inside the training step will result in an error stating that it is not allowed to read a tensor’s contents outside of a [`step_closure`](#cerebras.pytorch.step_closure "cerebras.pytorch.step_closure")

## Creation Ops[#](#creation-ops "Permalink to this headline")

Can be used to lazily initialize tensors with known shape, dtype and value to avoid have them unnecessarily take up memory.

### `full`[#](#full "Permalink to this headline")

**cerebras.pytorch.**`full`(***shape*, *value*, *dtype=None***)[\[source\]](../../../_modules/cerebras/pytorch/saver/storage.html#full)[#](#cerebras.pytorch.full "Permalink to this definition")

Returns an lazily initialized tensor filled with the provided value

**Parameters:**

* **shape** – The shape of the tensor.

* **value** (*float*) – The value to fill the tensor with.

* **dtype** – The dtype of the tensor.

### `full_like`

**cerebras.pytorch**.`full_like`(***other*, *value*, *dtype=None***)[\[source\]](../../../_modules/cerebras/pytorch/saver/storage.html#full_like)[#](#cerebras.pytorch.full_like "Permalink to this definition")

Returns an lazily initialized full tensor with the same properties as the provided tensor

**Parameters:**

* **other** ([*torch.Tensor*](https://pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.4)")) – The tensor to copy the properties from

* **value** (*float*) – The value to fill the tensor with

* **dtype** – The dtype of the tensor. If not provided, the dtype of the other tensor is used

### `ones`

**cerebras.pytorch**.`ones`(***shape*, *dtype=None***)[\[source\]](../../../_modules/cerebras/pytorch/saver/storage.html#ones)[#](#cerebras.pytorch.ones "Permalink to this definition")

Returns an lazily initialized tensor filled with ones

**Parameters:**

* **shape** – The shape of the tensor

* **dtype** – The dtype of the tensor

### `ones_like`

**cerebras.pytorch.**`ones_like`(***other*, *dtype=None***)[\[source\]](../../../_modules/cerebras/pytorch/saver/storage.html#ones_like)[#](#cerebras.pytorch.ones_like "Permalink to this definition")

Returns an lazily initialized tensor full of ones with the same properties as the provided tensor

Parameters

* **other** ([*torch.Tensor*](https://pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.4)")) – The tensor to copy the properties from

* **dtype** – The dtype of the tensor. If not provided, the dtype of the other tensor is used

### `zeros`[#](#zeros "Permalink to this headline")

**cerebras.pytorch.**`zeros`(***shape*, *dtype=None***)[\[source\]](../../../_modules/cerebras/pytorch/saver/storage.html#zeros)[#](#cerebras.pytorch.zeros "Permalink to this definition")

Returns an lazily initialized tensor filled with zeros

**Parameters:**

* **shape** – The shape of the tensor

* **dtype** – The dtype of the tensor

### `zeros_like`[#](#zeros-like "Permalink to this headline")

**cerebras.pytorch.**`zeros_like`(***other*, *dtype=None***)[\[source\]](../../../_modules/cerebras/pytorch/saver/storage.html#zeros_like)[#](#cerebras.pytorch.zeros_like "Permalink to this definition")

Returns an lazily initialized tensor full of zeros with the same properties as the provided tensor

**Parameters:**

* **other** ([*torch.Tensor*](https://pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.4)")) – The tensor to copy the properties from

* **dtype** – The dtype of the tensor. If not provided, the dtype of the other tensor is used

## Checkpoint Saving/Loading utilities

#### **cerebras.pytorch.**`save`(***obj*, *checkpoint\_file***)

[\[source\]](../../../_modules/cerebras/pytorch/saver.html#save)[#](#cerebras.pytorch.save "Permalink to this definition")

Save a PyTorch state dict to the given file.

**Parameters:**

* **obj** (*dict*) – The object to save.

* **checkpoint\_file** (*str*) – The path to save the object to.

#### **cerebras.pytorch**.`load`(***checkpoint\_file*, *map\_location=None*, \_**kwargs\_\*\*)

Load a PyTorch checkpoint from a file.

Parameters

* **checkpoint\_file** (*Union*\_\[**cerebras.appliance.utils.file.StrPath**,\_ *IO*\_]\_) – The path to the checkpoint to load.

* **map\_location** (*Optional*\_\[**Union**\[**str**,\_ [*torch.device*](https://pytorch.org/docs/stable/tensor_attributes.html#torch.device "(in PyTorch v2.4)")*,* *Callable*\_,\_ *dict*\_]\_\_]\_) – A mapping of where to load the checkpoint content to. If the map\_location is None, then the tensors will be lazily loaded from the checkpoint file every single time the tensor is accessed. If the map\_location is “cache”, then the tensors will be cached once they are lazily loaded from the checkpoint file. If the map location is “cpu”, then the tensors will be eagerly loaded into memory from the checkpoint file.

* \*\***kwargs** – Additional keyword arguments to pass to the vanilla torch checkpoint loader. These are ignored if the checkpoint is a Cerebras HDF5 checkpoint.

**Returns:** The loaded checkpoint file.

**Raises:** **RuntimeError** – If the checkpoint file does not exist or checkpoint is not a valid HDF5 or vanilla torch checkpoint.

**Return type:** *Any*

## Data Utilities

### `utils.data.DataLoader`[#](#utils-data-dataloader "Permalink to this headline")

#### ***class* cerebras.pytorch.utils.data.**`DataLoader`(***\*args*, \_**kwargs\_\*\*)

Wrapper around torch.utils.data.DataLoader that facilitates moving data generated by the dataloader to a Cerebras system

Parameters

* **input\_fn** (*Callable*\_\[**\[**...**]**,\_ *Union*\_\[*[*torch.utils.data.DataLoader*](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader "(in PyTorch v2.4)")*,\_ *Iterable*\_]\_\_]\_) – A callable that returns a torch.utils.data.DataLoader instance or an iterable that returns a structure containing torch tensors.

* \***args** – Any other positional or keyword arguments are passed into the input\_fn when each worker instantiates their respective dataloaders

* \*\***kwargs** –

  Any other positional or keyword arguments are passed into the input\_fn when each worker instantiates their respective dataloaders

### \`state\_dict()

#### `load_state_dict()`

Each worker will call this input function to construct their own dataloader object. This means that some data sharding scheme is required if the intent is for each worker to stream in a unique set of data.

### `utils.data.SyntheticDataset`

***class* cerebras.pytorch.utils.data.**`SyntheticDataset`(***\*args*, \_**kwargs\_\*\*)[\[source\]](../../../_modules/cerebras/pytorch/utils/data/dataset.html#SyntheticDataset)[#](#cerebras.pytorch.utils.data.SyntheticDataset "Permalink to this definition")

A synthetic dataset that generates samples from a SampleSpec.

Constructs a SyntheticDataset instance.

A synthetic dataset can be used to generate samples on the fly with an expected dtype/shape but without needing to create a full-blown dataset. This is especially useful for compile validation.

**Parameters:**

* **sample\_spec** (*Union*\_\[*[*torch.Tensor*](https://pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.4)")*,\_ *Callable*\_\[**\[**int**]**,\_ [*torch.Tensor*](https://pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.4)")*]\_\_,* *List*\_\[**SampleSpecT**]**,\_ *Tuple*\_\[**SampleSpecT**,\_ \_...**]**,\_ *Dict*\_\[**str**,\_ *SampleSpecT*\_]**,\_ *OrderedDict*\_\[**str**,\_ *SampleSpecT*\_]\_\_,\_ *NamedTuple*\_]\_) –

  Specification of the samples to generate. This can be a nested structure of one of the following types:

  * torch.Tensor: A tensor to be cloned.
  * Callable: A callable that takes the sample index and
    returns a tensor.

  Supported data structures for holding the above leaf nodes are list, tuple, dict, OrderedDict, and NamedTuple.

* **num\_samples** (*Optional* *\[int]*) – Total size of the dataset. If None, the dataset will generate samples indefinitely.

### `utils.data.DataExecutor`[#](#utils-data-dataexecutor "Permalink to this headline")

#### ***class* cerebras.pytorch.utils.data.**`DataExecutor`(*\_*args\_, *\*\*kwargs*\*)

Defines a single execution run on a Cerebras wafer scale cluster.

**Parameters:**

* **dataloader** (*cerebras.appliance.log.named\_class\_logger*) – the dataloader to use for the run

* **num\_steps** (*Optional*\_\[**int**]\_) – the number of steps to run. Defaults to 1 if the backend was configured for compile or validate only

* **checkpoint\_steps** (*Optional*\_\[**Union**\[**int**,\_ *cerebras.pytorch.utils.data.utils.Schedule*\_]\_\_]\_) –

  Checkpoint cadence. This can be: - None: To disable checkpointing. This is the default. - int: To take checkpoints at this frequency and at num\_steps. - Schedule: To take checkpoints at the given arbitrary schedule. Note that when

  > using a schedule object, the intervals must be zero-indexed.

* **activation\_steps** (*Optional*\_\[**int**]\_) – the interval at which to schedule fetching activations from the cluster

* **cs\_config** (*Optional*\_\[**cerebras.appliance.CSConfig.CSConfig**]\_) – (DEPRECATED) An optional CSConfig object

* **writer** (*Optional*\_\[*[*SummaryWriter*](#cerebras.pytorch.utils.tensorboard.SummaryWriter "cerebras.pytorch.utils.tensorboard.SummaryWriter")*]\_) – (DEPRECATED) The summary writer to be used to write any summarized scalars or tensors to tensorboard

* **profiler\_activities** (*Optional*\_\[**List**\[**Type**\[**cerebras.pytorch.utils.profiler.Activity**]**]**]\_) – The list of activities to profile. By default the total samples, the client side rate and global rate are tracked and accessible via the profiler attribute.

* **micro\_batch\_size** (*Union*\_\[**None**,\_ *int*\_,\_ *Literal*\_\[**'explore'**,\_ *'auto'**]**,* *typing.Dict*\_\[**str**,\_ *typing.Dict*\_\[**str**,\_ *int*\_]**]**]\_) – (DEPRECATED) Can be one of “auto”, “explore”, or an int representing the micro batch size that the Cerebras compiler should use. Only applicable in CSX runs.

<Info>
  Note

  As of Cerebras Release 2.0, we don’t officially support multiple CS runs in a single process. This means that the above executor can only be run/iterated once. Any runs with different configurations must be run in separate processes.
</Info>

### `utils.data.RestartableDataLoader`[#](#utils-data-restartabledataloader "Permalink to this headline")

#### **class* cerebras.pytorch.utils.data.*`RestartableDataLoader`(***\*args*, \_**kwargs\_\*\*)

[\[source\]](../../../_modules/cerebras/pytorch/utils/data/dataloader.html#RestartableDataLoader)[#](#cerebras.pytorch.utils.data.RestartableDataLoader "Permalink to this definition")

Defines interface for the restartable dataloader protocol.

#### `state_dict()`[\[source\]](../../../_modules/cerebras/pytorch/utils/data/dataloader.html#RestartableDataLoader.state_dict)[#](#cerebras.pytorch.utils.data.RestartableDataLoader.state_dict "Permalink to this definition")

Use this method to specify what state information should be saved by each CSX Worker.

**Returns:** dict holding state information for the CSX Worker

**Return type:** *Dict*\[str, *Any*]

In order to access Cerebras internal data checkpoint info per CSX Worker at some checkpoint step, follow the steps in the example below. Cerebras internal data checkpoint format is recorded in the [`DataLoaderCheckpoint`](#cerebras.pytorch.utils.data.DataLoaderCheckpoint "cerebras.pytorch.utils.data.DataLoaderCheckpoint") dataclass.

Usage:

```
import cerebras.pytorch as cstorch
...
def state_dict(self) -> Dict[str, Any]:
    worker_state = cstorch.distributed.get_worker_state()
    state_dict = {}
    if worker_state:
        state_dict["worker_step"] = worker_state.worker_step
        state_dict["worker_id"] = worker_state.global_worker_id
    return state_dict
```

<Info>
  Note

  The call to [`get_worker_state`](#cerebras.pytorch.distributed.get_worker_state "cerebras.pytorch.distributed.get_worker_state") is well-defined only inside of the state\_dict method; using this anywhere else will result in a RuntimeError exception. See linked docs for more details.
</Info>

#### `load_state_dict`**(*state\_dict*, *strict=True*)**

[\[source\]](../../../_modules/cerebras/pytorch/utils/data/dataloader.html#RestartableDataLoader.load_state_dict)[#](#cerebras.pytorch.utils.data.RestartableDataLoader.load_state_dict "Permalink to this definition")

Use this method to load CSX Worker state for the dataloader instance, as captured from a previous run.

**Parameters:**

* **state\_dict** (*Dict*\_\[**str**,\_ *Any*\_]\_) – dict holding worker state info, specified in [`deaggregate_state_dict`](#cerebras.pytorch.utils.data.RestartableDataLoader.deaggregate_state_dict "cerebras.pytorch.utils.data.RestartableDataLoader.deaggregate_state_dict")

* **strict** (*bool*) – Whether to enforce strict matching of the incoming state\_dict. It is up to the implementation to decide what “strict matching” is.

Usage:

```
def load_state_dict(self, state_dict, strict=True):
    wrk_state_dict = state_dict.get("worker_0", {})

    worker_step = wrk_state_dict.get("worker_step", 0)
    worker_id = wrk_state_dict.get("worker_id")

    print(f"WRK {worker_id} loaded step: {worker_step}")
```

#### `aggregate_state_dict`**(*worker\_states*)**

[\[source\]](../../../_modules/cerebras/pytorch/utils/data/dataloader.html#RestartableDataLoader.aggregate_state_dict)[#](#cerebras.pytorch.utils.data.RestartableDataLoader.aggregate_state_dict "Permalink to this definition")

Use this method to specify how to combine the list of CSX Worker state dicts. Each CSX Worker state in the worker\_states list is to be specified in [`state_dict`](#cerebras.pytorch.utils.data.RestartableDataLoader.state_dict "cerebras.pytorch.utils.data.RestartableDataLoader.state_dict")

**Returns:** The consolidated state dict that will be saved in a checkpoint.

**Return type:** *Dict*\[str, *Any*]

Usage:

```
def aggregate_state_dict(self, worker_states):
    return {
        "worker_0": worker_states[0],
        "worker_1": worker_states[1]
    }
```

#### `deaggregate_state_dict`**(*aggregated\_state\_dict*, *strict=True*)**

[\[source\]](../../../_modules/cerebras/pytorch/utils/data/dataloader.html#RestartableDataLoader.deaggregate_state_dict)[#](#cerebras.pytorch.utils.data.RestartableDataLoader.deaggregate_state_dict "Permalink to this definition")

Use this method to specify how to load an individual CSX Worker state given a consolidated list of state dicts, as specified in [`aggregate_state_dict`](#cerebras.pytorch.utils.data.RestartableDataLoader.aggregate_state_dict "cerebras.pytorch.utils.data.RestartableDataLoader.aggregate_state_dict").

**Parameters:**

* **aggregated\_state\_dict** (*Dict*\_\[**str**,\_ *Any*\_]\_) – The aggregated state dict to deaggregate.

* **strict** (*bool*) – Whether to enforce strict matching of the incoming state\_dict. It is up to the implementation to decide what “strict matching” is.

**Returns:**

The state dict will be passed to the above-defined [`load_state_dict`](#cerebras.pytorch.utils.data.RestartableDataLoader.load_state_dict "cerebras.pytorch.utils.data.RestartableDataLoader.load_state_dict") method.

**Return type:** *Dict*\[str, *Any*]

Usage:

```
def deaggregate_state_dict(self, aggregated_state_dict, strict=True):
    return {
        "worker_0": aggregated_state_dict.get("worker_0", {})
    }
```

### `utils.data.DataLoaderCheckpoint`

#### ***class* cerebras.pytorch.utils.data.**`DataLoaderCheckpoint`

Dataclass representing the Cerebras internal dataloader checkpoint format. Each CSX Worker captures its state information via this class at a checkpoint step.

**Parameters:**

* **global\_worker\_id** (*int*) – ID of this worker amongst all other workers across all boxes

* **local\_worker\_id** (*int*) – ID of this worker amongst all other workers across the same box

* **total\_num\_workers** (*int*) – The total number of workers for the run across all boxes

* **num\_workers\_per\_csx** (*int*) – The total number of workers per box for the run

* **num\_csx** (*int*) – The total number of CSXs (boxes) for the run

* **wse\_id** (*int*) – ID of the Wafer-Scale Engine (CSX) to which this worker streams data

* **appliance\_step** (*int*) – The appliance step at which this checkpoint state info is captured

* **worker\_step** (*int*) – The worker step at which this state info is captured. Note that this is simply equal to appliance\_step if num\_workers\_per\_csx = 1; for the multi-worker scenario, the appliance step is distributed across workers on a single box in a round-robin fashion based on the local worker id

* **samples\_streamed** (*int*) – The total number of samples streamed by this worker at checkpoint step. This is simply worker\_step \* per\_box\_batch\_size

<Info>
  Note

  appliance\_step, worker\_step and samples\_streamed are the attributes that vary across different steps; whereas the other attributes provide constant state information for the current run.
</Info>

### `get_worker_state`

#### **cerebras.pytorch.distributed**.`get_worker_state()`

[\[source\]](../../../_modules/cerebras/pytorch/distributed.html#get_worker_state)[#](#cerebras.pytorch.distributed.get_worker_state "Permalink to this definition")

API exposing internal state info captured by each CSX Worker for the current run at a checkpoint step. This state info is represented in the `DataLoaderCheckpoint` dataclass format:

**Returns:**

`DataLoaderCheckpoint` instance holding worker state information at the checkpoint step

<Info>
  Note

  * This method may only be called inside of a custom implementation of state\_dict for
</Info>

dataloaders conforming to the `RestartableDataLoader` protocol, since state\_dict is well-defined only at a checkpoint step. - Use this method to save any of the aforementioned state info recorded by each worker when defining state\_dict for custom implementations of restartable dataloaders. - This state info captured by each worker is for the current run only, i.e. if you pause and restart a run, the counters gathering information returned by this function will be reset.

### `utils.CSConfig`

#### ***class* cerebras.pytorch.utils.**`CSConfig`

Contains config details for the Cerebras Wafer Scale Cluster

**Parameters:**

* **mgmt\_address** (*Optional*\_\[**str**]\_) – Address to connect to appliance. If not provided, query the cluster management node for it. Default: `None`.

* **credentials\_path** (*Optional*\_\[**str**]\_) – Credentials for connecting to appliance. If not provided, query the cluster management node for it. Default: `None`.

* **num\_csx** (*int*) – Number of Cerebras Systems to run on. Default: `1`.

* **max\_wgt\_servers** (*int*) – Number of weight servers to support run. Default: `24`.

* **max\_act\_per\_csx** (*int*) – Number of activation servers per system. Default: `1`.

* **num\_workers\_per\_csx** (*int*) – Number of streaming workers per system. Default: `1`.

* **transfer\_processes** (*int*) – Number of processes to transfer data to/from appliance. Default: `5`.

* **job\_time\_sec** (*int*) – Time limit for the appliance jobs, not including the queue time. Default: `None`.

* **mount\_dirs** (*List*\_\[**str**]\_) – Local storage to mount to appliance (ex. training data). Default: `None`.

* **python\_paths** (*List*\_\[**str**]\_) – A list of path that worker pods respect as PYTHONPATH in addition to the PYTHONPATH set in the container image. Default: `None`.

* **job\_labels** (*List*\_\[**str**]\_) – A list of equal-sign-separated key-value pairs that get applied as part of job metadata. Default: `None`.

* **debug\_args** (*DebugArgs*) – Optional debugging arguments object. Default: `None`.

* **precision\_opt\_level** (*int*) – The precision optimization level. Default: `1`.

## `numpy` utilities[#](#numpy-utilities "Permalink to this headline")

### `from_numpy`[#](#from-numpy "Permalink to this headline")

**cerebras.pytorch.**`from_numpy`(*array*)[\[source\]](../../../_modules/cerebras/pytorch/utils/data/utils.html#from_numpy)[#](#cerebras.pytorch.from_numpy "Permalink to this definition")

Converts a numpy array to a torch tensor.

### `to_numpy`[#](#to-numpy "Permalink to this headline")

**cerebras.pytorch.**`to_numpy`(*tensor*)[\[source\]](../../../_modules/cerebras/pytorch/utils/data/utils.html#to_numpy)[#](#cerebras.pytorch.to_numpy "Permalink to this definition")

Converts a torch tensor to a numpy array.

## Step Closure related utilities[#](#step-closure-related-utilities "Permalink to this headline")

### `step_closure`

#### **cerebras.pytorch.**`step_closure`(***closure***)

[\[source\]](../../../_modules/cerebras/pytorch/utils/step_closures.html#step_closure)[#](#cerebras.pytorch.step_closure "Permalink to this definition")

Decorator to automatically wrap a function call in a step closure.

Step closures are queued and all run at the end of each step. This is to ensure that the tensor arguments to the closures are computed and are retrieved before they are used.

Usage:

```
@step_closure
def closure(...):
    ...
...
closure(...)
```

**Parameters:** **closure** (*Callable*) – The function to wrap in a step closure.

\*\*Returns:\*\*The wrapped function.

**Return type:** *Callable*

### `checkpoint_closure`

#### **cerebras.pytorch.**`checkpoint_closure`(***closure***)

[\[source\]](../../../_modules/cerebras/pytorch/utils/step_closures.html#checkpoint_closure)[#](#cerebras.pytorch.checkpoint_closure "Permalink to this definition")

Decorator to wrap function so it is only ever called on checkpoint steps.

With this decorator, the closure may be called at any time. But it will only ever run if on a checkpoint step, as configured by setting checkpoint\_steps when creating a DataExecutor.

An example of a function that would benefit from using this decorator is a function that saves the checkpoint. It ensures that checkpoints are only saved on steps on which the checkpoint is available to be retrieved from the Cerebras wafer-scale cluster.

Example Usage:

```
@checkpoint_closure
def save_checkpoint(...):
    ...
    cstorch.save(...)
...
executor = cstorch.utils.data.DataExecutor(..., checkpoint_steps=100)
for batch in executor:
    ...
    # Always call save checkpoint
    # But save checkpoint only actually runs on checkpoint steps
    save_checkpoint(...)
```

**Parameters:** **closure** (*Callable*) – The function to wrap in a step closure that only runs on checkpoint steps.

\*\*Returns:\*\*The wrapped function.

**Return type:** *Callable*

## Tensorboard utilities[#](#tensorboard-utilities "Permalink to this headline")

**cerebras.pytorch.**`summarize_scalar`**(*\*args*, \_**kwargs\_)\*\*[\[source\]](../../../_modules/cerebras/pytorch.html#summarize_scalar)[#](#cerebras.pytorch.summarize_scalar "Permalink to this definition")

<Info>
  Note

  Scalars summarized using this API are only visible in Tensorboard if a [`SummaryWriter`](#cerebras.pytorch.utils.tensorboard.SummaryWriter "cerebras.pytorch.utils.tensorboard.SummaryWriter") was passed to the [`DataExecutor`](#cerebras.pytorch.utils.data.DataExecutor "cerebras.pytorch.utils.data.DataExecutor") object.

  **cerebras.pytorch.**`summarize_tensor`**(*\*args*, \_**kwargs\_)\*\*[\[source\]](../../../_modules/cerebras/pytorch.html#summarize_tensor)[#](#cerebras.pytorch.summarize_tensor "Permalink to this definition")
</Info>

<Info>
  Note

  Tensors summarized using this API are only visible if a [`SummaryWriter`](#cerebras.pytorch.utils.tensorboard.SummaryWriter "cerebras.pytorch.utils.tensorboard.SummaryWriter") was passed to the [`DataExecutor`](#cerebras.pytorch.utils.data.DataExecutor "cerebras.pytorch.utils.data.DataExecutor") object.
</Info>

***class* cerebras.pytorch.utils.tensorboard.**`SummaryWriter`**(*\*args*, \_**kwargs\_)\*\*[#](#cerebras.pytorch.utils.tensorboard.SummaryWriter "Permalink to this definition")

Thin wrapper around torch.utils.tensorboard.SummaryWriter.

Additional features include the ability to add a tensor summary

**Parameters:**

* **base\_step** (*int*) – The base step to use in `summarize_{scalar,tensor}` functions.

* \***args** – Any other positional and keyword arguments are forwarded directly to the base class

* \*\***kwargs** –

  Any other positional and keyword arguments are forwarded directly to the base class

`add_tensor()`[#](#cerebras.pytorch.utils.tensorboard.SummaryWriter.add_tensor "Permalink to this definition")

***class* cerebras.pytorch.utils.tensorboard.**`SummaryReader`**(*\*args*, \_**kwargs\_)\*\*[#](#cerebras.pytorch.utils.tensorboard.SummaryReader "Permalink to this definition")

Class for reading summaries saved using the SummaryWriter.

**Parameters:**

* **log\_dirs** (*str*) – The directories at which the event files can be found

* **filter** (*Optional*\_\[**str**]\_) – If provided, filter only for scalar/tensor names that belong groups that begin with the filter string. In other words, only names matching filter\*/\* are kept.

* **kwargs** – The remaining keyword arguments are forwarded to the internal EventMultiplexer object.

`reload()`[#](#cerebras.pytorch.utils.tensorboard.SummaryReader.reload "Permalink to this definition")

`read_scalar()`[#](#cerebras.pytorch.utils.tensorboard.SummaryReader.read_scalar "Permalink to this definition")

`Scalars()`[#](#cerebras.pytorch.utils.tensorboard.SummaryReader.Scalars "Permalink to this definition")

`read_tensor()`[#](#cerebras.pytorch.utils.tensorboard.SummaryReader.read_tensor "Permalink to this definition")

`scalar_names()`[#](#cerebras.pytorch.utils.tensorboard.SummaryReader.scalar_names "Permalink to this definition")

`scalar_groups()`[#](#cerebras.pytorch.utils.tensorboard.SummaryReader.scalar_groups "Permalink to this definition")

`tensor_names()`[#](#cerebras.pytorch.utils.tensorboard.SummaryReader.tensor_names "Permalink to this definition")

`text\_summary\_names()`[#](#cerebras.pytorch.utils.tensorboard.SummaryReader.text_summary_names "Permalink to this definition")

`read\_text\_summary()`[#](#cerebras.pytorch.utils.tensorboard.SummaryReader.read_text_summary "Permalink to this definition")

`Tags()`[#](#cerebras.pytorch.utils.tensorboard.SummaryReader.Tags "Permalink to this definition")

## Dataloader benchmark utilities[#](#dataloader-benchmark-utilities "Permalink to this headline")

**cerebras.pytorch.utils.benchmark.**`benchmark_dataloader`**(*input\_fn*, *num\_epochs=None*, *steps\_per\_epoch=None*, *sampling\_frequency=None*, *profile\_activities=None*, *print\_metrics=True*)**[\[source\]](../../../_modules/cerebras/pytorch/utils/benchmark/utils/dataloader.html#benchmark_dataloader)[#](#cerebras.pytorch.utils.benchmark.benchmark_dataloader "Permalink to this definition")

Utility to benchmark a dataloader.

Parameters

* **input\_fn** (*Callable*\_\[**\[**...**]**,\_ *Iterable*\_]\_) – Function that creates and returns a dataloader.

* **num\_epochs** (*Optional*\_\[**int**]\_) – Number of epochs to iterate over the dataloader. If None, the dataloader is only iterated for one epoch.

* **steps\_per\_epoch** (*Optional*\_\[**int**]\_) – Number of steps to iterate over the dataloader in each epoch. If None, the dataloader is iterated in its entirety.

* **sampling\_frequency** (*Optional*\_\[**int**]\_) – Frequency at which to sample metrics. If None, a default value of 100 (i.e. every 100 steps) is used. First step of each epoch is always sampled.

* **profile\_activities** (*Optional*\_\[**List**\[**str**]\_\_]\_) – List of optional activities to profile. If None, no extra activities are profiled. Note that these may incur additional overhead and could affect overall performance of the dataloader, especially if the sampling frequency is high.

* **print\_metrics** (*bool*) – Whether to pretty print the final metrics to console.

**Returns:** Metrics for the dataloader experiment.

**Return type:** [cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics](#cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics "cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics")

***class* cerebras.pytorch.utils.benchmark.utils.dataloader.**`Metrics`**(*dataloader\_build\_time=\<factory>*, *epoch\_metrics=\<factory>*, *batch\_specs=\<factory>*, *total\_time=\<factory>*, *global\_rate=0.0*, *is\_partial=True*, *start\_time\_ns=\<factory>*, *end\_time\_ns=0*)**[\[source\]](../../../_modules/cerebras/pytorch/utils/benchmark/utils/dataloader.html#Metrics)[#](#cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics "Permalink to this definition")

Metrics for a single dataloader experiment.

**Parameters:**

* **dataloader\_build\_time** (*numpy.timedelta64*) – Time to build the dataloader.

* **epoch\_metrics** (*List*\_\[*[*cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics*](#cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics "cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics")*]\_) – List of metrics for each epoch.

* **batch\_specs** (*Dict*\_\[**cerebras.pytorch.utils.benchmark.utils.dataloader.BatchSpec**,\_ *cerebras.pytorch.utils.benchmark.utils.dataloader.BatchSpecOccurence*\_]\_) – Mapping between unique batch specs found and their occurences.

* **total\_time** (*numpy.timedelta64*) – Total time to iterate through all epochs.

* **global\_rate** (*float*) – Overall global rate in steps/second.

* **is\_partial** (*bool*) – Whether the metrics are partial. This can happen if the benchmark is interrupted in the middle of execution.

* **start\_time\_ns** (*int*) – Time at which the experiment started.

* **end\_time\_ns** (*int*) – Time at which the experiment ended.

`dataloader\_build\_time_`**:  numpy.timedelta64**[#](#cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics.dataloader_build_time "Permalink to this definition")

`epoch_metrics`**:  List**\[[cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics](#cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics "cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics")]\_[#](#cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics.epoch_metrics "Permalink to this definition")

`batch_specs_`:  **Dict\[cerebras.pytorch.utils.benchmark.utils.dataloader.BatchSpec,  cerebras.pytorch.utils.benchmark.utils.dataloader.BatchSpecOccurence]**[#](#cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics.batch_specs "Permalink to this definition")

* **total\_time**: `numpy.timedelta64`[#](#cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics.total_time "Permalink to this definition")

* **global\_rate**: `float` *= 0.0* [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics.global_rate "Permalink to this definition")

* **is\_partial**: `bool` *= True* [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics.is_partial "Permalink to this definition")

* **start\_time\_ns**: `int` [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics.start_time_ns "Permalink to this definition")

* **end\_time\_ns**: `int` *= 0* [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics.end_time_ns "Permalink to this definition")

* **total\_steps**: `int` [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics.total_steps "Permalink to this definition")

  Returns the total number of steps across all epochs.

* **global\_sample\_rate**: `Optional[float]` [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.Metrics.global_sample_rate "Permalink to this definition")

Returns the overall global rate in samples/second.

Note that this value only exists if all batches have the exact same structure, dtypes, and shapes. Otherwise, this value is None.

***class* cerebras.pytorch.utils.benchmark.utils.dataloader.**`EpochMetrics`**(*iterator\_creation=\<factory>*, *iteration\_time=\<factory>*, *total\_steps=0*, *batch\_metrics=\<factory>*, *start\_time\_ns=\<factory>*, *end\_time\_ns=0*)**[\[source\]](../../../_modules/cerebras/pytorch/utils/benchmark/utils/dataloader.html#EpochMetrics)[#](#cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics "Permalink to this definition")

Metrics for a single epoch of a dataloader experiment.

**Parameters:**

* **iterator\_creation** (*numpy.timedelta64*) – Time to create the dataloader iterator.

* **iteration\_time** (*numpy.timedelta64*) – Time to iterate the entire epoch excluding the creation of the iterator.

* **total\_steps** (*int*) – Total number of steps in the epoch.

* **batch\_metrics** (*List*\_\[*[*cerebras.pytorch.utils.benchmark.utils.dataloader.BatchMetrics*](#cerebras.pytorch.utils.benchmark.utils.dataloader.BatchMetrics "cerebras.pytorch.utils.benchmark.utils.dataloader.BatchMetrics")*]\_) – List of metrics for batches generated in the epoch.

* **start\_time\_ns** (*int*) – Time at which the epoch started.

* **end\_time\_ns** (*int*) – Time at which the epoch ended.

`iterator_creation`**:  numpy.timedelta64**[#](#cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics.iterator_creation "Permalink to this definition")

* **iteration\_time**: `numpy.timedelta64`[#](#cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics.iteration_time "Permalink to this definition")

* **total\_steps**: `int` *= 0* [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics.total_steps "Permalink to this definition")

* **batch\_metrics**: `List[cerebras.pytorch.utils.benchmark.utils.dataloader.BatchMetrics]` [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics.batch_metrics "Permalink to this definition")

* **start\_time\_ns**: `int` [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics.start_time_ns "Permalink to this definition")

* **end\_time\_ns**: `int` *= 0* [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics.end_time_ns "Permalink to this definition")

* **property\_total\_time**: `numpy.timedelta64` [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.EpochMetrics.total_time "Permalink to this definition")

Returns the total time to create and iterate the epoch.

***class* cerebras.pytorch.utils.benchmark.utils.dataloader.**`BatchMetrics`**(*epoch\_step*, *global\_step*, *local\_rate*, *global\_rate*, *profile\_activities=\<factory>*, *sampling\_time\_ns=\<factory>*)**[\[source\]](../../../_modules/cerebras/pytorch/utils/benchmark/utils/dataloader.html#BatchMetrics)[#](#cerebras.pytorch.utils.benchmark.utils.dataloader.BatchMetrics "Permalink to this definition")

Metrics for a single batch of a dataloader experiment.

**Parameters:**

* **epoch\_step** (*int*) – Epoch step at which the batch was generated.

* **global\_step** (*int*) – Global step at which the batch was generated.

* **local\_rate** (*float*) – Local rate (in steps/second) at the sampling frequency. This is the instantaneous rate (relative to previous batch) at which the batch was generated.

* **global\_rate** (*float*) – global rate (in steps/second) at the sampling frequency. This is the global rate since the start of the iterating epochs.

* **profile\_activities** (*Dict*\_\[**str**,\_ *Any*\_]\_) – Dictionary of profile activities and their values.

* **sampling\_time\_ns** (*int*) – Time at which the batch was sampled.

**epoch\_step**:  *`int`*[#](#cerebras.pytorch.utils.benchmark.utils.dataloader.BatchMetrics.epoch_step "Permalink to this definition")

* **global\_step**: *`int`* [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.BatchMetrics.global_step "Permalink to this definition")

* **local\_rate**: *`float`* [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.BatchMetrics.local_rate "Permalink to this definition")

* **global\_rate**: *`float`* [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.BatchMetrics.global_rate "Permalink to this definition")

* **profile\_activities**: *`Dict[str, Any]`* [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.BatchMetrics.profile_activities "Permalink to this definition")

* **sampling\_time\_ns**: *`int`* [#](#cerebras.pytorch.utils.benchmark.utils.dataloader.BatchMetrics.sampling_time_ns "Permalink to this definition")