> ## Documentation Index
> Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Model Zoo CLI Overview

> Learn how to use the ModelZoo CLI.

## Overview

The ModelZoo CLI is a comprehensive command-line interface that serves as a single entry point for all ModelZoo-related tasks. This tool streamlines various machine learning workflows, from data preprocessing to model training and validation.

## Commands

Below is a list of commands that can be used with the ModelZoo CLI tool. Expand each section to see examples and more information.

<AccordionGroup>
  <Accordion title="fit" description="Trains a model using specified configuration" defaultOpen={false}>
    Example: `cszoo fit params_model.yaml`
  </Accordion>

  <Accordion title="validate" description="Validates a model using specified configuration" defaultOpen={false}>
    Example: `cszoo validate params_model.yaml`
  </Accordion>

  <Accordion title="validate_all" description="Runs upstream and downstream validation" defaultOpen={false}>
    Example: `cszoo validate_all params_model.yaml`
  </Accordion>

  <Accordion title="checkpoint" description="Checkpoint converter" defaultOpen={false}>
    <AccordionGroup>
      <Accordion title="convert" description="Converts a checkpoint between Huggingface and Cerebras formats or between different Cerebras formats." defaultOpen={false}>
        Example: `cszoo checkpoint convert --model gpt2 --src-fmt cs-auto --tgt-fmt hf --config workdir/params_gpt_tiny.yaml model_dir/checkpoint.mdl`
      </Accordion>

      <Accordion title="convert-config" description="Converts a checkpoint between Huggingface and Cerebras formats or between different Cerebras formats." defaultOpen={false}>
        Example: `cszoo checkpoint convert-config --model gpt2 --src-fmt cs-auto --tgt-fmt hf workdir/params_gpt_tiny.yaml`
      </Accordion>

      <Accordion title="list" description="Lists all available checkpoint converters. Can also list all checkpoint converters for a specified model." defaultOpen={false}>
        Example: `cszoo checkpoint list`
      </Accordion>

      <Accordion title="diff" description="Compares two checkpoints to identify differences." defaultOpen={false}>
        Example: `cszoo checkpoint diff checkpoint_a.mdl checkpoint_b.mdl`
      </Accordion>
    </AccordionGroup>
  </Accordion>

  <Accordion title="model" description="Query information on ModelZoo models" defaultOpen={false}>
    <AccordionGroup>
      <Accordion title="list" description="Displays all supported models." defaultOpen={false}>
        Example: `cszoo model list`
      </Accordion>

      <Accordion title="info" description="Shows detailed model information." defaultOpen={false}>
        Example: `cszoo model info gpt2`
      </Accordion>

      <Accordion title="describe" description="Displays model configuration parameters." defaultOpen={false}>
        Example: `cszoo model describe gpt2`
      </Accordion>

      <Accordion title="init_checkpoint" description="Creates initial model checkpoint." defaultOpen={false}>
        Example: `cszoo model init_checkpoint <model_name>`
      </Accordion>
    </AccordionGroup>
  </Accordion>

  <Accordion title="data_preprocess" description="Preprocess data" defaultOpen={false}>
    <AccordionGroup>
      <Accordion title="list" description="Shows available preprocessing configurations." defaultOpen={false}>
        Example: `cszoo data_preprocess list`
      </Accordion>

      <Accordion title="pull" description="Copies a  model configuration file to a local directory." defaultOpen={false}>
        Example: `cszoo data_preprocess pull summarization_preprocessing -o workdir`
      </Accordion>

      <Accordion title="run" description="Executes preprocessing using specified configuration." defaultOpen={false}>
        Example: `cszoo data_preprocess run --config preprocessing.yaml`
      </Accordion>
    </AccordionGroup>
  </Accordion>

  <Accordion title="data_processor" description="Query information on ModelZoo data processors" defaultOpen={false}>
    <AccordionGroup>
      <Accordion title="list" description="Shows available data processors." defaultOpen={false}>
        Example: `cszoo data_processor list`
      </Accordion>

      <Accordion title="info" description="Displays data processor information." defaultOpen={false}>
        Example: `cszoo data_processor info GptHDF5DataProcessor`
      </Accordion>

      <Accordion title="describe" description="Shows processor configuration parameters." defaultOpen={false}>
        Example: `cszoo data_processor describe GptHDF5DataProcessor`
      </Accordion>

      <Accordion title="benchmark" description="Benchmarks a specified dataloader." defaultOpen={false}>
        Example: `cszoo data_processor benchmark params.yaml`
      </Accordion>
    </AccordionGroup>
  </Accordion>

  <Accordion title="config" description="Save and manage model config files" defaultOpen={false}>
    <AccordionGroup>
      <Accordion title="pull" description="Copies a  model config file to a local directory." defaultOpen={false}>
        Example: `cszoo config pull gpt2_tiny -o workdir`
      </Accordion>

      <Accordion title="validate" description="Validates a specified config file." defaultOpen={false}>
        Example: `cszoo config validate params.yaml`
      </Accordion>

      <Accordion title="convert_legacy" description="Upgrades V1 config files to V2 YAML." defaultOpen={false}>
        Example: `cszoo config convert_legacy old_config.yaml`
      </Accordion>

      <Accordion title="stats" description="Retrieves relevant statistics for a model using the specified configuration file." defaultOpen={false}>
        Example: `cszoo config stats params.yaml`
      </Accordion>
    </AccordionGroup>
  </Accordion>
</AccordionGroup>

## Example Workflow: Pretraining a model using the ModelZoo CLI

This workflow guides you through the steps to pretrain a model using the Cerebras ModelZoo CLI. Follow these steps to set up your environment, preprocess data, and run the pretraining process.

<Tip>
  **Prerequisite:** Before proceeding with the steps below, ensure that you have completed the setup and installation guide found [here](../getting-started/setup-and-installation).
</Tip>

<Steps>
  <Step title="Create model directory">
    Create a directory to store all the files for this pretraining workflow and copy the necessary configuration files.

    ```
    mkdir pretraining_tutorial
    cp modelzoo/src/cerebras/modelzoo/tutorials/pretraining/* pretraining_tutorial
    ```
  </Step>

  <Step title="Preprocess the data">
    Preprocess the training and validation datasets using the provided configuration files.

    ```bash theme={null}
    cszoo data_preprocess run --config pretraining_tutorial/train_data_config.yaml
    cszoo data_preprocess run --config pretraining_tutorial/valid_data_config.yaml
    ```
  </Step>

  <Step title="Run model">
    Run the pretraining process using the provided configuration.

    ```
    cszoo fit pretraining_tutorial/model_config.yaml
    ```
  </Step>

  <Step title="Convert checkpoint to HuggingFace">
    Convert the trained model checkpoint into a HuggingFace-compatible format.

    ```
    cszoo checkpoint convert \
      --model llama \
      --src-fmt cs-auto \
      --tgt-fmt hf \
      --config pretraining_tutorial/model_config.yaml \
      --output-dir pretraining_tutorial/to_hf \
      pretraining_tutorial/model/checkpoint_0.mdl
    ```
  </Step>
</Steps>

## Getting Help

For detailed information about any command, use the `--help` flag:

```
cszoo --help
cszoo <command> --help
```

### CSZoo Assistant

Need help? Our CSZoo Assistant is an LLM agent you can access from the command line with the `assistant` subcommand.

Use it to:

* Ask questions:
  `cszoo assistant "what is the checkpoint converter?"`

* Perform actions:
  `cszoo assistant "convert my checkpoint from huggingface to cerebras"`

CSZoo Assistant will always ask your permission before running a command.

<Note>
  Access to the Cerebras Inference API is required and you'll need to provide your API key with the following command:

  `export CEREBRAS_API_KEY=<your api key>`

  Don't have an API key? Follow [these instructions](https://inference-docs.cerebras.ai/quickstart#step-1-set-up-your-api-key).
</Note>

<Warning>
  CSZoo Assistant is a beta feature and it may make mistakes. Always double-check its reasoning and be aware of the following limitations:

  * CSZoo Assistant can currently only access the help manuals found with `cszoo ... -h`.

  * There are currently no advanced context length management mechanisms in place. The assistant will error out if it overflows the context length.
</Warning>