Skip to main content
EleutherAI’s Evaluation Harness (EEH) is a popular framework for evaluating large language models across various different datasets and tasks. You can configure EEH as part of your training workflow. See Pretraining with Downstream Validation. The examples in this guide will perform downstream validation on LLaMA3 8B.

​
Prerequisites

Please ensure that you have installed the Cerebras Model Zoo package by going through the installation guide. Note that EEH version tested and packaged in the Cerebras Model Zoo is the official release v0.4.7. Please also read through the Trainer Overview and Trainer Configuration Overview, as these guides will help understand how to configure running EEH standalone.

​
Configure the Run

This section covers the required steps for setting up an EEH run to perform standalone downstream validation on various tasks. In particular, you will need to write a YAML configuration file to configure an instance of the Trainer callback. The example in this section configures evaluation for LLaMA3 8B via the multiple choice (non-generative) eval harness task winogrande using a single CSX. If you aren’t interested in seeing the break down of the configuration, feel free to skip ahead to the Putting it All Together section to see the full YAML configuration.

​
Configure the CSX Backend

The first step is to specify the CSX backend and resources required for the run. Create a YAML configuration file with the following cluster config:
This example uses a single CSX, but you can readily update num_csx to run EEH on multiple CSXs for improved performance.

​
Configure the Model

Next, please add the following model configuration in the YAML for LLaMA3 8B with 8K context length:
To run downstream validation harness, you must specify the name setting in the model configuration. Valid names corresponding to the supported models include:
  • btlm
  • bloom
  • gpt2
  • gptj
  • falcon
  • gpt3
  • gpt-neox
  • llama
  • mistral
  • mpt
  • jais
  • santacoder
  • starcoder

​
Configure the EEH Callback

EEH is implemented as an extension to the Trainer callback. Add the following section in the YAML to set up the EleutherEvalHarness callback:
The eeh_args section exposes the following settings to configure the EEH run: You can either specify the settings here or pass them via CLI arguments to the standalone EEH run script. The callback configuration also accepts dataloader settings that you must specify in the YAML to set up input data preprocessing for the run: Additionally, you may optionally specify the following, CSX-specific eval harness setting:
  • keep_data_dir: Use this to preserve the preprocessed eval harness task data samples, i.e. the directory specified under data_dir. Defaults to False, i.e. data samples are deleted after the run.

​
(Optional) Configure HuggingFace (HF) Cache Directory

EEH utilizes HF’s APIs to download task data and other configurations. This data is by default cached under $HOME/.cache/huggingface. However, you may choose to specify a different directory for this cached data via the HFCacheDir callback:

​
Putting it All Together

Here’s what the full YAML configuration looks like once you follow this guide for configuring the individual pieces:

​
Running EEH on CS-X

Now that the the YAML configuration is complete, use the Model Zoo CLI to run EEH on various tasks. This script accepts the following arguments:
lm_eval
  1. We support a subset of Eleuther’s command line interface (CLI) arguments above. For a more detailed descrition of these supported arguments, see the EEH documentation.
  2. You may also specify these arguments in the YAML under the eeh_args key of the EleutherEvalHarness configuration, but please note that the CLI setting will override the settings in the YAML.
  3. The params CLI argument is required. Use it to specify the path to the YAML configuration file.
  4. Use the --checkpoint_path CLI argument to specify the path to the checkpoint file to load model weights from. If a checkpoint path is not provided, we support checkpoint autoloading in this flow such that the latest checkpoint file will be picked up from the specified model_dir.

​
Supported Tasks

  • We support lm_eval@v0.4.7 .
  • You may perform downstream validation on all EEH tasks with output_type: loglikelihood or output_type: multiple_choice in the task specification. See asdiv and arc_easy for respective examples. You may specify each of these types of tasks separately or together in a single EleutherEvalHarness callback.
  • We currently do not support eval harness tasks with output_type: loglikelihood_rolling.

​
Adding New Tasks

Please refer to Eleuther’s new task implementation guide here to add new tasks.

​
Limitations

  • We currently do not support running multiple generative eval harness tasks in the same callback.
  • EEH task groups, such as agieval, comprise multiple generative sub tasks that you will have to configure in the YAML via separate callbacks.
  • Please turn on grad accumulation and choose a small micro batch size (between 16 to 32) under the flags configuration of the EleutherEvalHarness callback of the YAML,

​
Examples

Let’s assume that the YAML configuration file above is written to ./llama3_8B_eeh.yaml. Then, to run evaluation for task winogrande, please set up a bash script as follows:
The output logs are as follows:
To run evaluation on more non-generative tasks, you may update the run script to the following:
The output logs are as follows:
The EEH flow also supports running generative (autoregressive) eval harness tasks, i.e. tasks that specify output_type: generate_until in the task specification, such as triviaqa or drop. Refer to task triviqa for an example specification from the official EEH repository.In order to run generative inference on CSX, you must specify the following inference settings in the model config of YAML file:
  • start_token - ID of the special token that indicates where to start inferring for each sample, as described above. You may specify a list of token IDs instead of a single ID. If you do, the model will start inference at the first token that matches any one of the provided IDs. The model will pad inferred predictions with the first ID in the list.
  • stop_sequences - List of sequences (each one being a list of token IDs). If any one of these sequences is emitted by the model, inference will stop for that sample. For example, suppose you would like to stop inferring after either a newline character (e.g. token id 1), or a combination of a period (e.g. token id 2) followed by a space (e.g. token id 3). In this case, set stop_sequences to [[1], [2, 3]]. To stop inferring after seeing a newline character only, set stop_sequences to [[1]]. To disable this feature, set stop_sequences to an empty list []. Additionally, the following optional parameters may be set:
  • max_tokens - Maximum tokens to infer for each sample.
  • loop_dim - Indicates the sequence dimension in the input and output data. Default value is 1. If set to 0, indicates that both input and output data is transposed (i.e. sequence X samples instead of samples X sequence).
For your LLaMA3 8B example, please update the YAML file ./llama3_8B_eeh.yaml to add these inference settings:
  1. For start_token, it is ideal to choose a value that’s not going to be generated by the model, i.e. vocab_size in the example above.
  2. The generative task itself defines stop_sequences under setting generation_kwargs.until of the task spec. For instance, triviqa specifies "\n", "." and "," as the stop tokens. The EEH flow will internally override the stop_sequences config with the value from the task, so you can also specify an arbitrary value in the YAML.
Finally, please update the bash script as follows:
The output logs are as follows:
By default, the model will perform greedy sampling of the inferred tokens, i.e. for all of the model’s outputs, pick the token with the highest probability.In order to perform non-greedy sampling, you can pass in temperature, top_k or top_p to either the bash script or under eeh_args of the YAML. For example:
You can combine evaluation for generative and non-generative eval harness tasks simply via updating the bash script as follows:
The output logs are as follows:
  1. Since inference settings are baked into the model configuration and that inference requires different resources, there is a separate compile and execution for running downstream validation on generative tasks.
  2. You may specify at most one generative task at a time (per callback).
In order to run multiple generative tasks, you may update the run script to the following:

​
What’s next?

To run downstream validation on code generation tasks, please see check out: You can also perform downstream validation using EEH as part of your pretraining runs with upstream validation. Check out the following guide: