> ## Documentation Index
> Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLaMA

> Series of decoder-only transfomer LLMs from Meta.

## Model Description

The LLaMA family is a series of decoder-only transformer models designed for efficient, high-performance language modeling. Architecturally similar to GPT-2, the original LLaMA model uses RMSNorm instead of LayerNorm, SwiGLU activations, and rotary positional embeddings. LLaMA-2 improves on this with a larger training corpus, doubled context length, and grouped-query attention in its largest model. Code LLaMA specializes in programming tasks through continued pretraining on code-heavy data. LLaMA-3 introduces a more efficient 128K-token tokenizer, expands context to 8K tokens, and adopts grouped-query attention across all sizes. These models excel at text generation, summarization, reasoning, coding, and instruction following.

## Code Structure

The code for this model is located in the [`/llama`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama) directory within ModelZoo. Here’s how it’s organized:

* [`/configs`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs): Contains YAML configuration files.
* [`model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/llama/model.py): The implementation of the the LLaMA model.

<Note>
  Our implementation of LLaMA is built on top of our GPT-2 implementation. For more details, see [`gpt2_model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt2/gpt2_model.py).
</Note>

## Available Configurations

<Accordion title="LLaMa 3">
  <Tabs>
    <Tab title="LLaMa 3.1">
      | Configuration                                                                                                                                                                                      | Description                                                                                                                                                |
      | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
      | [`params_llama3p1_70b_msl_128k.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_llama3p1_70b_msl_128k.yaml)                             | A 70B parameter model with a maximum sequence length of 128K, configured as described in the LLaMa 3.1 blog.                                               |
      | [`params_llama3p1_70b_msl_8k.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_llama3p1_70b_msl_8k.yaml)                                 | A 70B parameter model with a maximum sequence length of 8K, configured as described in the LLaMa 3.1 blog.                                                 |
      | [`params_llama3p1_8b_msl_128k.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_llama3p1_8b_msl_128k.yaml)                               | A 8B parameter model with a maximum sequence length of 128K, configured as described in the LLaMa 3.1 blog.                                                |
      | [`params_llama3p1_8b_msl_32k_swa_8k_sink_512.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_llama3p1_8b_msl_32k_swa_8k_sink_512.yaml) | A 8B parameter model with a maximum sequence length of 32K, SWA starting at 8K, and sink tokens set to 512. Configured as described in the LLaMa 3.1 blog. |
      | [`params_llama3p1_8b_msl_8k.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_llama3p1_8b_msl_8k.yaml)                                   | A 8B parameter model with a maximum sequence length of 8K, configured as described in the LLaMa 3.1 blog.                                                  |
    </Tab>

    <Tab title="LLaMa 3.2">
      | Configuration                                                                                                                                                    | Description                                                                                               |
      | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
      | [`params_llama3p2_1b_msl_8k.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_llama3p2_1b_msl_8k.yaml) | A 1B parameter model with a maximum sequence length of 8K, configured as described in the LLaMa 3.2 blog. |
      | [`params_llama3p2_3b_msl_8k.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_llama3p2_3b_msl_8k.yaml) | A 3B parameter model with a maximum sequence length of 8K, configured as described in the LLaMa 3.2 blog. |
    </Tab>

    <Tab title="LLaMa 3.3">
      | Configuration                                                                                                                                                          | Description                                                                                                                                                                                              |
      | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
      | [`params_llama3p3_70b_msl_128k.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_llama3p3_70b_msl_128k.yaml) | A 70B parameter model with a maximum sequence length of 128K, configured as described in the [LLaMa 3.3 model card.](https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/MODEL_CARD.md) |
    </Tab>
  </Tabs>
</Accordion>

<Accordion title="LLaMa-2">
  | Configuration                                                                                                                                    | Description                                                                                             |
  | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------- |
  | [`params_llama2_7b.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_llama2_7b.yaml)   | A 7B parameter model configured as described in the [LLaMa-2 paper](https://arxiv.org/abs/2307.09288).  |
  | [`params_llama2_13b.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_llama2_13b.yaml) | A 13B parameter model configured as described in the [LLaMa-2 paper](https://arxiv.org/abs/2307.09288). |
  | [`params_llama2_70b.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_llama2_70b.yaml) | A 70B parameter model configured as described in the [LLaMa-2 paper](https://arxiv.org/abs/2307.09288). |
</Accordion>

<Accordion title="Code LLaMa">
  | Configuration                                                                                                                                            | Description                                                                                                                                                     |
  | -------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  | [`params_code_llama_7b.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_code_llama_7b.yaml)   | A 7B parameter model configured as described in the [Code LLaMa paper](https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/).  |
  | [`params_code_llama_70b.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_code_llama_70b.yaml) | A 70B parameter model configured as described in the [Code LLaMa paper](https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/). |
</Accordion>

<Accordion title="WizardLM">
  | Configuration                                                                                                                                        | Description                                                                                              |
  | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
  | [`params_wizardlm_13b.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/llama/configs/params_wizardlm_13b.yaml) | A 13B parameter model configured as described in the [WizardLM](https://arxiv.org/pdf/2304.12244) paper. |
</Accordion>

All configs are meant to be run on Weight Streaming mode using Appliance mode and Kubernetes flow.

## Workflow

For example workflows using language models from the Cerebras Model Zoo, see our tutorials on [pretraining](../../../getting-started/pre-train-your-first-model) and [fine-tuning](../../../getting-started/fine-tune-your-first-model).

For a complete list of Cerebras ModelZoo CLI commands, see the [command reference](../../cli-overview).

## References

* Radford, A. et al. (2019). [Language Models are Unsupervised Multitask Learners](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf).
* Touvron, Hugo, et al. (2023). [Llama: Open and efficient foundation language models](https://arxiv.org/pdf/2302.13971)
* Touvron, Hugo, et al. (2023). [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288)
* Rozière, Baptiste, et al. (2023). [Code Llama: Open Foundation Models for Code](https://arxiv.org/pdf/2308.12950)
* Meta AI (2024). [Build the future of AI with Meta Llama 3](https://llama.meta.com/llama3)