> ## Documentation Index
> Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Gemma 2

> Decoder-only language models by Google DeepMind, using interleaved attention and GQA for high-quality performance at practical scale.

## Model Description

Gemma 2 is a family of decoder-only transformer models developed by Google DeepMind, ranging from 2B to 27B parameters. Architecturally, Gemma 2 builds upon the Transformer backbone with several enhancements: it interleaves local sliding window and global attention layers, adopts grouped-query attention (GQA), and uses GeGLU activations with RMSNorm. The models support a context length of 8K and utilize a 256K-token multilingual tokenizer inherited from Gemini.

Gemma 2 models are well-suited for tasks involving instruction following, long-context understanding, multilingual reasoning, and coding.

## Code Structure

The code for this model is located in the [`/gemma2`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gemma2) directory within ModelZoo. Here’s how it’s organized:

* [`/configs`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gemma2/configs): Contains YAML configuration files.
* [`model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gemma2/model.py): The implementation of the Gemma 2 model.

<Note>
  Our implementation of Gemma is built on top of our GPT-2 backbone. For more details, see [`gpt2_model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt2/gpt2_model.py).
</Note>

## Available Configurations

| Configuration                                                                                                                                                                               | Description                                                                    |
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------- |
| [`params_gemma2_9b_msl8k.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gemma2/configs/params_gemma2_9b_msl8k.yaml)                                 | 9B parameter Gemma 2 model with 8K MSL.                                        |
| [`params_gemma2_9b_msl8k_swa_4k_sink_512.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gemma2/configs/params_gemma2_9b_msl8k_swa_4k_sink_512.yaml) | Variant of the 9B model using 4K sliding window attention and 512 sink tokens. |
| [`params_gemma2_27b_msl8k.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gemma2/configs/params_gemma2_27b_msl8k.yaml)                               | 27B parameter Gemma 2 model with 8K MSL.                                       |

## Workflow

For example workflows using language models from the Cerebras Model Zoo, see our tutorials on [pretraining](../../../getting-started/pre-train-your-first-model) and [fine-tuning](../../../getting-started/fine-tune-your-first-model).

For a complete list of Cerebras ModelZoo CLI commands, see the [command reference](../../cli-overview).

## References

* Gemma Team. (2024). [Gemma 2: Improving Open Language Models at a Practical Size](https://arxiv.org/abs/2408.00118)
* Ainslie, J. et al. (2023). [GQA: Generalized Multi-Query Attention](https://arxiv.org/abs/2305.13245)
* Hinton, G. et al. (2015). [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531)