> ## Documentation Index
> Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Jais

> Decoder-only language models optimized for Arabic and English, developed by Inception, MBZUAI, and Cerebras.

## Model Description

Jais is a family of decoder-only transformer models developed collaboratively by Inception, MBZUAI, and Cerebras. These models are trained from scratch on 395B tokens of high-quality Arabic, English, and code data, with an emphasis on cross-lingual and code-switching capabilities.

Architecturally, Jais models follow the standard transformer decoder design and adopt several enhancements: they use rotary positional embeddings (RoPE), grouped-query attention (GQA), SwiGLU activations, and RMSNorm. The tokenizer is a 51.2K vocabulary SentencePiece model trained on the full multilingual corpus.

The Jais family includes both base and instruction-tuned variants and is particularly well-suited for tasks involving multilingual reasoning, dialogue, and programming.

## Code Structure

The code for this model is located in the [`/jais`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/jais) directory within ModelZoo. Here’s how it’s organized:

* [`/configs`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/jais/configs): Contains YAML configuration files.
* [`model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/jais/model.py): The implementation of the Jais model.

<Note>
  Our implementation of Jais is built on top of our GPT-2 backbone. For more details, see [`gpt2_model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt2/gpt2_model.py).
</Note>

## Available Configurations

| Configuration                                                                                                                                             | Description                               |
| :-------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------------- |
| [`params_jais_13b.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/jais/configs/params_jais_13b.yaml)               | 13B parameter base Jais model.            |
| [`params_jais_13b_chat.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/jais/configs/params_jais_13b_chat.yaml)     | Instruction-tuned variant of 13B.         |
| [`params_jais_30b.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/jais/configs/params_jais_30b.yaml)               | 30B parameter base Jais model.            |
| [`params_jais_30b_chat.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/jais/configs/params_jais_30b_chat.yaml)     | Instruction-tuned variant of 30B.         |
| [`params_jais_30b_phase1.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/jais/configs/params_jais_30b_phase1.yaml) | 30B checkpoint after Phase 1 pretraining. |

## Workflow

For example workflows using language models from the Cerebras Model Zoo, see our tutorials on [pretraining](../../../getting-started/pre-train-your-first-model) and [fine-tuning](../../../getting-started/fine-tune-your-first-model).

For a complete list of Cerebras ModelZoo CLI commands, see the [command reference](../../cli-overview).

## References

* Almazrouei, E., et al. (2023). [Jais: An Open Arabic-Centric Foundation Model](https://arxiv.org/abs/2308.16149)