> ## Documentation Index
> Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# BLOOM

> Multilingual decoder-only language model with ALiBi positional embeddings, designed to generalize across 46 natural and 13 programming languages.

## Model Description

BLOOM is a decoder-only Transformer-based language model developed by the BigScience project. It supports multilingual training across 46 natural languages and 13 programming languages, with models ranging in size up to 176B parameters.

Architecturally, BLOOM resembles GPT-2 but introduces two important differences:

* **Tokenizer**: BLOOM uses a tokenizer and vocabulary specifically designed for multilingual generalization, consisting of \~250K tokens.
* **Position Embeddings**: Instead of learnable absolute position embeddings (as in GPT-2), BLOOM uses [ALiBi](https://arxiv.org/pdf/2108.12409.pdf) — Attention with Linear Biases — which allows extrapolation to longer sequence lengths and introduces a recency bias in attention computation.

## Code Structure

The code for this model is implemented under the `bloom` directory.

* [`configs/`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/bloom/configs): Contains YAML configuration files for various BLOOM variants.
* [`model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt2/model.py): Shared model code reused from GPT-2, modified to support ALiBi embeddings.

<Note>
  Our implementation of BLOOM is built on top of our GPT-2 backbone. For more details, see [`gpt2_model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt2/gpt2_model.py).
</Note>

## Available Configurations

| Configuration                                                                                                                                | Description                                                                                                         |
| -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| [`params_bloom_7b.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/bloom/configs/params_bloom_7b.yaml) | BLOOM-7B model with `hidden_size=4096`, `num_hidden_layers=30`, and `num_heads=32`. Uses ALiBi position embeddings. |

## Workflow

For example workflows using language models from the Cerebras Model Zoo, see our tutorials on [pretraining](../../getting-started/pre-train-your-first-model) and [fine-tuning](../../getting-started/fine-tune-your-first-model).

For a complete list of Cerebras ModelZoo CLI commands, see the [command reference](../../model-zoo/core-workflows/cli-reference).

## Implementation Notes

The core model logic reuses components from the GPT-2 implementation, but the presence of ALiBi position embeddings enables better extrapolation to longer sequence lengths during inference. To enable this in your config file:

* Set `model.position_embedding_type: alibi`
* Optionally, set `model.alibi_trainable_slopes: false` (recommended, based on the ALiBi paper’s findings)

## References

* [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://arxiv.org/abs/2211.05100)
* [TRAIN SHORT, TEST LONG: Attention with Linear Biases Enables Input Length Extrapolation](https://arxiv.org/pdf/2108.12409.pdf)
* [Language Models are Unsupervised Multitask Learners (GPT-2)](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf)