> ## Documentation Index
> Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# SantaCoder

> Decoder-only language model for code generation by BigCode, trained on Java, JavaScript, and Python with support for fill-in-the-middle and multi-query attention.

## Model Description

SantaCoder is a 1.1B parameter decoder-only transformer model developed by the BigCode community. It is trained on Java, JavaScript, and Python code from The Stack v1.1.

Architecturally, SantaCoder uses Multi-Query Attention (MQA) for efficient inference and supports Fill-in-the-Middle (FIM) generation, allowing the model to complete and infill code based on context. It employs a 49K BPE tokenizer trained on raw bytes and achieves strong performance on MultiPL-E and HumanEval benchmarks despite being significantly smaller than competing models.

SantaCoder is well-suited for code completion, interactive development tools, and multilingual code generation.

## Code Structure

The code for this model is located in the [`/santacoder`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/santacoder) directory within ModelZoo. Here’s how it’s organized:

* [`/configs`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/santacoder/configs): Contains YAML configuration files.
* [`model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/santacoder/model.py): The implementation of the SantaCoder model.

<Note>
  Our implementation of SantaCoder is built on top of our GPT-2 backbone. For more details, see [`gpt2_model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt2/gpt2_model.py).
</Note>

## Available Configurations

| Configuration                                                                                                                                               | Description                      |
| :---------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------- |
| [`params_santacoder_1b.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/santacoder/configs/params_santacoder_1b.yaml) | 1.1B parameter SantaCoder model. |

## Workflow

For example workflows using language models from the Cerebras Model Zoo, see our tutorials on [pretraining](../../../getting-started/pre-train-your-first-model) and [fine-tuning](../../../getting-started/fine-tune-your-first-model).

For a complete list of Cerebras ModelZoo CLI commands, see the [command reference](../../cli-overview).

## References

* Ben Allal, L., et al. (2023). [SantaCoder: Don’t Reach for the Stars!](https://arxiv.org/abs/2301.03988)
* Bavarian, M., et al. (2022). [Efficient Training of Language Models to Fill in the Middle](https://arxiv.org/abs/2207.14255)
* Shazeer, N. (2019). [Fast Transformer Decoding: One Write-Head is All You Need](https://arxiv.org/abs/1911.02150)