> ## Documentation Index
> Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# StarCoder

> Decoder-only language models for code generation by BigCode, trained on permissively licensed code with support for fill-in-the-middle and multi-query attention.

## Model Description

StarCoder is a family of decoder-only transformer models developed by the BigCode initiative, optimized for high-quality code generation. The flagship model, StarCoder (15.5B), was trained on 1 trillion tokens, with an emphasis on multilingual support and strong performance in Python.

Architecturally, StarCoder builds on the transformer decoder backbone with several enhancements: it uses multi-query attention (MQA) for fast inference, supports fill-in-the-middle (FIM) generation, and extends context lengths to 8K tokens. Variants of StarCoder have been fine-tuned for specific domains such as SQL, OctoPack, and WizardCoder-style instruction following.

These models are well-suited for tasks such as code completion, documentation generation, and interactive programming support.

## Code Structure

The code for this model is located in the [`/starcoder`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/starcoder) directory within ModelZoo. Here’s how it’s organized:

* [`/configs`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/starcoder/configs): Contains YAML configuration files.
* [`model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/starcoder/model.py): The implementation of the StarCoder model.

<Note>
  Our implementation of StarCoder is built on top of our GPT-2 backbone. For more details, see [`gpt2_model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt2/gpt2_model.py).
</Note>

## Available Configurations

| Configuration                                                                                                                                                  | Description                                            |
| :------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------- |
| [`params_starcoder_15b.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/starcoder/configs/params_starcoder_15b.yaml)     | Base StarCoder model with 15.5B parameters.            |
| [`params_octocoder_15b.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/starcoder/configs/params_octocoder_15b.yaml)     | StarCoder variant fine-tuned with OctoPack-style data. |
| [`params_sqlcoder_15b.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/starcoder/configs/params_sqlcoder_15b.yaml)       | StarCoder variant specialized for SQL generation.      |
| [`params_wizardcoder_15b.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/starcoder/configs/params_wizardcoder_15b.yaml) | StarCoder variant tuned for instruction-style prompts. |

## Workflow

For example workflows using language models from the Cerebras Model Zoo, see our tutorials on [pretraining](../../../getting-started/pre-train-your-first-model) and [fine-tuning](../../../getting-started/fine-tune-your-first-model).

For a complete list of Cerebras ModelZoo CLI commands, see the [command reference](../../cli-overview).

## References

* Li, R., et al. (2023). [StarCoder: May the Source Be With You!](https://arxiv.org/abs/2305.06161)
* Ainslie, J. et al. (2023). [GQA: Generalized Multi-Query Attention](https://arxiv.org/abs/2305.13245)