> ## Documentation Index > Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt > Use this file to discover all available pages before exploring further. # GPT-3 > A decoder-only transformer language model, scaled to billions of parameters, trained on autoregressive next-token prediction with support for µP scaling and Cerebras-optimized workflows. ## Model Description GPT-3 is a decoder-only transformer language model architecture designed for large-scale autoregressive pretraining. It extends GPT-2 with significantly more parameters (ranging from 1.3B to 175B) and introduces architectural refinements such as sparse attention layers, used in alternating blocks to reduce compute costs during training. However, this implementation uses the GPT-2-style dense attention in all layers. Training occurs on next-token prediction using large text corpora like The PILE, with inputs represented as token sequences padded and masked to a fixed maximum sequence length. ## Code Structure The code for this model is located in the `gpt3` directory within ModelZoo. Here's how it's organized: * [`configs/`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gpt3/configs): Contains YAML configuration files for various GPT-3-sized models. * [`run.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/run.py): Training and evaluation entry point. Accepts CLI arguments for mode, config path, checkpointing, and output directories. Our implementation of GPT-3 is built on top of our GPT-2 backbone. For more details, see [`gpt2_model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt2/gpt2_model.py). ## Available Configurations | Configuration | Description | | ---------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | [`111m.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT/111m.yaml) | 111M parameter model using standard parametrization. | | [`111m_mup.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT/111m_mup.yaml) | 111M parameter model with Maximal Update Parametrization (µP). | | [`256m.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT/256m.yaml) | 256M parameter model using standard parametrization. | | [`256m_mup.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT/256m_mup.yaml) | 256M parameter model with µP. | | [`590m.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT/590m.yaml) | 590M parameter model using standard parametrization. | | [`590m_mup.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT/590m_mup.yaml) | 590M parameter model with µP. | | [`1p3b.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT/1p3b.yaml) | 1.3B parameter model (GPT-3 XL equivalent). | | [`1p3b_mup.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT/1p3b_mup.yaml) | 1.3B parameter model with µP. | | [`2p7b.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT/2p7b.yaml) | 2.7B parameter model. | | [`2p7b_mup.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT/2p7b_mup.yaml) | 2.7B parameter model with µP. | | [`6p7b.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT/6p7b.yaml) | 6.7B parameter model. | | [`13b_bs720.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT/13b_bs720.yaml) | 13B parameter model, batch size 720. | | [`13b_bs1080.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT/13b_bs1080.yaml) | 13B parameter model, batch size 1080. | | Configuration | Description | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------ | | [`params_gpt3_125m_rigl75.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/sparsity/pretraining/params_gpt3_125m_rigl75.yaml) | 125M parameter model with 75% sparsity using RigL pruning. | | [`params_gpt3_125m_set75.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/sparsity/pretraining/params_gpt3_125m_set75.yaml) | 125M parameter model with 75% sparsity using SET pruning. | | [`params_gpt3_125m_static75.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/sparsity/pretraining/params_gpt3_125m_static75.yaml) | 125M parameter model with 75% fixed sparse weights. | | [`params_gpt3_125m_sparsewide-ift_dense.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/sparsity/pretraining/params_gpt3_125m_sparsewide-ift_dense.yaml) | 125M dense model for sparsewide-IFT comparison. | | [`params_gpt3_125m_sparsewide-ift_rigl75.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/sparsity/pretraining/params_gpt3_125m_sparsewide-ift_rigl75.yaml) | 125M model with 75% RigL sparsity in sparsewide-IFT setup. | | [`params_gpt3_125m_sparsewide-ift_static50.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/sparsity/pretraining/params_gpt3_125m_sparsewide-ift_static50.yaml) | 125M model with 50% static sparsity in sparsewide-IFT setup. | | [`params_gpt3_6p7b_vspdf_phase1.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/sparsity/pretraining/params_gpt3_6p7b_vspdf_phase1.yaml) | 6.7B sparse model for VSPDF Phase 1 training. | | [`params_gpt3_6p7b_vspdf_phase2.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/sparsity/pretraining/params_gpt3_6p7b_vspdf_phase2.yaml) | 6.7B sparse model for VSPDF Phase 2 training. | | [`params_gpt3_6p7b_vspdf_dart.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/sparsity/finetuning/params_gpt3_6p7b_vspdf_dart.yaml) | 6.7B model with DART sparsity applied for VSPDF fine-tuning. | The 1.3b(xl), 2.7b, 6.7b and 13b configs above show an example of setting micro batch size explicitly in the train\_input section of the config. Without this setting, the best micro batch size search will be performed automatically during compilation which could take long time for larger models. ## Model Input Tensor Specifications | **Input Name** | **Shape** | **Data Type** | **Description** | | ---------------- | ------------------------------------ | ------------- | ----------------------------------------------- | | `input_ids` | (batch\_size, max\_sequence\_length) | `torch.int32` | Token IDs, padded to full sequence length. | | `attention_mask` | (batch\_size, max\_sequence\_length) | `torch.int32` | 1s for valid tokens, 0s for padding. | | `labels` | (batch\_size, max\_sequence\_length) | `torch.int32` | Targets for language modeling (same as inputs). | These are generated using `GptHDF5DataProcessor.py`, which consumes PILE-formatted datasets and outputs `.h5` files via [`preprocess_data.py`](../../../data_preparation/data_preprocessing/preprocess_data.py). ## Workflow For example workflows using language models from the Cerebras Model Zoo, see our tutorials on [pretraining](../../../getting-started/pre-train-your-first-model) and [fine-tuning](../../../getting-started/fine-tune-your-first-model). For a complete list of Cerebras ModelZoo CLI commands, see the [command reference](../../cli-overview). ## Advanced Features This implementation supports: * **µP (Maximal Update Parametrization)**: For hyperparameter transfer from small proxy models to large target models.\ See [µP Tutorial](https://docs.cerebras.net/en/latest/wsc/Model-zoo/tutorials/mup/mup_docs.html). * **Cerebras-GPT Recipes**: Prebuilt configs under [`configs/Cerebras_GPT/`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gpt3/configs/Cerebras_GPT) to reproduce results from the [Cerebras-GPT blog](https://www.cerebras.net/cerebras-gpt). ## References * [Language Models are Few-Shot Learners (GPT-3 paper)](https://arxiv.org/abs/2005.14165) * [Language Models are Unsupervised Multitask Learners (GPT-2 paper)](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf)