> ## Documentation Index
> Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# GPT-J & GPT-Neox

> Decoder-only language models by EleutherAI, designed for high-throughput training and capable zero-shot performance on a range of natural language tasks.

## Model Description

GPT-J and GPT-Neox are families of decoder-only language models developed by EleutherAI and trained on the [Pile](https://arxiv.org/abs/2101.00027) dataset — a curated mixture of diverse text sources. Both models are designed to be efficient, flexible, and performant in zero-shot settings without the need for task-specific fine-tuning.

### GPT-J

GPT-J is a 6B parameter auto-regressive transformer with architectural similarities to GPT-3. It introduces a parallel decoder block where attention and feed-forward layers are computed in parallel and added together, improving throughput by approximately 15% compared to traditional sequential transformer blocks. This design is especially beneficial for distributed training and single-device setups where minimizing cross-device communication is critical.

GPT-J also adopts Rotary Position Embeddings (RoPE) — applying them to 25% of the features while using sinusoidal embeddings for the remainder. This hybrid approach balances convergence speed with long-context modeling capabilities. Additionally, GPT-J employs dense attention, prioritizing simplicity and training stability at this scale over sparse alternatives.

### GPT-Neox

GPT-Neox shares the same architecture as GPT-J with a few refinements:

* Untied LayerNorm: Each transformer block uses two independent layer normalization layers instead of a shared one.
* Enhanced Tokenizer: Tokenizers were retrained on the Pile and optimized for whitespace handling, repeated tokens, and programming languages — making Neox more robust for structured text like code.

These design choices allow GPT-Neox to generalize well across a broad range of domains and sequence lengths, including natural language and code generation tasks.

## Code Structure

The code for these models is located in the [`/gptj`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gptj) directory within ModelZoo. Here’s how it's organized:

* [`/configs`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gptj/configs): Contains YAML configuration files for GPT-J.
* [`/continuous_pretraining/configs`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gptj/continuous_pretraining/configs): Contains configs for continuous pretraining of GPT-J.
* [`model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gptj/model.py): The implementation of the GPT-Neox model.
* [`gptneox/model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gptj/gptj_model.py): The implementation of the GPT-J model.

<Note>
  Our implementations of GPT-J and GPT-Neox are built on top of our GPT-2 backbone. For more details, see [`gpt2_model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/gpt2/gpt2_model.py).
</Note>

## Available Configurations

| Configuration                                                                                                                                                                      | Description                                                 |
| :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------- |
| [`params_gptj_6B.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gptj/configs/params_gptj_6B.yaml)                                          | Standard 6B parameter GPT-J model.                          |
| [`params_gptj_6B_muP.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gptj/configs/params_gptj_6B_muP.yaml)                                  | GPT-J model configured with μ-parameterization for scaling. |
| [`params_gptj_6B_TRC2.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gptj/configs/continuous_pretraining/configs/params_gptj_6B_TRC2.yaml) | GPT-J 6B model for continued pretraining on TRC2 datasets.  |
| [`params_gpt_neox_20B.yaml`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/gptneox/configs/params_gpt_neox_20B.yaml)                             | GPT-Neox model with 20B parameters.                         |

## Workflow

For example workflows using language models from the Cerebras Model Zoo, see our tutorials on [pretraining](../../../getting-started/pre-train-your-first-model) and [fine-tuning](../../../getting-started/fine-tune-your-first-model).

For a complete list of Cerebras ModelZoo CLI commands, see the [command reference](../../cli-overview).

## References

* Wang, S. et al. (2021). [Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX](https://github.com/kingoflolz/mesh-transformer-jax)
* Shoeybi, M. et al. (2019). [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053)
* Su, J. et al. (2021). [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864)
* Brown, T. et al. (2020). [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
* EleutherAI (2021). [Rotary Embeddings: A Relative Revolution](https://blog.eleuther.ai/rotary-embeddings/)
* Black, S. et al. (2022). [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://arxiv.org/pdf/2204.06745)