> ## Documentation Index
> Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Truncation of Sequences

> Our data processing pipeline supports truncation of sequences to MSL, in order to avoid skipping them.

Truncation ensures that the total sequence length remains within the specified `max_sequence_length` limit. It's currently supported for FineTuning and VSLFineTuning mode, and applies to both prompt/completion datasets and multi-turn datasets structured as user/assistant interactions.

Truncation supports two modes:

* `keep_start`: Truncates from sequence end, retains beginning

* `keep_end`: Truncates from sequence start, retains end

By default, truncation prioritizes preserving responses (i.e., the completion/assistant part of the sequence). Truncation begins with the first part of the sequence (prompts or user inputs). If the required truncation to fit within the `max_sequence_length` cannot be achieved from this first part, the second part (responses or assistant outputs) is truncated as well.

Finally, if the tokens available for truncation are insufficient to meet the `max_sequence_length` requirement, the sequence is skipped.

<Info>
  The `keep_start` and `keep_end` modes apply only when truncating the first part of the sequence (i.e., prompts or user inputs). If truncation extends to the second part (responses or assistant outputs), the default mode is `keep_start`, i.e tokens are removed from the end.
</Info>

## Specifying Truncation in the Config File

The `truncate_to_msl` section in the configuration file specifies the parameters for sequence truncation to ensure the total sequence length remains within the `max_sequence_length` (MSL).

Available options are:

* `truncation_mode`: Specifies the truncation strategy.

  * `keep_start`: Removes tokens from the end of the sequence, preserving the start.

  * `keep_end`: Removes tokens from the start of the sequence, preserving the end.

* `max_turn_length`: Sets the maximum allowed length for any single turn (segment) in the sequence.

  * Turns exceeding this limit are truncated according to the specified `truncation_mode`.

Example Configuration:

```yaml theme={null}
truncate_to_msl:
    truncation_mode: "keep_start"
    max_turn_length: 512
```

In this example:

* Truncation will prioritize removing tokens from the end (`keep_start` mode).

* Each turn in the sequence will be restricted to a maximum of 512 tokens after truncation.