> ## Documentation Index
> Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Input Data Configuration

> Learn how to configure local or Hugging Face data as input for preprocessing.

For local data:

* Set `type` to `local`.

* Use `source` to provide the path to the input directory.

For Hugging Face data:

* Set `type` to `huggingface`.

* Use `source` to specify the dataset name from the Hugging Face hub.

* Use `split` to specify the dataset split.

The preprocessing pipeline passes these parameters to the Hugging Face `load_dataset` API.

<Note>
  When calling the API, parameters are passed as keyword arguments and they must conform to the specifications outlined by HuggingFace. Refer to the `load_dataset` documentation [here](https://huggingface.co/docs/datasets/v2.20.0/en/package_reference/loading_methods#datasets.load_dataset).
</Note>

## Config Examples

Use the tabs to view examples:

<CodeGroup>
  ```yaml Local theme={null}
  setup:
    data:
        source: "/input/dir/here"
        type: "local"

    mode: "pretraining"
    output_dir: "./output/dir/here/"
    processes: 1

  ```

  ```yaml Hugging Face theme={null}
  setup:
    data:
        source: "stanfordnlp/imdb"
        type: "huggingface"
        split: "test"
        cache_dir: "path/to/cache_dir"
        ...other parameters accepted by HuggingFace ``load_dataset`` API...


    mode: "pretraining"
    output_dir: "./output/dir/here/"
    processes: 1

  ```
</CodeGroup>

<Tip>View example configs for various use cases [here](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/data_preparation/data_preprocessing/configs). </Tip>

## What’s Next?

Now that you’ve configured your input data, learn how to process it with [read hooks](./read-hooks).