> ## Documentation Index > Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt > Use this file to discover all available pages before exploring further. # Data Deduplication Pipeline > Learn how to set up and run a deduplication pipeline on the Cerebras platform. Data deduplication is a crucial step in managing large-scale datasets, ensuring data quality, and optimizing storage. The data deduplication tool is designed to remove redundant data from text datasets to improve model training quality and efficiency. The pipeline consists of several stages, which our `dedup.py` script executes: 1. MinHash generation 2. duplicate pairs generation 3. duplicate graph construction 4. generating the final list of duplicates If needed, you can run each stage of the pipeline individually. See [Manual Instructions](#optional-manual-instructions) for more details. To optimize the deduplication process, we leverage the [datasketch library](http://ekzhu.com/datasketch/minhash.html) for efficient similarity detection and incorporate enhancements to address memory usage and runtime performance. Our approach implements a **producer-consumer model** to parallelize I/O operations, which are the primary bottleneck in runtime. Additionally, memory consumption is minimized by retaining only a single representative document from each group of duplicates in memory at any given time, ensuring efficient resource utilization during processing. ## Before You Begin Ensure the Cerebras Model Zoo and its dependencies are [installed](../../getting-started/setup-and-installation) if you're running on a *Cerebras Wafer-Scale cluster* (specifically the prerequisites listed [here](https://github.com/Cerebras/modelzoo/blob/main/requirements.txt)). If you'd like to run this *locally*, do the following: ```bash theme={null} virtualenv source /bin/activate pip install -r requirements.txt ``` ## Run the Script To run the deduplication pipeline, execute the following command: ```bash theme={null} python dedup.py --dataset_name --input_dir --jsonl_key --format --output_dir ``` The process generates an output directory containing compressed `.jsonl.zst` files. By default, each compressed file will be of size 16 MB, and the `jsonl_key` will be `text`. ## (Optional) Manual Instructions While the `dedup.py` script runs the deduplication pipeline end-to-end, you can also run each stage individually if needed. ### MinHash Generation MinHash generation can be a very slow process. We recommend running it separately before creating a MinHashLSH index. To calculate MinHash object for each document, we strip, lowercase, remove punctuation, consecutive spaces, newlines and tabs from each document. Afterwards, we construct a list of 13-grams that are later used as features to create a document signature to add into MinHashLSH index. (More details about MinHash can be found at [Identifying and Filtering Near-Duplicate Documents](https://cs.brown.edu/courses/cs253/papers/nearduplicate.pdf).) We also apply NFC normalization, as well as filter out short documents, before yielding the documents. For custom datasets, you also need to specify the `jsonl_key` as well as the format of the dataset. By default, the `jsonl_key` is set to be `text` and the format to be `jsonl`. Here is the format of the command to run MinHash generation: ```bash theme={null} python to_hash.py --dataset_name --input_dir --output_dir --job_id --jsonl_key --format ``` To reduce the total processing time, multiple jobs can be run for each corpus in parallel by using multiple job IDs - starting from 0. By default, the script works expecting one job, but if you wish, you can replicate the scripts across multiple machines and the script chunks the list of files to be processed equally, to do the MinHash generation. This assumes that the dataset is present at a common place that is accessible by all the machines. ### Duplicate Pairs Generation In this step, we build a MinHashLSH index and query it to locate near duplicates (More reading here: [Chapter 3, Mining of Massive Datasets](http://infolab.stanford.edu/~ullman/mmds/ch3.pdf). ) We use a Jaccard similarity threshold of 0.8 by default to determine whether a pair of documents should be considered as a duplicate, but you can specify it according to your own needs. ```bash theme={null} python generate_duplicate_pairs.py --input_dir --out_file /duplicates/duplicate_pairs.txt ``` ### Duplicate Graph Construction & Search for Connected Components After locating duplicate pairs, we need to find connected components containing documents that are duplicates with each other. To make it more illustrative, consider these pairs: (A, B), (A, C), (A, E). We are going to form a cluster of (A, B, C, E) and keep only one document from the component. We evaluated the performance and memory consumption of [networkx](https://networkx.org/), [graphtool](https://graph-tool.skewed.de/), and [networkit](https://networkit.github.io/). `networkit` offered most efficient implementation as it is designed to work with large graphs and features great parallelism. Below you can find an example command how to construct a graph from documents pairs: ```bash theme={null} python generate_connected_components.py --input_dir /duplicates/connected_components.pickle ``` ### Generate Final List of Duplicates In this step, we generate the final deduplicated dataset. We dump the original dataset in fixed-size files, in the `jsonl.zst` format, whose size is configurable in the `deduplicate_dataset.py` file. By default, we create files of 16 MB. Below you can find an example command on how to generate the final deduplicated dataset: ```bash theme={null} python deduplicate_dataset.py --input_file /duplicates/connected_components.pickle --input_dir --output_dir --format ```