> ## Documentation Index > Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt > Use this file to discover all available pages before exploring further. # DINOv2 > Self-supervised vision model that learns general-purpose visual features without labeled data, excelling in diverse image and pixel-level tasks ## Model Description DINOv2 is a self-supervised vision transformer model by Meta that learns high-quality image representations without needing labeled data. It builds on the success of DINO by introducing architectural and training enhancements that deliver state-of-the-art performance across various computer vision tasks, including classification. ## Code Structure The code for this model is located in the [`dino`](https://github.com/Cerebras/modelzoo/tree/rel-2.4.2/src/cerebras/modelzoo/models/vision/dino) directory within ModelZoo. Here’s how it's organized: * [`configs/`](https://github.com/Cerebras/modelzoo/tree/rel-2.4.2/src/cerebras/modelzoo/models/vision/dino/configs): Contains YAML configuration files. * [`scripts/`](https://github.com/Cerebras/modelzoo/tree/rel-2.4.2/src/cerebras/modelzoo/models/vision/dino/scripts): Contains scripts for various workflows, including checkpoint conversion and image resizing. * [`model.py`](https://github.com/Cerebras/modelzoo/blob/rel-2.4.2/src/cerebras/modelzoo/models/vision/dino/model.py): The implementation of the DINOv2 model. * [`DinoImageDataProcessor.py`](https://github.com/Cerebras/modelzoo/blob/rel-2.4.2/src/cerebras/modelzoo/models/vision/dino/DinoImageDataProcessor.py): Data processor for DINOv2. ## Available Configurations | Configuration | Description | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | [`params_dinov2_large_224_bs1024.yaml`](https://github.com/Cerebras/modelzoo/blob/rel-2.4.2/src/cerebras/modelzoo/models/vision/dino/configs/params_dinov2_large_224_bs1024.yaml) | Config for pretraining, batch size 1024 | | [`params_dinov2_large_eval_linear.yaml`](https://github.com/Cerebras/modelzoo/blob/rel-2.4.2/src/cerebras/modelzoo/models/vision/dino/configs/params_dinov2_large_eval_linear.yaml) | Config for finetuning with downstream evaluation. | | [`params_dinov2_large_patch14_img224.yaml`](https://github.com/Cerebras/modelzoo/blob/rel-2.4.2/src/cerebras/modelzoo/models/vision/dino/configs/params_dinov2_large_patch14_img224.yaml) | Reference implementation config of DINOv2. | ## Model Input Tensor Specifications The tables below outline the expected input tensor formats for pretraining and fine-tuning. These formats are based on the configurations listed in the [Available Configurations](#available-configurations) section above. If you are using a custom configuration, you can check the tensor specifications by running: ```bash theme={null} cszoo data_processor benchmark --num_epochs 1 --steps_per_epoch 10 ``` | **Input Name** | **Shape** | **Data Type** | **Description** | | ---------------- | ----------------------------- | --------------- | --------------------------------------------------------------------------- | | `collated_masks` | (batch\_size, 2, 256) | `torch.bool` | Boolean mask indicating which patches are masked during training. | | `global_view` | (batch\_size, 2, 3, 224, 224) | `torch.float32` | Global image views (2 samples per batch, 3-channel images of size 224x224). | | `local_view` | (batch\_size, 8, 3, 98, 98) | `torch.float32` | Local image views (8 samples per batch, 3-channel images of size 98x98). | | **Input Name** | **Shape** | **Data Type** | **Description** | | -------------- | -------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `images` | (batch\_size, 3, 224, 224) | `torch.float32` | Preprocessed images. For training, images are augmented (e.g., random resized crop, horizontal flip) and normalized; for evaluation, they are resized and center-cropped. | | `labels` | (batch\_size,) | `torch.int32` | Ground truth labels corresponding to each input image. | ## About this Implementation This implementation of DINOv2 uses the `generic_image_encoders` architecture as its backbone. You can find the model architecture details in its [directory](https://github.com/Cerebras/modelzoo/tree/rel-2.4.2/src/cerebras/modelzoo/models/vision/generic_image_encoders). ### Differences Between Our Implementation and Meta's Unlike Meta’s version, which includes KoLeo loss, this implementation only includes DinoDistillationLoss and iBOTPatchLoss, which was introduced in DINOv2. Pretrained models from Meta and Hugging Face only include the backbone, meaning they cannot be used for continuous pretraining and are limited to downstream tasks. In contrast, our implementation provides everything needed for continuous pretraining. ## Workflow In this workflow we'll demonstrate how to get started using DINOv2, inlcuding for pretraining, continuous pretraining, and finetuning tasks. This workflow utilizes the ModelZoo CLI. For a list of all commands, please visit our [CLI page](https://training-docs.cerebras.ai/model-zoo/cli-overview). Before getting started, ensure that you've gone through our [setup and installation guide](https://training-docs.cerebras.ai/getting-started/setup-and-installation). Next, create a dedicated folder for assets (configs, data) and generated files (processed data files, checkpoints, logs, etc.): ``` mkdir dinov2 ``` Copy the sample model config for pretraining into your folder. ```bash theme={null} cszoo config pull dinov2_large_224_bs1024 -o dinov2 ``` Copy the sample model config for pretraining into your folder. ```bash theme={null} cszoo config pull dinov2_large_224_bs1024 -o dinov2 ``` Copy the sample model config for pretraining into your folder. ```bash theme={null} cszoo config pull dinov2_large_eval_linear -o dinov2 ``` Our implementation of DINOv2 supports all torchvision datasets. In our internal testing, we used ImageNet1K. To get started, set the dataset path to where your torchvision dataset is stored, ensuring it conforms to the torchvision standard. For more information on how to prepare datasets using `torchvision`, please visit our guide [here](https://github.com/Cerebras/monolith/tree/feacbef3c42fc6095c574deac7ed70d6c72e8ad2/src/models/src/cerebras/modelzoo/data/vision/classification/data). Once completed, your dataset directory should look as follows: ``` root_directory │-- meta.bin │-- train/ │ │-- n01440764 │ │ │-- n01440764_10026.JPEG │ │ │-- ... │ │-- n01443537 │ │ │-- ... │ │-- ... │ val/ │ │-- n01440764 │ │ │-- ILSVRC2012_val_00000946.JPEG │ │ │-- ... │ │-- n01443537 │ │ │-- ... │ │-- ... ``` This implementation does not support on-demand downloading, so make sure to download the dataset beforehand. Once your data directory is ready, modify the `root` parameter under `dataset` in the model config to point to the desired dataset location. Run the pretraining process using the provided configuration. ```bash theme={null} cszoo fit dinov2/params_dinov2_large_224_bs1024.yaml \ --mgmt_namespace= ``` ```bash theme={null} python src/cerebras/modelzoo/models/vision/dino/run.py CSX \ --mode train \ --params dinov2/params_dinov2_large_224_bs1024.yaml \ --mount_dirs \ --mgmt_namespace \ --python_paths ``` In addition to pretraining from scratch, you can continue training from an existing DINOv2 checkpoint. You can do this with your own pretrained checkpoint, using Meta's checkpoints (after converting to a Cerebras-compatible format) or by using the checkpoint we provide. In this workflow we will use our provided pretrained checkpoint. Download it before getting started. ```bash theme={null} wget -P dinov2 https://cerebras-public.s3.us-west-2.amazonaws.com/DINOv2/DINOv2Pretraining_ViTL_img224.mdl ``` Once the checkpoint has finished downloading, run the model with the new checkpoint. ```bash theme={null} cszoo fit dinov2/params_dinov2_large_224_bs1024.yaml \ --checkpoint_path dinov2/DINOv2Pretraining_ViTL_img224.mdl \ --load_checkpoint model \ --mgmt_namespace= ``` ```bash theme={null} python src/cerebras/modelzoo/models/vision/dino/run.py CSX \ --checkpoint_path dinov2/DINOv2Pretraining_ViTL_img224.mdl \ --mode train \ --params dinov2/params_dinov2_large_224_bs1024.yaml \ --mount_dirs \ --mgmt_namespace \ --python_paths ``` To begin finetuning, update the `data_dir` parameter in your configuration file. You can find this parameter under `fit` > `train_dataloader` and `val_dataloader`. Set it to the directory containing the data you want to use for fine-tuning. You can finetune DINOv2 using your own pretrained checkpoint or by using the checkpoint we provide. To download our pretrained checkpoint: ``` wget -P dinov2 https://cerebras-public.s3.us-west-2.amazonaws.com/DINOv2/DINOv2Pretraining_ViTL_img224.mdl ``` Next, convert the pre-trained DINOv2 checkpoint into a ViT-compatible classification format. Since DINOv2 is a self-supervised model, it does not include a classification head by default. The conversion process extracts the ViT backbone and attaches the required classification head. To perform this conversion, run the `convert_dinov2_to_vit.py` script as follows: ```bash theme={null} python convert_dinov2_to_vit.py \ --input_config dinov2/params_dinov2_large_eval_linear_cszoov2.yaml \ --output_config dinov2/finetuning_params_vit_classification.yaml \ --dataset_path --input_ckpt dinov2/DINOv2Pretraining_ViTL_img224.mdl --output_ckpt dinov2/finetuning_ViTClassification_DINOv2_ViTL_img224.mdl ``` Once the conversion script has finished running, open up the output configuration that was generated `finetuning_params_vit_classification.yaml` and specify the `backend` to be CSX in the `init` section as follows: ``` trainer: init: backend: backend_type: CSX cluster_config: num_csx: 1 ``` The classification head is randomly initialized after the conversion, so the classification accuracy can be expected to be low at the beginning of the training run. ### Running Finetuning Once the conversion script has finished running, use the output configuration and checkpoint to train the classification head. ```bash theme={null} cszoo fit dinov2/finetuning_params_vit_classification.yaml \ --checkpoint_path dinov2/finetuning_ViTClassification_DINOv2_ViTL_img224.mdl \ --load_checkpoint model \ --mgmt_namespace= ``` Once the conversion script has finished running, use the output configuration and checkpoint to train the classification head. ```bash theme={null} python src/cerebras/modelzoo/models/vision/dino/run.py CSX \ --checkpoint_path dinov2/finetuning_ViTClassification_DINOv2_ViTL_img224.mdl \ --mode train \ --params dinov2/finetuning_params_vit_classification.yaml \ --mount_dirs \ #optional --mgmt_namespace \ --python_paths #optional ``` We also provide a finetuned ViT classification checkpoint that you can download and use: ```bash theme={null} wget -P dinov2 https://cerebras-public.s3.us-west-2.amazonaws.com/DINOv2/ViTClassification_DINOv2_ViTL_img224.mdl ``` ## Advanced Use Cases In addition to the workflows outlined above, we provide a number of scripts for more advanced and experimental use cases. ### Adjusting Image Size Our DINOv2 implementation has been tested only with an image size of 224. Using other image sizes may lead to unexpected behavior, therefore this should be considered an experimental feature. You can continue training from an existing DINOv2 checkpoint while adjusting parameters such as image size. For this purposes, we provide the `change_image_size.py` to modify the checkpoint and config. ```bash theme={null} python change_image_size.py \ --input_config dinov2/params_dinov2_large_patch14.yaml \ --input_ckpt \ --output_config dinov2/params_dinov2_continuous_pretraining.yaml \ --output_ckpt dinov2/dinov2_continuous_pretraining_chkpt.mdl \ --global_size 518 \ --local_size 224 ``` After the script finishes running, it will generate new configuration and checkpoint files. Use these files to start your training run as follows: ```bash theme={null} cszoo fit dinov2/params_dinov2_continuous_pretraining.yaml --checkpoint_path dinov2/output_config --load_checkpoint model ``` ### Configuring Per-Layer Learning Rate Schedulers As part of our DINOv2 model offering, we provide a [script](https://github.com/Cerebras/monolith/tree/rel-2.4.2/src/models/src/cerebras/modelzoo/models/vision/dino/scripts) for generating a config file that includes learning rate schedulers, following the approach used in Meta's original implementation of the model. Users can modify the learning rate settings to experiment with different schedules, but we recommend adhering to Meta’s specifications for optimal results. For a detailed explanation of Meta's training methods for DINOv2, please refer to the [paper](https://arxiv.org/pdf/2304.07193). This script should be used with the [reference implementation config](https://github.com/Cerebras/modelzoo/blob/rel-2.4.2/src/cerebras/modelzoo/models/vision/dino/configs/params_dinov2_large_patch14_img224.yaml) that is provided. It will output a configuration similar to [`params_dinov2_large_224_bs1024_cszoov2.yaml`](https://github.com/Cerebras/monolith/blob/rel-2.4.2/src/models/src/cerebras/modelzoo/models/vision/dino/configs/mz/params_dinov2_large_224_bs1024_cszoov2.yaml). **To run the script:** To use Meta’s predefined learning rate schedulers without modifications, simply specify only the `input_file_name` and `output_file_name` flags. ``` python /src/modelzoo/models/vision/dino/scripts/create_dinov2_config_with_schedulers.py \ --input_file_name \ --output_file_name \ --base_lr \ --batch_size \ --total_iters \ --lr_decay_rate \ --patch_emb_multiplier ``` ## References * [DINOv2: Learning Robust Visual Features without Supervision](https://arxiv.org/pdf/2304.07193) * [Emerging Properties in Self-Supervised Vision Transformers](https://arxiv.org/pdf/2104.14294)