If your dataloader is too slow to keep up with your model running on Cerebras Wafer-Scale Cluster, you’ll see the following warning message


WARNING: Input starvation detected
Please check dataloader throughput

If the problem persists, the following error will be thrown


ERROR:   Declaring stall due to input starvation, no change in status for 630 secs

This error means that the model is too fast compared to the speed of your dataloader. Therefore you will need to rethink how you can speed up your data pipeline. Visit the section Creating custom dataloaders that includes:

  • Examples on how to convert dataloaders that already work on GPUs

  • Best practices to improve the performance of your dataloaders.

Failure To Trace Due To Functionalization Error Out Of Memory Errors And System Resources