Knowledge Distillation with ResNet for Raw Image Classification
· Tech · Python / PyTorch / ResNet / Knowledge Distillation / EuroSAT
I ran a small raw-image classification experiment comparing a plain ResNet-18, an augmented ResNet-50 teacher, and a ResNet-18 trained with augmentation and knowledge distillation.
Dataset and setup
The dataset was EuroSAT. I used the multispectral TIFF files from EuroSAT_MS.zip, which contain all 13 Sentinel-2 bands in their original value range. Every experiment used the same 80/20 train-test split.
The baseline was a ResNet-18 modified for 32×32×13 input, trained without augmentation. A ResNet-50 on the same data quickly overfit: training accuracy reached 98%, while test accuracy stayed near 85%. After random flips and other augmentation, the teacher reached 93.09% test accuracy.
Finally, I initialized another ResNet-18 from scratch and trained it from the ResNet-50 teacher using both soft-label and hard-label losses. After 10 epochs, it reached 96.65%.
| Network | Epochs | Test accuracy |
|---|---|---|
| ResNet-18 baseline | 10 | 89.57% |
| ResNet-50 teacher | 20 | 93.09% |
| ResNet-18, distilled and augmented | 10 | 96.65% |
The result is a useful reminder that the larger network is not automatically the best deployed model. In this setup, its softened predictions provided a better training signal for the smaller network than hard labels alone.