ImageNet ResNet
Category of Task: Vision
Kind of Task: Image Classification
Overview
The ResNet model family is a set of convolutional neural networks that can be used as the base for a variety of vision tasks. ImageNet ResNets are a subset of the ResNet family which were designed specifically for classification on the ImageNet dataset.
Attribution
Paper: Deep Residual Learning for Image Recognition by Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
Code and hyperparameters: DeepLearningExamples Github repository by Nvidia
Architecture
The basic architecture defined in the original papers is as follows:
The first layer is a 7x7 Convolution with stride 2 and 64 filters.
Subsequent layers follow 4 stages with {64, 128, 256, 512} input channels with a varying number of residual blocks at each stage that depends on the family member. At the end of every stage, the resolution is reduced by half using a convolution with stride 2.
The final section consists of a global average pooling followed by a linear + softmax layer that outputs values for the specified number of classes
Family Members
ResNet family members are defined by their number of layers. Parameter count, accuracy, and training time are provided below.
Model Family Members |
Parameter Count |
Our Accuracy |
Training Time on 8xA100s |
---|---|---|---|
ResNet-18 |
11.5M |
TBA |
TBA |
ResNet-34 |
21.8M |
TBA |
TBA |
ResNet-50 |
25.6M |
76.5% |
3.83 hrs |
ResNet-101 |
44.5M |
78.1% |
5.50 hrs |
ResNet-152 |
60.2M |
TBA |
TBA |
Implementation details
The implementation follows He et al. except for the ResNet v1.5 change from Nvidia’s DeepLearningExamples. For bottleneck residual blocks that downsample, the downsampling is performed by the 3x3 convolution instead of the 1x1 convolution. Nvidia reports approximately +0.5% absolute accuracy and 5% decrease in throughput from this change.
Default Training Hyperparameters
Optimizer: SGD
Learning rate: 2.048
Momentum: 0.875
Weight decay: 1/32768
Batch size: 2048
LR Schedulers
Linear warmup for 8 epochs
Cosine decay after warmup
Number of epochs: 90