After my Swish and SwiGLU adventure, I wanted to do another set of chronological reading, this time on normalization techniques. Much like the activation functions, these are very commonplace in deep learning architecture, but beyond BatchNorm I hadn’t sat down to read these as a group to parse out the use cases and the limitations that subsequent techniques answer.

Organizing them chronologically:

  1. BatchNorm (March 2015)
  2. LayerNorm (July 2016)
  3. InstanceNorm (November 2017)
  4. GroupNorm (June 2018)
  5. RMSNorm (October 2019)

And an additional paper I want to read:

  1. On Layer Normalization in the Transformer Architecture (June 2020)

Yesterday I read the BatchNorm, LayerNorm, and GroupNorm paper (and somewhat skimmed over RMSNorm). I actually read InstanceNorm paper quite carefully many years ago because it was quite relevant to the GAN-related work I was doing at that time. Some notes I took yesterday:

  • BatchNorm significantly sped up the convergence of models by addressing Internal Covariate Shift. In the paper, the authors also showed BatchNorm allowed training with sigmoid activation functions in Inception-like network, if I remember correctly. I thought that was pretty cool.
  • BatchNorm introduces two parameters to learn (scale and shift) per feature.
  • LayerNorm does something similar but averages across features instead of batch, making it batch size-independent. This was addressing the limitation of BN in settings such as RNN and online learning.
  • GroupNorm addresses the limitation of BatchNorm when the batch size is small (i.e. memory constrained problems). This can happen during large image segmentation or video processing, where high-res images limit the batch size one can process.
  • GroupNorm is also a generalization of Layer and Instance Norm, depending on the Group $G$. Best explained by their figure below.

More to come about InstanceNorm and RMSNorm.