The Normalizations I wanted to look into
After my Swish and SwiGLU adventure, I wanted to do another set of chronological reading, this time on normalization techniques. Much like the activation functions, these are very commonplace in deep learning architecture, but beyond BatchNorm I hadn’t sat down to read these as a group to parse out the use cases and the limitations that subsequent techniques answer. ...