Deep Learning

After my Swish and SwiGLU adventure, I wanted to do another set of chronological reading, this time on normalization techniques. Much like the activation functions, these are very commonplace in deep learning architecture, but beyond BatchNorm I hadn’t sat down to read these as a group to parse out the use cases and the limitations that subsequent techniques answer. ...

Yesterday I had the joy of randomly reading papers that introduced Swish and SwiGLU . Recently, I decided I wanted to update my knowledge of deep learning from around 2020, when I pivoted slightly more towards generative models (diffusion and GANs) and neuroscience to complete my PhD. I knew there were a lot of exciting things coming out that I’ve heard of, but haven’t had a chance to sit down and read. ...