Swish and SwiGLU
Yesterday I had the joy of randomly reading papers that introduced Swish and SwiGLU . Recently, I decided I wanted to update my knowledge of deep learning from around 2020, when I pivoted slightly more towards generative models (diffusion and GANs) and neuroscience to complete my PhD. I knew there were a lot of exciting things coming out that I’ve heard of, but haven’t had a chance to sit down and read. ...