Swish and SwiGLU

Yesterday I had the joy of randomly reading papers that introduced Swish and SwiGLU . Recently, I decided I wanted to update my knowledge of deep learning from around 2020, when I pivoted slightly more towards generative models (diffusion and GANs) and neuroscience to complete my PhD. I knew there were a lot of exciting things coming out that I’ve heard of, but haven’t had a chance to sit down and read.

Swish is an activation function:

$$f(x) = x \cdot \text{sigmoid}(\beta x)$$

The paper (“Searching for Activation Functions”) was published back in 2017 by the Google Brain team, and it’s interesting in that they tried to use automatic search techniques (some reinforcement learning with RNN controller).

SwiGLU is more of a replacement for FFN (Feed Forward Networks) in Transformer architecture with a GLU (Gated Linear Unit). The actual GLU was introduced by Dauphin et al. in 2016, while this paper (“GLU Variants Improve Transformer”) was published in 2020 by Noam Shazeer. The paper compares several GLU variants, but for the purpose of this post, SwiGLU is:

$$\text{SwiGLU}(x, W, V, b, c) = \text{Swish}(xW + b) \odot (xV + c)$$

The cool thing about GLUs in general is that there are two parallel branches, i.e. $xW+b$ and $xV+c$, one of which gates the other after being processed by an activation such as ReLU, GELU, or Swish.