https://hojaeklee.github.io/tags/swish/Recent content in Swish on Hojae LeeHugo -- 0.160.1enSun, 27 Sep 2026 00:00:00 +0000https://hojaeklee.github.io/til/deep_learning/20260927_swish_and_swiglu/Sun, 27 Sep 2026 00:00:00 +0000https://hojaeklee.github.io/til/deep_learning/20260927_swish_and_swiglu/<p>Yesterday I had the joy of randomly reading papers that introduced <a href="https://www.alphaxiv.org/abs/1710.05941" target="_blank" rel="noopener noreferrer">Swish</a> and <a href="">SwiGLU</a> . Recently, I decided I wanted to update my knowledge of deep learning from around 2020, when I pivoted slightly more towards generative models (diffusion and GANs) and neuroscience to complete my PhD. I knew there were a lot of exciting things coming out that I&rsquo;ve heard of, but haven&rsquo;t had a chance to sit down and read.</p>