https://pwaabdullah.github.io/tags/activation-functions/ Recent content in Activation Functions on Abdullah Al Mamun https://pwaabdullah.github.io/og-default.png https://pwaabdullah.github.io/og-default.png Hugo en newabdullah 2025 Wed, 16 Sep 2026 00:00:00 +0000 https://pwaabdullah.github.io/posts/activation-functions/ Wed, 16 Sep 2026 00:00:00 +0000 https://pwaabdullah.github.io/posts/activation-functions/ A practical guide to activation functions: sigmoid, tanh, the ReLU family, GELU, SiLU/Swish, and the gated variants (SwiGLU, GeGLU) that modern LLMs actually use, with the vanishing-gradient and dying-ReLU math, PyTorch APIs, and interview traps.