Conservation Laws for Modern Neural Architectures
National University of Singapore (NUS) · Center for AI Research (CAIR) at VinUniversity · Hanoi University of Science and Technology · School of Information and Communications Technology, Hanoi University of Science and Technology · National University of Singapore
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Understanding gradient descent dynamics is key to explaining the success of over-parameterized models, where implicit bias manifests through conservation laws in gradient flow. While such laws are well understood for linear and ReLU networks, they remain largely unexplored for modern architectures. This work develops a unified framework to characterize conservation laws for contemporary models, including feedforward networks with GELU, SiLU, and SwiGLU activations, multihead attention with sinusoidal and rotary positional encodings, and Mixture-of-Experts architectures under diverse gating designs. Our theoretical findings are supported by experiments that validate the predicted invariants.