Intrinsic Task Symmetry Drives Generalization in Algorithmic Tasks

摘要

Grokking, a sudden transition from memorization to generalization, has been closely linked to the emergence of low-dimensional representations; yet the mechanism driving this organization remains elusive. Here, we propose that intrinsic task symmetries are the key drivers of grokking, inducing structured geometries in representation space. Our analysis reveals a consistent three-stage training dynamic: (i) data memorization, (ii) intrinsic symmetry acquisition, and (iii) geometric organization. We show that generalization emerges during the symmetry acquisition phase, and subsequently the embedding space organizes into a low-dimensional structured geometry. We validate this mechanism across diverse algorithmic domains, spanning algebraic (modular arithmetic), structural (graph metric completion), and relational (comparison) reasoning tasks. Leveraging these insights, we formulate a symmetry-based criterion for generalization and propose symmetry- and geometry-prompting training strategies that can accelerate generalization. Together, our results establish intrinsic symmetry as a central mechanism enabling neural networks to move beyond memorization and achieve robust algorithmic reasoning.