nD-RoPE: A Generalized RoPE for n-Dimensional Position Embedding

摘要

Rotary Position Embedding (RoPE) is widely adopted in Transformer models, yet its extension to high-dimensional domains lacks a unified theoretical formulation. Most existing approaches either apply rotations independently along each axis or mix frequencies empirically, which limits cross-dimensional interactions and yields direction-dependent representations. To address these limitations, we propose *nD-RoPE*, a decomposition-free generalization of rotary embeddings to arbitrary dimensions. From a translation-invariant formulation in continuous Hilbert space, we derive a spectral condition for isotropy that requires treating positions and frequencies as coupled $n$-dimensional vectors. We instantiate this principle with a multi-scale regular simplex wave-vector design that provides uniform directional coverage with maximal symmetry. Experiments across images, videos, and point clouds demonstrate consistent performance gains and improved generalization in high-dimensional settings.