Rotary Position Encodings for Graphs
University of Cambridge · Kensho Technologies/S&P Global · Attention Engineering · Cambridge University Press and Assessment · Google DeepMind · University of Cambridge, Alan Turing Institute · Google DeepMind Robotics & Columbia University · Google DeepMind / University of Cambridge
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data. We find that rotating tokens depending on the spectrum of the graph Laplacian efficiently injects structural information into the attention mechanism, boosting performance in synthetic and real-world graph learning tasks. This approach, coined _Wave-Induced Rotary Encodings_ (WIRE), enjoys intriguing theoretical properties: it recovers regular RoPE on grids, and depends asymptotically on the graph effective resistance. Unlike bias-based relative position encodings, WIRE is compatible with linear attention.