Walrus: A Cross-domain Foundation Model for Continuum Dynamics
Google Research · University of Cambridge · Flatiron Institute · Ellison Institute of Technology · Simons Foundation · Johns Hopkins University · New York University · RIKEN · CNRS · Flatiron Institute Polymathic AI · UC Berkeley / Polymathic AI · University of Wisconsin - Madison · Princeton University · Simons Foundation/ NYU/ Polymathic
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Foundation models have transformed machine learning for language and vision, but achieving comparable impact in physical simulation remains a challenge. Data heterogeneity and unstable long-term dynamics inhibit learning from sufficiently diverse dynamics, while varying resolutions and dimensionalities challenge efficient training on modern hardware. Through empirical and theoretical analysis, we incorporate new approaches to mitigate these obstacles, including a harmonic-analysis–based stabilization method, load-balanced distributed 2D-3D training strategies, and compute-adaptive tokenization. Using these tools, we develop \Walrus, a transformer-based foundation model developed primarily for fluid-like continuum dynamics. \Walrus\ is pretrained on nineteen diverse scenarios spanning astrophysics, geoscience, rheology, plasma physics, acoustics, and classical fluids. Experiments show that \Walrus\ outperforms prior foundation models on both short- and long-term prediction horizons on downstream tasks and across the breadth of pretraining data, while ablation studies confirm the value of our contributions to forecast stability, training throughput, and transfer performance over conventional approaches.