EquiformerV2: Improved Equivariant Transformer for Scaling to
  Higher-Degree Representations

Yi-Lun Liao; Brandon Wood; Abhishek Das; Tess Smidt

返回文献探索

Paper · arXiv 2306.12059

EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations

Yi-Lun Liao, Brandon Wood, Abhishek Das, Tess Smidt

5 upvotesJune 21, 2023arXiv 预印本

AI 摘要

Equivariant Transformers, specifically EquiformerV2, achieve superior performance in 3D atomistic systems by using eSCN convolutions and architectural improvements, demonstrating better scalability, speed, and accuracy.

Equivariant TransformersEquiformer$SO(3)$ convolutionseSCN convolutionshigher-degree tensorsattention re-normalizationseparable $S^2$ activationseparable layer normalizationEquiformerV2OC20 datasetforcesenergiesDFT calculationsadsorption energies

Abstract

Equivariant Transformers such as Equiformer have demonstrated the efficacy of applying Transformers to the domain of 3D atomistic systems. However, they are still limited to small degrees of equivariant representations due to their computational complexity. In this paper, we investigate whether these architectures can scale well to higher degrees. Starting from Equiformer, we first replace SO(3) convolutions with eSCN convolutions to efficiently incorporate higher-degree tensors. Then, to better leverage the power of higher degrees, we propose three architectural improvements -- attention re-normalization, separable S^2 activation and separable layer normalization. Putting this all together, we propose EquiformerV2, which outperforms previous state-of-the-art methods on the large-scale OC20 dataset by up to 12% on forces, 4% on energies, offers better speed-accuracy trade-offs, and 2times reduction in DFT calculations needed for computing adsorption energies.