H-AMR: A New GPU-accelerated GRMHD Code for Exascale Computing with 3D Adaptive Mesh Refinement and Local Adaptive Time Stepping

• Published in: The Astrophysical journal. Supplement series Vol. 263; no. 2; pp. 26 - 42
• Main Authors: Liska, M. T. P., Chatterjee, K., Issa, D., Yoon, D., Kaaz, N., Tchekhovskoy, A., van Eijnatten, D., Musoke, G., Hesp, C., Rohoza, V., Markoff, S., Ingram, A., van der Klis, M.
• Format: Journal Article
• Language: English
• Published: Saskatoon The American Astronomical Society 01.12.2022; IOP Publishing
• Subjects: Accretion; Accretion disks; Astronomy & Astrophysics; Black hole physics; Black holes; Finite element method; Fluid flow; Grid refinement (mathematics); High energy astrophysics; Magnetic fields; Magnetohydrodynamical simulations; Magnetohydrodynamics; Simulation
• ISSN: 0067-0049; 1538-4365
• DOI: 10.3847/1538-4365/ac9966

Abstract General relativistic magnetohydrodynamic (GRMHD) simulations have revolutionized our understanding of black hole accretion. Here, we present a GPU-accelerated GRMHD code H-AMR with multifaceted optimizations that, collectively, accelerate computation by 2–5 orders of magnitude for a wide range of applications. First, it introduces a spherical grid with 3D adaptive mesh refinement that operates in each of the three dimensions independently. This allows us to circumvent the Courant condition near the polar singularity, which otherwise cripples high-resolution computational performance. Second, we demonstrate that local adaptive time stepping on a logarithmic spherical-polar grid accelerates computation by a factor of ≲10 compared to traditional hierarchical time-stepping approaches. Jointly, these unique features lead to an effective speed of ∼10 9 zone cycles per second per node on 5400 NVIDIA V100 GPUs (i.e., 900 nodes of the OLCF Summit supercomputer). We illustrate H-AMR's computational performance by presenting the first GRMHD simulation of a tilted thin accretion disk threaded by a toroidal magnetic field around a rapidly spinning black hole. With an effective resolution of 13,440 × 4608 × 8092 cells and a total of ≲22 billion cells and ∼0.65 × 10 8 time steps, it is among the largest astrophysical simulations ever performed. We find that frame dragging by the black hole tears up the disk into two independently precessing subdisks. The innermost subdisk rotation axis intermittently aligns with the black hole spin, demonstrating for the first time that such long-sought alignment is possible in the absence of large-scale poloidal magnetic fields.