Global Convergence of Deep Galerkin and PINNs Methods for Solving Partial Differential Equations

• Published in: arXiv.org
• Main Authors: Jiang, Deqing, Sirignano, Justin, Cohen, Samuel N
• Format: Paper
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 10.05.2023
• Subjects: Algorithms; Convergence; Deep learning; Finite difference method; Galerkin method; Kernel functions; Machine learning; Mathematical analysis; Neural networks; Operators (mathematics); Ordinary differential equations; Partial differential equations; Teaching methods

Numerically solving high-dimensional partial differential equations (PDEs) is a major challenge. Conventional methods, such as finite difference methods, are unable to solve high-dimensional PDEs due to the curse-of-dimensionality. A variety of deep learning methods have been recently developed to try and solve high-dimensional PDEs by approximating the solution using a neural network. In this paper, we prove global convergence for one of the commonly-used deep learning algorithms for solving PDEs, the Deep Galerkin Method (DGM). DGM trains a neural network approximator to solve the PDE using stochastic gradient descent. We prove that, as the number of hidden units in the single-layer network goes to infinity (i.e., in the ``wide network limit"), the trained neural network converges to the solution of an infinite-dimensional linear ordinary differential equation (ODE). The PDE residual of the limiting approximator converges to zero as the training time \(\rightarrow \infty\). Under mild assumptions, this convergence also implies that the neural network approximator converges to the solution of the PDE. A closely related class of deep learning methods for PDEs is Physics Informed Neural Networks (PINNs). Using the same mathematical techniques, we can prove a similar global convergence result for the PINN neural network approximators. Both proofs require analyzing a kernel function in the limit ODE governing the evolution of the limit neural network approximator. A key technical challenge is that the kernel function, which is a composition of the PDE operator and the neural tangent kernel (NTK) operator, lacks a spectral gap, therefore requiring a careful analysis of its properties.