Adjoint-based exact Hessian computation

• Published in: BIT Vol. 61; no. 2; pp. 503 - 522
• Main Authors: Ito, Shin-ichi, Matsuda, Takeru, Miyatake, Yuto
• Format: Journal Article
• Language: English
• Published: Dordrecht Springer Netherlands 01.06.2021; Springer Nature B.V
• Subjects: Algorithms; Approximation; Boundary value problems; Computational Mathematics and Numerical Analysis; Hessian matrices; Mathematical analysis; Mathematics; Mathematics and Statistics; Matrix methods; Memory tasks; Multiplication; Numeric Computing; Numerical integration; Numerical methods; Optimization; Runge-Kutta method; Subspace methods; 65L05; 65L06; 65P10; 65F10; 65L09; Symplectic partitioned Runge–Kutta method; Adjoint method; 2nd-order adjoint method; 65K10; 34H05; 49M25; Hessian
• ISSN: 0006-3835; 1572-9125
• DOI: 10.1007/s10543-020-00833-0

We consider a scalar function depending on a numerical solution of an initial value problem, and its second-derivative (Hessian) matrix for the initial value. The need to extract the information of the Hessian or to solve a linear system having the Hessian as a coefficient matrix arises in many research fields such as optimization, Bayesian estimation, and uncertainty quantification. From the perspective of memory efficiency, these tasks often employ a Krylov subspace method that does not need to hold the Hessian matrix explicitly and only requires computing the multiplication of the Hessian and a given vector. One of the ways to obtain an approximation of such Hessian-vector multiplication is to integrate the so-called second-order adjoint system numerically. However, the error in the approximation could be significant even if the numerical integration to the second-order adjoint system is sufficiently accurate. This paper presents a novel algorithm that computes the intended Hessian-vector multiplication exactly and efficiently. For this aim, we give a new concise derivation of the second-order adjoint system and show that the intended multiplication can be computed exactly by applying a particular numerical method to the second-order adjoint system. In the discussion, symplectic partitioned Runge–Kutta methods play an essential role.