An Optimal Property of Principal Components

• Published in: The Annals of mathematical statistics Vol. 36; no. 5; pp. 1579 - 1582
• Main Author: Darroch, J. N.
• Format: Journal Article
• Language: English
• Published: Institute of Mathematical Statistics 01.10.1965; The Institute of Mathematical Statistics
• Subjects: Common property ownership; Linear transformations; Property titles
• ISSN: 0003-4851; 2168-8990
• DOI: 10.1214/aoms/1177699920

Let x' = (x1, x2, ⋯, xp) be a random vector and let E[ x] = 0, E[ xx'] = Σ = (σij) where we assume that Σ is non-singular. Further let \[\Sigma = T\Lambda T' = (t_1, t_2, \cdots, t_p) \left[\begin{array}{c|c|c|c|c|c|c}\lambda_1 & 0 & \cdot & \cdot & \cdot & 0 \\ 0 & \lambda_2 & \cdot & \cdot & \cdot & 0 \\ \vdots & \vdots & \vdots & \vdots \\0 & 0 & \cdot & \cdot & \cdot & \lambda_p\end{array} \right] \] \[\left[\begin{array}{|c}t'_1 \\ t'_2 \\ \vdots \\ t'_p\end{array}\right] \] where TT' = I and where we suppose that the eigenvalues of Σ are in order of decreasing magnitude, that is $\lambda_1 \geqq \lambda_2 \geqq \cdots \geqq \lambda_p > 0$. The principal components of x, namely u1= t'1x, u2= t'2x, ⋯, up= tp' x were introduced by Hotelling (1933) who characterised them by certain optimal properties. Since then Girshick (1936), Anderson (1958) and Kullback (1959) have characterised the principal components by slightly different sets of optimal properties. Thus Anderson shows that u1is the linear function α1' x having maximum variance subject to α1' α1= 1; u2is the linear function α2'x which is uncorrelated with u1and has maximum variance subject to α2'α2= 1; and so on. The above mentioned characterisations have two properties in common; they introduce the principal components one by one and, more importantly, the optimal properties hold only with-in the class of linear functions of x1, x2, ⋯, xp. In the following theorem the first k principal components are characterized by an optimal property within the class of all random variables.