Part 4: Time Series II

EOF Analysis

Principal Component

Rotated EOF

Complex EOF

Empirical Orthogonal Function Analysis

Empirical Orthogonal Function (EOF) analysis attempts to find a relatively small number of independent variables (predictors; factors) which convey as much of the original information as possible without redundancy.

EOF analysis can be used to explore the structure of the variability within a data set in a objective way, and to analyze relationships within a set of variables.

EOF analysis is also called principal component analysis or factor analysis.

What Does EOF Analysis do?

In brief, EOF analysis uses a set of orthogonal functions (EOFs) to represent a time series in the following way:

Z(x,y,t) is the original time series as a function of time (t) and space (x, y).

EOF(x, y) show the spatial structures (x, y) of the major factors that can account for the temporal variations of Z.

PC(t) is the principal component that tells you how the amplitude of each EOF varies with time.

Slide 4

An Example

Another View of the Rotation

Rotation of Coordinates

Suppose the Pacific SSTs are described by values at grid points: x₁, x₂, x₃, ...x_N. We know that the x_i’s are probably correlated with each other.

Now, we want to determine a new set of independent predictors z_i to describe the state of Pacific SST, which are linear combinations of x_i:

Mathematically, we are rotating the old set of variable (x) to a new set of variable (z) using a projection matrix (e):

Determine the Projection Coefficients

The EOF analysis asks that the projection coefficients are determined in such a way that:

     (1) z₁ explains the maximum possible amount of the variance of the x’s;

     (2) z₂ explains the maximum possible amount of the remaining variance

          of the x’s;

     (3) so forth for the remaining z’s.

With these requirements, it can be shown mathematically that the projection coefficient functions (e_ij) are the eigenvectors of the covariance matrix of x’s.

The fraction of the total variance explained by a particular eigenvector is equal to the ratio of that eigenvalue to the sum of all eigenvalues.

Slide 9

Covariance Matrix

The EOF analysis has to start from calculating the covariance matrix.

For our case, the state of the Pacific SST is described by values at model grid points X_i.

Let’s assume the observational network in the Pacific has 10 grids in latitudinal direction and 20 grids in longitudinal direction, then there are 10x20=200 grid points to describe the state of pacific SST. So we have 200 state variables:

                         X_m(t), m =1, 2, 3, 4, …, 200

In our case, there are monthly observations of SSTs over these 200 grid points from 1900 to 1998. So we have N (12*99=1188) observations at each Xm:

                           X_mn = X_m(t_n), m=1, 2, 3, 4, …., 200

                                              n=1, 2, 3, 4, ….., 1188

Covariance Matrix – cont.

The covariance between two state variables X_i and X_j is:

The covariance matrix is as following:

Eigenvectors of a Symmetric Matrix

Any symmetric matrix R can be decomposed in the following way through a diagonalization, or eigenanalysis:

Where E is the matrix with the eigenvectors e_i as its columns, and L is the matrix with the eigenvalues l_i, along its diagonal and zeros elsewhere.

The set of eigenvectors, e_i, and associated eigenvalues, l_i, represent a coordinate transformation into a coordinate space where the matrix R becomes diagonal.

Orthogonal Constrains

There are orthogonal constrains been build in in the EOF analysis:

The principal components (PCs) are orthogonal in time.

There are no simultaneous temporal correlation between any two principal components.

(2) The EOFs are orthogonal in space.

There are no spatial correlation between any two EOFs.

The second orthogonal constrain is removed in the rotated EOF analysis.

Mathematic Background

I don’t want to go through the mathematical details of EOF analysis. Only some basic concepts are described in the following few slids.

Through mathematic derivations, we can show that the empirical orthogonal functions (EOFs) of a time series Z(x, y, t) are the eigenvectors of the covarinace matrix of the time series.

The eigenvalues of the covariance matrix tells you the fraction of variance explained by each individual EOF.

Some Basic Matrix Operations

A two-dimensional data matrix X:

The transpose of this matrix is X^T:

The inner product of these two matrices:

Correlation Matrix

Sometime, we use the correlation matrix, in stead of the covariance matrix, for EOF analysis.

For the same time series, the EOFs obtained from the covariance matrix will be different from the EOFs obtained from the correlation matrix.

The decision to choose the covariance matrix or the correlation matrix depends on how we wish the variance at each grid points (X_i) are weighted.

In the case of the covariance matrix formulation, the elements of the state vector with larger variances will be weighted more heavily.

With the correlation matrix, all elements receive the same weight and only the structure and not the amplitude will influence the principal components.

Correlation Matrix – cont.

The correlation matrix should be used for the following two cases:

The state vector is a combination of things with different units.

(2) The variance of the state vector varies from point to point so much that this distorts the patterns in the data.

How to Get Principal Components?

If we want to get the principal component, we project a single eigenvector onto the data and get an amplitude of this eigenvector at each time, e^TX:

For example, the amplitude of EOF-1 at the first measurement time is calculated as the following:

Presentations of EOF – Variance Map

There are several ways to present EOFs. The simplest way is to plot the values of EOF itself. This presentation can not tell you how much the real amplitude this EOF represents.

One way to represent EOF’s amplitude is to take the time series of principal components for an EOF, normalize this time series to unit variance, and then regress it against the original data set.

This map has the shape of the EOF, but the amplitude actually corresponds to the amplitude in the real data with which this structure is associated.

If we have other variables, we can regress them all on the PC of one EOF and show the structure of several variables with the correct amplitude relationship, for example, SST and surface vector wind fields can both be regressed on PCs of SST.

Presentations of EOF – Correlation Map

Another way to present EOF is to correlate the principal component of an EOF with the original time series at each data point.

This way, present the EOF structure in a correlation map.

In this way, the correlation map tells you what are the co-varying part of the variable (for example, SST) in the spatial domain.

In this presentation, the EOF has no unit and is non-dimensional.

Using SVD to Get EOF&PC

We can use Singular Value Decomposition (SVD) to get EOFs, eigenvalues, and PC’s directly from the data matrix, without the need to calculate the covariance matrix from the data first.

If the data set is relatively small, this may be easier than computing the covariance matrices and doing the eigenanalysis of them.

If the sample size is large, it may be computationally more efficient to use the eigenvalue method.

What is SVD?

Any m by n matrix A can be factored into

The columns of U (m by m) are the EOFs

The columns of V (n by n) are the PCs.

The diagonal values of S are the eigenvalues represent the amplitudes of the EOFs, but not the variance explained by the EOF.

The square of the eigenvalue from the SVD is equal to the eigenvalue from the eigen analysis of the covariance matrix.

An Example – with SVD method

An Example – With Eigenanalysis

How Many EOFs Should We Retain?

There are no definite ways to decide this. Basically, we look at the eigenvalue spectrum and decide:

The 95% significance errors in the estimation of the eigenvalues is:

If the eigenvalues of adjacent EOF’s are closer together than this standard error, then it is unlikely that their particular structures are significant.

(2) Or we can just look at the slope of the eigenvalue spectrum.

We would look for a place in the eigenvalue spectrum where it levels off so that successive eigenvalues are indistinguishable. We would not consider any eigenvectors beyond this point as being special.

An Example

Rotated EOF

The orthogonal constrain on EOFs sometime cause the spatial structures of EOFS to have significant amplitudes all over the spatial domain.

      è We can not get localized EOF structures.

      è Therefore, we want to relax the spatial orthogonal constrain

           on EOFs (but still keep the temporal orthogonal constrain).

      è We apply the Rotated EOF analysis.

To perform the rotated EOF analysis, we still have to do the regular EOF first.

     è We then only keep a few EOF modes for the rotation.

     è We “rotated” these selected few EOFs to form new EOFs (factors).

          based on some criteria.

     è These criteria determine how “simple” the new factors are.

Criteria for the Rotation

Basically, the criteria of rotating EOFs is to measure the “simplicity” of the EOF structure.

Basically, simplicity of structure is supposed to occur when most of the elements of the eigenvector are either of order one (absolute value) or zero, but not in between.

There are two popular rotation criteria:

Quartimax Rotation

Varimax Rotation

Quartimax and Varimax Rotation

Ouartimax Rotation

      It seeks to rotate the original EOF matrix into a new EOF matrix for which the variance of squared elements of the eigenvectors is a maximum.

Varimax Rotation (more popular than the Quartimax rotation)

      It seeks to simplify the individual EOF factors.

     The criterion of simplicity of the complete factor matrix is defined as the maximization of the sum of the simplicities of the individual factors.

Reference For the Following Examples

The following few examples are from a recent paper published on Journal of Climate:

Example 1:
North Atlantic SST Variability

Example 2: Indian Ocean SST Variability

Example 3:
SLP Variability
(Arctic Oscillation)

Example 4:
Low-Dimensional Variability
(Variance Based)

Slide 35


	Empirical Orthogonal Function (EOF) analysis attempts to find a relatively small number of independent variables (predictors; factors) which convey as much of the original information as possible without redundancy.
	EOF analysis can be used to explore the structure of the variability within a data set in a objective way, and to analyze relationships within a set of variables.
	EOF analysis is also called principal component analysis or factor analysis.


	In brief, EOF analysis uses a set of orthogonal functions (EOFs) to represent a time series in the following way:



	Z(x,y,t) is the original time series as a function of time (t) and space (x, y).
	EOF(x, y) show the spatial structures (x, y) of the major factors that can account for the temporal variations of Z.
	PC(t) is the principal component that tells you how the amplitude of each EOF varies with time.


	Suppose the Pacific SSTs are described by values at grid points: x₁, x₂, x₃, ...x_N. We know that the x_i’s are probably correlated with each other.
	Now, we want to determine a new set of independent predictors z_i to describe the state of Pacific SST, which are linear combinations of x_i:



	Mathematically, we are rotating the old set of variable (x) to a new set of variable (z) using a projection matrix (e):


	The EOF analysis asks that the projection coefficients are determined in such a way that:
	(1) z₁ explains the maximum possible amount of the variance of the x’s;
	(2) z₂ explains the maximum possible amount of the remaining variance
	of the x’s;
	(3) so forth for the remaining z’s.

	With these requirements, it can be shown mathematically that the projection coefficient functions (e_ij) are the eigenvectors of the covariance matrix of x’s.
	The fraction of the total variance explained by a particular eigenvector is equal to the ratio of that eigenvalue to the sum of all eigenvalues.


	The EOF analysis has to start from calculating the covariance matrix.
	For our case, the state of the Pacific SST is described by values at model grid points X_i.
	Let’s assume the observational network in the Pacific has 10 grids in latitudinal direction and 20 grids in longitudinal direction, then there are 10x20=200 grid points to describe the state of pacific SST. So we have 200 state variables:
	X_m(t), m =1, 2, 3, 4, …, 200
	In our case, there are monthly observations of SSTs over these 200 grid points from 1900 to 1998. So we have N (12*99=1188) observations at each Xm:
	X_mn = X_m(t_n), m=1, 2, 3, 4, …., 200
	n=1, 2, 3, 4, ….., 1188


	The covariance between two state variables X_i and X_j is:



	The covariance matrix is as following:


	Any symmetric matrix R can be decomposed in the following way through a diagonalization, or eigenanalysis:


	Where E is the matrix with the eigenvectors e_i as its columns, and L is the matrix with the eigenvalues l_i, along its diagonal and zeros elsewhere.
	The set of eigenvectors, e_i, and associated eigenvalues, l_i, represent a coordinate transformation into a coordinate space where the matrix R becomes diagonal.


	There are orthogonal constrains been build in in the EOF analysis:
	The principal components (PCs) are orthogonal in time.
	There are no simultaneous temporal correlation between any two principal components.
	(2) The EOFs are orthogonal in space.
	There are no spatial correlation between any two EOFs.
	The second orthogonal constrain is removed in the rotated EOF analysis.


	I don’t want to go through the mathematical details of EOF analysis. Only some basic concepts are described in the following few slids.
	Through mathematic derivations, we can show that the empirical orthogonal functions (EOFs) of a time series Z(x, y, t) are the eigenvectors of the covarinace matrix of the time series.
	The eigenvalues of the covariance matrix tells you the fraction of variance explained by each individual EOF.


	A two-dimensional data matrix X:



	The transpose of this matrix is X^T:


	The inner product of these two matrices:


	Sometime, we use the correlation matrix, in stead of the covariance matrix, for EOF analysis.
	For the same time series, the EOFs obtained from the covariance matrix will be different from the EOFs obtained from the correlation matrix.
	The decision to choose the covariance matrix or the correlation matrix depends on how we wish the variance at each grid points (X_i) are weighted.
	In the case of the covariance matrix formulation, the elements of the state vector with larger variances will be weighted more heavily.
	With the correlation matrix, all elements receive the same weight and only the structure and not the amplitude will influence the principal components.


	The correlation matrix should be used for the following two cases:
	The state vector is a combination of things with different units.
	(2) The variance of the state vector varies from point to point so much that this distorts the patterns in the data.


	If we want to get the principal component, we project a single eigenvector onto the data and get an amplitude of this eigenvector at each time, e^TX:




	For example, the amplitude of EOF-1 at the first measurement time is calculated as the following:


	There are several ways to present EOFs. The simplest way is to plot the values of EOF itself. This presentation can not tell you how much the real amplitude this EOF represents.
	One way to represent EOF’s amplitude is to take the time series of principal components for an EOF, normalize this time series to unit variance, and then regress it against the original data set.
	This map has the shape of the EOF, but the amplitude actually corresponds to the amplitude in the real data with which this structure is associated.
	If we have other variables, we can regress them all on the PC of one EOF and show the structure of several variables with the correct amplitude relationship, for example, SST and surface vector wind fields can both be regressed on PCs of SST.


	Another way to present EOF is to correlate the principal component of an EOF with the original time series at each data point.
	This way, present the EOF structure in a correlation map.
	In this way, the correlation map tells you what are the co-varying part of the variable (for example, SST) in the spatial domain.
	In this presentation, the EOF has no unit and is non-dimensional.


	We can use Singular Value Decomposition (SVD) to get EOFs, eigenvalues, and PC’s directly from the data matrix, without the need to calculate the covariance matrix from the data first.
	If the data set is relatively small, this may be easier than computing the covariance matrices and doing the eigenanalysis of them.
	If the sample size is large, it may be computationally more efficient to use the eigenvalue method.


	Any m by n matrix A can be factored into



	The columns of U (m by m) are the EOFs
	The columns of V (n by n) are the PCs.
	The diagonal values of S are the eigenvalues represent the amplitudes of the EOFs, but not the variance explained by the EOF.
	The square of the eigenvalue from the SVD is equal to the eigenvalue from the eigen analysis of the covariance matrix.


	There are no definite ways to decide this. Basically, we look at the eigenvalue spectrum and decide:
	The 95% significance errors in the estimation of the eigenvalues is:


	If the eigenvalues of adjacent EOF’s are closer together than this standard error, then it is unlikely that their particular structures are significant.
	(2) Or we can just look at the slope of the eigenvalue spectrum.
	We would look for a place in the eigenvalue spectrum where it levels off so that successive eigenvalues are indistinguishable. We would not consider any eigenvectors beyond this point as being special.


	The orthogonal constrain on EOFs sometime cause the spatial structures of EOFS to have significant amplitudes all over the spatial domain.
	è We can not get localized EOF structures.
	è Therefore, we want to relax the spatial orthogonal constrain
	on EOFs (but still keep the temporal orthogonal constrain).
	è We apply the Rotated EOF analysis.

	To perform the rotated EOF analysis, we still have to do the regular EOF first.
	è We then only keep a few EOF modes for the rotation.
	è We “rotated” these selected few EOFs to form new EOFs (factors).
	based on some criteria.
	è These criteria determine how “simple” the new factors are.


	Basically, the criteria of rotating EOFs is to measure the “simplicity” of the EOF structure.
	Basically, simplicity of structure is supposed to occur when most of the elements of the eigenvector are either of order one (absolute value) or zero, but not in between.
	There are two popular rotation criteria:
	Quartimax Rotation
	Varimax Rotation


	Ouartimax Rotation
	It seeks to rotate the original EOF matrix into a new EOF matrix for which the variance of squared elements of the eigenvectors is a maximum.



	Varimax Rotation (more popular than the Quartimax rotation)
	It seeks to simplify the individual EOF factors.



	The criterion of simplicity of the complete factor matrix is defined as the maximization of the sum of the simplicities of the individual factors.


	The following few examples are from a recent paper published on Journal of Climate: