Visualizing Principal Component Analysis
Altair EmployeePCA is very common technique in machine learning. Conceptualizing the method with images helps add insight into this ubiquitous method.
I decided to treat myself by picking this week’s subject from a list of my favorite things. Video games or baseball? No. To the surprise of no one who knows me, I choose eigenvalue problems, and specifically how they relate the Ordination tab and data mining in Altair HyperStudy.
Imagine a coordinate plot that has data scattered around the 45 degree axis. We can describe this using two coordinates, x and y. Now imagine performing a coordinate transformation into an alternative set of coordinates (x1,y1) indicated in red.
In this new coordinate system, most of the data’s variance is explained entirely by the coordinate x1 and there is little variation in y1. This means we can explain most of what is happening with a reduced set of coordinates. This process becomes much harder to visualize in multiple dimensions, but the process of finding the new coordinates is only an eigenvalue problem. This is the exact same thing that happens in many modal dynamics engineering applications, for example – you only retain a subset of eigenvectors that sufficiently represent the original problem.
I know you may be thinking: “This is really boring, why should I care?”. The magic starts to happen when you redraw the original axis in the new “principal coordinate” systems; that is why this type of thing is called Principal Component Analysis (PCA). Let’s look at the same image with a projection of “unit size” x and y axes (shown in blue) onto the new x1 axis (the red one). In this case they are the same (as shown in green). Two projected coordinate lines that align in direction indicate a positive correlation, lines in opposite directions are negatively correlated, and the length of the line indicates how strong the relationship is. The original data is highly correlated so the two projections are nearly identical.
In more dimensions, these relationships might be harder to see, but take a look at the image below from the data coming from the deflection of a cantilevered i-beam. These first two coordinates account for nearly 75% of the variance, and we can see the displacement and primary moment of inertia Iy are strongly negatively correlated, while the Total Height is reasonably strongly correlated to the interia, Iy. This all makes sense, given that it is an i-beam.
PCA has many more machine learning related use cases than this visualization of multi-dimensional correlations, for example feature engineering and dimension reduction. Regardless, data mining with PCA lends itself to visual insights into its inner workings, which removes some of the abstraction, and helps us all understand it better.