Skip to main content

Introduction to Linear Algebra: Projection and a Simple Kalman Filter (2)

Linear algebra way

In the linear algebra way, the best x is a projection of right hand side onto the column space of the matrix. Because the solution is only possible in the A's column space (means the solution is only represented by the A's column vector's linear combination), the best solution is the projection of b onto the column space. Figure 1 shows the geometric interpretation. The matrix A and b are:

Here A is a vector, so the projected best x is:

The result is the same to the calculus way. This is also the average. (If you are not familiar with projection matrix, see the Figure 1. In the Figure, e is perpendicular with A, i.e., A^T e = 0. You can derive it from here.)
Figure 1. Project b to A's column space

My interest is why these are the same. Minimization of quadric is a calculus (analysis) problem, and projection is more a geometric problem. Is this just a coincidence? Of course not. They have the same purpose.

As in Figure 1, projection is the minimal distance between b and A's column space. The distance is square root of squared sum in the Euclidean space (Pythagoras's theorem). Minimizing it is the same to the calculus way, though, the idea and how to compute it looks different.

Once I wrote the following, I like Keigo Higashino's books. One of his book, ``Yougisha X no Kenshin'', describes about deceiving a problem makes wrong conclusion. The example is mathematics, if you deceive someone as this is a calculus problem, but actually it is a geometry problem, one can not reach the correct answer. A clever guy deceive the police by putting an wrong answer, that's a fun detective story. Higashino's book usually describes science quite well, though, I have a question on this point. I found interesting that when the problem setting is the same, even different approcaches can conclude the same result in mathematics. I feel this more and more when I study mathematics more. I am surprised when the different looking problems are originated from the same idea. It looks like climbing a mountain. When I was at the sea level, I can only see some of the views from specific directions only, these views look different, however, once I see the overview from the mountain, they are just the same view from different positions. Studying mathematics is like to have a map. Some of the roads seems not connected, but if I have a map, I can see actually they are connected. I think all the roads are somewhat connected. If I could not see, that means I need to study more.

Here we found the closest vector from a vector by

  • calculus: minimize the error
  • linear algebra: projection.

Next time, I would like to talk about Kalman filter. Kalman filter estimates a near future based on observed data that have some noise. The basic idea to predict near future is the same as mentioned in this blog. But I recently happen to know it has one interesting aspect and I would like to explain that.

Comments

Popular posts from this blog

Why A^{T}A is invertible? (2) Linear Algebra

Why A^{T}A has the inverse Let me explain why A^{T}A has the inverse, if the columns of A are independent. First, if a matrix is n by n, and all the columns are independent, then this is a square full rank matrix. Therefore, there is the inverse. So, the problem is when A is a m by n, rectangle matrix.  Strang's explanation is based on null space. Null space and column space are the fundamental of the linear algebra. This explanation is simple and clear. However, when I was a University student, I did not recall the explanation of the null space in my linear algebra class. Maybe I was careless. I regret that... Explanation based on null space This explanation is based on Strang's book. Column space and null space are the main characters. Let's start with this explanation. Assume  x  where x is in the null space of A .  The matrices ( A^{T} A ) and A share the null space as the following: This means, if x is in the null space of A , x is also in the null spa

Gauss's quote for positive, negative, and imaginary number

Recently I watched the following great videos about imaginary numbers by Welch Labs. https://youtu.be/T647CGsuOVU?list=PLiaHhY2iBX9g6KIvZ_703G3KJXapKkNaF I like this article about naming of math by Kalid Azad. https://betterexplained.com/articles/learning-tip-idea-name/ Both articles mentioned about Gauss, who suggested to use other names of positive, negative, and imaginary numbers. Gauss wrote these names are wrong and that is one of the reason people didn't get why negative times negative is positive, or, pure positive imaginary times pure positive imaginary is negative real number. I made a few videos about explaining why -1 * -1 = +1, too. Explanation: why -1 * -1 = +1 by pattern https://youtu.be/uD7JRdAzKP8 Explanation: why -1 * -1 = +1 by climbing a mountain https://youtu.be/uD7JRdAzKP8 But actually Gauss's insight is much powerful. The original is in the Gauß, Werke, Bd. 2, S. 178 . Hätte man +1, -1, √-1) nicht positiv, negative, imaginäre (oder gar um

Why parallelogram area is |ad-bc|?

Here is my question. The area of parallelogram is the difference of these two rectangles (red rectangle - blue rectangle). This is not intuitive for me. If you also think it is not so intuitive, you might interested in my slides. I try to explain this for hight school students. Slides:  A bit intuitive (for me) explanation of area of parallelogram  (to my site, external link) .