Skip to main content

Introduction to Linear Algebra: Projection and a Simple Kalman Filter (1)

Introduction about linear algebra, calculus, and a simple Kalman filter

I found interesting that the relationship among linear algebra, calculus, and a simple Kalman filter. So I will make a memo about this.

Problem

Given a simple observed data sequence (x_i), we can assume these are equally plausible and we can apply the least square method. In this simple setting, the outcome is just an average. You can find this topic in Gilbert Strang's an Introduction to Linear Algebra, chapter 4.2.

For example, the observed data is [70 80 120]^T, if these are equally plausible,

 x = 70
 x = 80
 x = 120.

These euqations are strange. They look like, x is 70 and 80 and 120, simultaneously. Here, these are observed data, so they have some errors. But actually we observe the same data. The motivation is to find the best possible data we could  have from the observations.

Therefore, the system is:
There is no such x that satisfies the system. But, let us think about what is the best x.

Calculus way

We could use an idea from calculus to find the best x. The idea is: the observation has errors and what kind of x can minimize the error. This is called Gauss's least square method. First, we compute the squared error (E). This is also in the Strang's book.

 E = (x-70)^2 + (x-80)^2 + (x-120)

E is x^2's equation, therefore this is parabola. That means we can find a minimal point. Such point has 0 tangent, therefore, we could find it with
Let's compute this.

You can see this is actually an average. The best x is  an average, this fits my intuition. However, I did not think about why average is the best in what kind of sense. In this case, variance is relatively small, then, my intuition works. However, if the variance becomes larger, my intuition stops working. I once wrote about this in my blog ``A 6σ Woman.'' Average is the best here in the least square sense.

Next, let's find the best x with linear algebra way.

Comments

Popular posts from this blog

Why A^{T}A is invertible? (2) Linear Algebra

Why A^{T}A has the inverse Let me explain why A^{T}A has the inverse, if the columns of A are independent. First, if a matrix is n by n, and all the columns are independent, then this is a square full rank matrix. Therefore, there is the inverse. So, the problem is when A is a m by n, rectangle matrix.  Strang's explanation is based on null space. Null space and column space are the fundamental of the linear algebra. This explanation is simple and clear. However, when I was a University student, I did not recall the explanation of the null space in my linear algebra class. Maybe I was careless. I regret that... Explanation based on null space This explanation is based on Strang's book. Column space and null space are the main characters. Let's start with this explanation. Assume  x  where x is in the null space of A .  The matrices ( A^{T} A ) and A share the null space as the following: This means, if x is in the null space of A , x is also in the null spa

Gauss's quote for positive, negative, and imaginary number

Recently I watched the following great videos about imaginary numbers by Welch Labs. https://youtu.be/T647CGsuOVU?list=PLiaHhY2iBX9g6KIvZ_703G3KJXapKkNaF I like this article about naming of math by Kalid Azad. https://betterexplained.com/articles/learning-tip-idea-name/ Both articles mentioned about Gauss, who suggested to use other names of positive, negative, and imaginary numbers. Gauss wrote these names are wrong and that is one of the reason people didn't get why negative times negative is positive, or, pure positive imaginary times pure positive imaginary is negative real number. I made a few videos about explaining why -1 * -1 = +1, too. Explanation: why -1 * -1 = +1 by pattern https://youtu.be/uD7JRdAzKP8 Explanation: why -1 * -1 = +1 by climbing a mountain https://youtu.be/uD7JRdAzKP8 But actually Gauss's insight is much powerful. The original is in the Gauß, Werke, Bd. 2, S. 178 . Hätte man +1, -1, √-1) nicht positiv, negative, imaginäre (oder gar um

Why parallelogram area is |ad-bc|?

Here is my question. The area of parallelogram is the difference of these two rectangles (red rectangle - blue rectangle). This is not intuitive for me. If you also think it is not so intuitive, you might interested in my slides. I try to explain this for hight school students. Slides:  A bit intuitive (for me) explanation of area of parallelogram  (to my site, external link) .