Gaussian kernel matrix can be factorized into (ΦX)HΦX=XHΦHΦX=XHX, where Φ is Gaussian kernel basis matrix and X is coefficients matrix of reproducing kernel Hilbert space K(⋅,x)∈HK https://www.jkangpathology.com/post/reproducing-kernel-hilbert-space/.
A matrix is a system. A system takes input and gives output. A matrix is a linear system. Differentiation and Integration are linear systems. Fourier transformation matches input basis and operator (differentiation) basis.
The Fourier series represents a periodic function as a descrete vectors. The Fourier transformation turns a time domain non-periodic function into a frequency domain continuous function. The Fourier series and transformation change a single time base t into infinite frequency basis einx or eiwx. The function on infinite basis domain can be represented by a vector or a function of basis domain vn or f(w). This is a coefficients of Fourier series or Fourier transformation.
Convolution is a vector operation on two vectors.
Convolutionc∗d=d∗c(c∗d)n=Σi+jcidj=Σicidn−i. This is multiplying polynomials. The parameters of multiplied polynomial become convolution of two polynomials. Fourier transformation expands x base to infinite exponential basis eiwk. The multiplication on x (time) space becomes convolutionn on k (frequency) space.
If time space is periodic, its Fourier transformation is discrete i.
The optimization problem have two components that are objective function f0:Rn→R and the constraints. The objective function and constraints keep in check each other and make balance at saddle point i.e. optimal point. The dual (Lagrange) problem of the optimal problem also solve the optimization problem by making low boundary.
The dual problem can be explained as a conjugate function f∗=sup(xTy−f(x)).
The purpose of approximation is finding optimal point x∗ i.e. ∇F(x∗)=0. We need a step/search direction Δx and step size t. Taylor approximation has polynomial arguments that is a step and parameters of derivatives at the start point. The first degree of Taylor approximation has one adding term from start point (x0,F(x0)). The adding term ∇F(x)Δx is consistent with a parameter (gradient ∇F(x)) and a argument (step Δx).
Bases are the central idea of linear algebra. An invertable square matrix has eigenvectors. A symetric matrix has orthogonal eigenvectors with non-negative eigenvalues, i.e. positive semidefinite. A matrix has two types of singular vectors, left and right signular vectors, A=UΣVT.
When we think the matrix A is data points of rows A=UΣVT like data table, The right singular vectors V build bases, the sigular values Σ are magnitude of the bases and the left singular values U becomes new data points on new bases.
This is a note for part III of Linear Algebra and learning from data, Gilbert Strang
The main themes are sparsity (Low rank), Information theory (compression), and of course linear transformation.
A full rank matrix is inefficient. Finding low lank matrix which is close with original matrix can save computation.
The rank one matrix uvT is a unit of a matrix. The full rank matrix can be decomposed by sum of rank one matrices i.
The meaning of AT
Steady state equilibrium Graph Laplacian matrix ATCA Differential equation and Laplacian matrix Derivative is a graph without branch. Row space and column space are dual. A and AT are dual. ref) Linear algebra and learning from data, Part IV, Gilbert Strang
This is summary of Boyd convex optimization. Steepest descent method is a convex optimization algorithm. The normalized steepest descent direction xnsd is a vector of unit ball of a norm that extends in the direction −∇f(x). The inner product of xnsd and −∇f(x) is maximized. The first order Taylor approximation of f(x+v)=f(x)+∇f(x)Tv is most efficient when v=xnsd.
The xnsd is unnormalized into xsd.