Reproducing Kernel Hilbert Space

Finally arrive at reproducing kernel Hilbert space. https://nzer0.github.io/reproducing-kernel-hilbert-space.html

The above post introduces RKHS in Korean. It was helpful. I had struggled to understand some concepts in RKHS. What does mean Hilbert space in terms of feature expansion? ($f:X\to R$, $f\in H_K$) It was confusing the difference between $f$ and $f\left(x\right)$. $f$ means the function in Hilbert space and $f\left(x\right)$ is evaluation.

I thought that the function can be represented by the inner product of the basis of feature space $K(\cdot ,x)$ and coefficients $f$, and the coefficients are vectors in feature space.

The reproducing property of Kernel is $⟨f,K(\cdot ,x){⟩}_H=f\left(x\right)$. Thus $K(\cdot ,x)\in H_K$. $K(\cdot ,x)$ is a $x$ specified function in Hilbert space $H_K$ and an evaluator of the specific point x. This means the inner product of $f$ and $K_x$ is the value of $f$ at point $x$, $f\left(x\right)$.

In a nutshell, kenel method is a different way of evaluating f in a specific point $x$. Evaluating a function $f$ at a point $x$ is inner product of $f$ and $L_x$, where $L_x\in H_K$ is a evaluation functional which is a kernal function and linear $K(\cdot ,x)$. Reproducing property of $H_K$ can be achieved if all $f\in H$ has bounded evaluation functionals ($L_x$).

In least square methods, the parameters ($\hat{\beta }$) are determined by inner product of $X$ $\hat{\beta }=(X^{T}X{)}^{-1}{X}^{T}y$. In Kernel method, $\hat{\beta }$ is determined $⟨K(\cdot ,x_i),K(\cdot ,x_j),{⟩}_{H_K}=K(x_i,x_j)$. Each $K(\cdot ,x)$ is a parameter and a argument (variable like $x$).

Some subclass of the loss function and penalty functions can be generated by a positive definite kernel. A Kernel accepts two arguments and a Kernel function does one argument and the other argument becomes parameter. Reproducing Kernel Hilbert space is a function space with Kernal function space with the evaluation functional as a Kernel. The feature expansion into the RKHS can use the Kernel matrix instead of the inner product of each variable $X^{T}X$.

The important concepts are Hilbert space, inner product, Kernel function, evaluation functional, feature expansion, Fourier transformation, Reisz representation theorem (dual space $H_K^{\ast }$ of Hibert space $H_K$)