Finally arrive at reproducing kernel Hilbert space. https://nzer0.github.io/reproducing-kernel-hilbert-space.html
The above post introduces RKHS in Korean. It was helpful. I had struggled to understand some concepts in RKHS. What does mean Hilbert space in terms of feature expansion? (\(f:\mathcal{X} \to \mathbb{R}\), \(f \in \mathcal{H}_K\)) It was confusing the difference between \(f\) and \(f(x)\). \(f\) means the function in Hilbert space and \(f(x)\) is evaluation.
I thought that the function can be represented by the inner product of the basis of feature space \(K(\cdot,x)\) and coefficients \(f\), and the coefficients are vectors in feature space.
The reproducing property of Kernel is \(\langle f, K(\cdot,x)\rangle_{\mathcal{H}} = f(x)\). Thus \(K(\cdot,x) \in \mathcal{H}_K\). \(K(\cdot,x)\) is a \(x\) specified function in Hilbert space \(\mathcal{H}_K\) and an evaluator of the specific point x. This means the inner product of \(f\) and \(K_{x}\) is the value of \(f\) at point \(x\), \(f(x)\).
In a nutshell, kenel method is a different way of evaluating f in a specific point \(x\). Evaluating a function \(f\) at a point \(x\) is inner product of \(f\) and \(L_x\), where \(L_x \in \mathcal{H}_K\) is a evaluation functional which is a kernal function and linear \(K(\cdot, x)\). Reproducing property of \(\mathcal{H}_K\) can be achieved if all \(f \in \mathcal{H}\) has bounded evaluation functionals (\(L_x\)).
In least square methods, the parameters (\(\hat{\beta}\)) are determined by inner product of \(X\) \(\hat{\beta} = (X^{T}X)^{-1}X^{T}y\). In Kernel method, \(\hat{\beta}\) is determined \(\langle K(\cdot,x_i), K(\cdot,x_j), \rangle_{\mathcal{H}_K} = K(x_i, x_j)\). Each \(K(\cdot, x)\) is a parameter and a argument (variable like \(x\)).
Some subclass of the loss function and penalty functions can be generated by a positive definite kernel. A Kernel accepts two arguments and a Kernel function does one argument and the other argument becomes parameter. Reproducing Kernel Hilbert space is a function space with Kernal function space with the evaluation functional as a Kernel. The feature expansion into the RKHS can use the Kernel matrix instead of the inner product of each variable \(X^TX\).
The important concepts are Hilbert space, inner product, Kernel function, evaluation functional, feature expansion, Fourier transformation, Reisz representation theorem (dual space \(\mathcal{H}_{K}^*\) of Hibert space \(\mathcal{H}_K\))
