[@Red-Horse2009]

Two types of uncertainty: inherent uncertainty; reducible uncertainty (ignorance).

**Inherent uncertainty** is uncertainty that cannot be reduced by gathering more information.
The origins of this class of uncertainty typically occur at scales
that are smaller than either the model or the observations in a given study.
Other names for this uncertainty are *irreducible uncertainty*, *aleatoric uncertainty*,
and *variability*.

**Ignorance** is typically associated with limitations in models or available experimental data.

Two approaches to probability:

- Statistical descriptions of probabilistic entity: distribution, moments.
- Function analytic approach: random variables

For a probability space $(\Omega, \Sigma, P)$ and an arbitrary measurable space $(H, \mathcal{H})$,
**random variable** $X: \Omega \to H$ is a $(\Sigma, \mathcal{H})$-measuralbe function.
It has induced distribution $\eta_X (X \in A) = P(X^{-1}(A)), \quad A \in \mathcal{H}$.

$H$ be a metric space.

- A Borel sigma-algebra on $H$ can be induced by its metric, denote as $\mathcal{H}$.
- Denote the space of bounded continuous functions from $H$ to $\mathbb{R}$ as $C(H)$. It is a Banach space under the supremum norm. If $H$ is compact then any positive linear functional on $C(H)$ can be uniquely identified with the expectation operator relative to some probability measure on $H$.
- Denote the space of all probability measures on $H$ as $\mathcal{M}(H)$. It can be metrized as a separable metric space if and only if $H$ is a separable metric space.

**Topological dual** is the space of continuous linear functionals
defined on a topological vector space.

**Weak topology**:
the topological dual of a topological vector space induce a topology in the original space
where neighborhoods are defined in terms of their images under the duality pairing.
It is a subset of the original topology, i.e. a weaker topology.

duality pairing

Reproducing kernel Hilbert spaces (RKHS): The operator $S_X^{-1}$ is a reproducing kernel in $H_X$.

A countably-normed space is a topological vector space over $C$ with topology given by
a countable family of norms (i.e. intersection of all norm-induce neighborhoods).
A **countably-Hilbert space** is a countably-normed space complete with respect to its topology,
and each norm is induced from an inner product.

The compatibility requirement on the norms associated with a countably-Hilbert space implies an ordering in these norms.

Nuclear operator

Both the covariance operator and the characteristic functional of a stochastic process are nuclear operators.

Nuclear space

canonical embedding

Kolmogorov consistency requirements

When $H$ is a Hilbert space, the covariance opertator $S_X$ is Hilbert–Schmidt, self-adjoint, positive, and completely continuous, and therefore has a finite or countable set of eigenvalues and corresponding orthonormal eigenfunctions ${ (\lambda_n, \phi_n) }$.

The RKHS structure ensures that the evaluation functional in this space is continuous.

Thus we have Karhunen–Loeve expansion for such defined RVs.

- Often times people only use limited description of a random entity, such as mean and variance in the case of random variable, and mean process and correlation function in the case of random process. If only this extent of information is needed, why do we need to approximate r.v. with PCEs?