Functional data analysis

In the Hilbert space viewpoint, one considers an ${\displaystyle H}$-valued random element ${\displaystyle X}$, where ${\displaystyle H}$ is a separable Hilbert space such as the space of square-integrable functions ${\displaystyle L^{2}[0,1]}$. Under the integrability condition that ${\displaystyle \mathbb {E} \|X\|}$ is finite, one can define the mean of ${\displaystyle X}$ as the unique element ${\displaystyle \mu \in H}$ satisfying

${\displaystyle \mathbb {E} \langle X,h\rangle =\langle \mu ,h\rangle ,\qquad h\in H.}$

This formulation is the Pettis integral but the mean can also be defined as ${\displaystyle \mu =\mathbb {E} X}$ the Bochner sense. Under the integrability condition that ${\displaystyle \mathbb {E} \|X\|^{2}}$ is finite, the covariance operator of ${\displaystyle X}$ is a linear operator ${\displaystyle {\mathcal {C}}:H\to H}$ that is uniquely defined by the relation

${\displaystyle {\mathcal {C}}h=\mathbb {E} [\langle h,X-\mu \rangle (X-\mu )],\qquad h\in H,}$

or, in tensor form, ${\displaystyle {\mathcal {C}}=\mathbb {E} [(X-\mu )\otimes (X-\mu )]}$. The spectral theorem allows to decompose ${\displaystyle X}$ as the Karhunen-Loève decomposition

${\displaystyle X=\mu +\sum _{i=1}^{\infty }\langle X,\varphi _{i}\rangle \varphi _{i},}$

where ${\displaystyle \varphi _{i}}$ are eigenvectors of ${\displaystyle {\mathcal {S}}}$, corresponding to the nonnegative eigenvalues of ${\displaystyle {\mathcal {S}}}$, in a nonincreasing order. Truncating this infinite series to a finite order underpins functional principal component analysis.

The Hilbertian point of view is mathematically convenient, but abstract; the above considerations do not necessarily even view ${\displaystyle X}$ as a function at all, since common choices of ${\displaystyle H}$ like ${\displaystyle L^{2}[0,1]}$ and Sobolev spaces consist of equivalence classes, not functions. The stochastic process perspective views ${\displaystyle X}$ as a collection of random variables

${\displaystyle \{X(t)\}_{t\in [0,1]}}$

indexed by the unit interval (or more generally some compact metric space ${\displaystyle K}$). The mean and covariance functions are defined in a pointwise manner as

${\displaystyle \mu (t)=\mathbb {E} X(t),\qquad c(s,t)={\textrm {Cov}}[(X(s),X(t)],\qquad s,t\in [0,1]}$

(if ${\displaystyle \mathbb {E} [X(t)^{2}]<\infty }$ for all ${\displaystyle t\in [0,1]}$). We can hope to view ${\displaystyle X}$ as a random element on the Hilbert function space ${\displaystyle H=L^{2}[0,1]}$. However, additional conditions are required for such a pursuit to be fruitful, since if we let ${\displaystyle X(t)}$ be Gaussian white noise, i.e. ${\displaystyle X(t)}$ is standard Gaussian and independent from ${\displaystyle X(s)}$ for any ${\displaystyle s,t\in [0,1]}$, it is clear that we have no hope of viewing this as a square integrable function.

A convenient sufficient condition is mean square continuity, stipulating that ${\displaystyle \mu }$ and ${\displaystyle c}$ are continuous functions. In this case ${\displaystyle c}$ defines a covariance operator ${\displaystyle {\mathcal {C}}:H\to H}$ by

${\displaystyle ({\mathcal {C}}f)(t)=\int _{K}c(s,t)f(s)\,\mathrm {d} s.}$

The spectral theorem applies to ${\displaystyle {\mathcal {C}}}$, yielding eigenpairs ${\displaystyle (\lambda _{j},\varphi _{j})}$, so that in tensor product notation ${\displaystyle {\mathcal {C}}}$ writes

${\displaystyle {\mathcal {C}}=\sum _{j=1}^{\infty }\lambda _{j}\varphi _{j}\otimes \varphi _{j}.}$

Moreover, since ${\displaystyle {\mathcal {S}}f}$ is continuous for all ${\displaystyle f\in H}$, all the ${\displaystyle \varphi _{j}}$'s are continuous. Mercer's theorem then states that the covariance function ${\displaystyle c}$ admits an analogous decomposition

${\displaystyle \sup _{s,t\in [0,1]}\left|c(s,t)-\sum _{j=1}^{N}\lambda _{j}\varphi _{j}(s)\varphi _{j}(t)\right|\to 0,\qquad N\to \infty .}$

Finally, under the extra assumption that ${\displaystyle X}$ has continuous sample paths, namely that with probability one, the random function ${\displaystyle X:[0,1]\to \mathbb {R} }$ is continuous, the Karhunen-Loève expansion above holds for ${\displaystyle X}$ and the Hilbert space machinery can be subsequently applied. Continuity of sample paths can be shown using Kolmogorov continuity theorem.

A well-studied model for scalar-on-function regression is a generalisation of linear regression. Classical linear regression assumes that a scalar variable of interest ${\displaystyle Y}$ is related to a ${\displaystyle p}$-dimensional covariate vector ${\displaystyle X}$ through the equation

${\displaystyle Y=\langle X,\beta \rangle +\varepsilon =X_{1}\beta _{1}+\dots +X_{p}\beta _{p}+\varepsilon }$

for a ${\displaystyle p}$-dimensional vector of coefficients ${\displaystyle \beta }$ and a scalar noise variable ${\displaystyle \varepsilon }$, where ${\displaystyle \langle X,\beta \rangle }$ denotes the standard inner product on ${\displaystyle \mathbb {R} ^{p}}$. If we instead observe a functional variable ${\displaystyle X}$, which we assume is an element of the space of square-integrable functions on the unit interval, we can consider the same linear regression model as above using the ${\displaystyle L^{2}[0,1]}$ inner product. In other words, we consider the model

${\displaystyle Y=\int _{0}^{1}X(t)\beta (t)\,\mathrm {d} t+\varepsilon }$

for a square-integrable coefficient function ${\displaystyle \beta }$ and ${\displaystyle \varepsilon }$ as before (see Chapter 13[7]).

Analogously to the scalar-on-function regression model, we can consider a functional ${\displaystyle Y}$ and ${\displaystyle p}$-dimensional covariate vector ${\displaystyle X}$ and again draw inspiration from the usual linear regression model by modelling ${\displaystyle Y}$ as a linear combination of ${\displaystyle p}$ functions. In other words, we assume that the relationship between ${\displaystyle Y}$ and ${\displaystyle X}$ is

${\displaystyle Y(t)=X_{1}\beta _{1}(t)+\dots X_{p}\beta _{p}(t)+\varepsilon (t)}$

for functions ${\displaystyle \beta _{1},\dots ,\beta _{p}}$ and functional error term ${\displaystyle \varepsilon }$.

Both of the previous regression models can be viewed as instances of a general linear model between Hilbert spaces. Assuming that ${\displaystyle X}$ and ${\displaystyle Y}$ are elements of Hilbert spaces ${\displaystyle H_{X}}$ and ${\displaystyle H_{Y}}$, the Hilbertian linear model assumes that

${\displaystyle Y={\mathcal {S}}X+\varepsilon }$

for a Hilbert-Schmidt operator ${\displaystyle {\mathcal {S}}:H_{X}\to H_{Y}}$ and a noise variable ${\displaystyle \varepsilon }$ taking values in ${\displaystyle H_{Y}}$. If ${\displaystyle H_{X}=L^{2}[0,1]}$ and ${\displaystyle H_{Y}=\mathbb {R} }$, we get the scalar-on-function regression model above. Similarly, if ${\displaystyle H_{X}=\mathbb {R} ^{p}}$ and ${\displaystyle H_{Y}=L^{2}[0,1]}$ then we get the function-on-scalar regression model above. If we let ${\displaystyle H_{X}=H_{Y}=L^{2}[0,1]}$, we get the function-on-function linear regression model which can equivalently be written

${\displaystyle Y(t)=\int _{0}^{1}X(s)\beta (s,t)\,\mathrm {d} s+\varepsilon (t)}$

for a square-integrable coefficient function ${\displaystyle \beta }$ and functional noise variable ${\displaystyle \varepsilon }$.

While the presentation of the models above assumes fully-observed functions, software is available for fitting the models with discretely-observed functions in software such as R. Packages for R include refund [11] and FDboost [12] that utilise reformulations of the functional models as generalized additive models and boosted models, respectively.