Introduction of the Enhanced Envelope Estimator

The enhanced envelope estimator improves the prediction accuracy of multivariate linear regression by incorporating the concept of envelopes. This method is useful in both low- and high-dimensional scenarios, regardless of whether the number of predictors $p$ is smaller or larger than the number of observations $n$.

This overview first introduces the envelope model and then describes the enhanced envelope estimator.

Review of the Envelope Model

The envelope model builds upon the classical multivariate linear regression: $$y = Bx + \epsilon, ~ \epsilon \sim N(0,\Sigma_X),$$ where:

• $y\in\mathbb R^r$ is the vector of response variables,

• $x\in\mathbb R^p$ is the vector of predictor variables,

• $B\in\mathbb R^{r\times p}$ is the unknown coefficient matrix, and

• $\epsilon$ represents the error vector.

The envelope model improves this basic framework by identifying a subspace of the response vector that is unaffected by changes in the predictor vector. Specifically, it finds a subspace, called the envelope subspace, that separates the “material” variation (the part that is important for identifying the relationship of $y$ with $x$) from the “immaterial” variation (the part that does not affect this relationship). This separation reduces noise, leading to more accurate and efficient estimation and prediction.

By incorporating the envelope concept, the multivariate linear model can be reformulated as follows, which represents the envelope model: $$y = \Gamma \eta x + \epsilon, ~ \epsilon \sim N(0,\Gamma\Omega\Gamma^T + \Gamma_0\Omega_0\Gamma_0^T),$$ where:

• $\Gamma \in \mathbb{R}^{r \times u}$ is a semi-orthogonal basis matrix for the envelope subspace,

• $\Gamma_0 \in \mathbb{R}^{r \times (r-u)}$ is a basis matrix for the orthogonal complement of the envelope subspace,

• $\eta \in \mathbb{R}^{u \times p}$ is a reduced coefficient matrix, and

• $\Omega \in \mathbb R^{u\times u}$ and $\Omega_0 \in \mathbb R^{(r-u)\times (r-u)}$ are positive definite matrices representing the variances within and outside the envelope subspace.

Enhanced Envelope Estimator

The enhanced envelope estimator is a method for estimating parameters within the envelope model to improve prediction performance in both low- and high-dimensional settings. It incorporates a specialized regularization term during the estimation process, which helps enhance the model’s predictions by controlling overfitting.

The enhanced envelope estimator is obtained by maximizing a penalized likelihood function: $$\arg\max \left\{ \mathcal L(\eta,\Gamma, \Gamma_0, \Omega, \Omega_0) - \frac{n\lambda}{2} \rho(\eta,\Omega)\right\},$$ where:

• $\mathcal L(\eta,\Gamma, \Gamma_0, \Omega, \Omega_0)$ is the log-likelihood function of the envelope model,

• $\lambda>0$ is a regularization parameter, and

• $\rho$ is the regularization term defined as: $$\rho(\eta, \Omega) = \text{tr} (\eta^T\Omega^{-1}\eta).$$

This regularization term penalizes large values in the coefficient matrix $\eta$ adjusted by $\Omega^{-1/2}$, thereby improving model stability and reducing overfitting.

By allowing flexibility in selecting the regularization parameter $\lambda$, the enhanced envelope estimator can achieve better prediction performance than the original envelope estimator. Moreover, this regularization enables the model to handle cases where the number of predictors $p$ exceeds the number of observations $n$. The original envelope estimator cannot be applied directly in such high-dimensional situations, whereas the enhanced version, with its regularization, can effectively manage these cases.