7  Regression and Classification

Chapter Aim

Chapter 06 treated testing as a decision problem: define a null model, compute a statistic, obtain a p-value, and correct for multiplicity. Chapter 07 keeps the same design-based thinking but changes the emphasis. Regression models describe how an outcome changes with covariates, how a feature changes with an experimental condition, and how a class label can be predicted from high-dimensional measurements.

There are two recurring directions. In feature-wise regression, each molecular feature is treated as a response and the experimental design is the predictor:

\[\begin{equation} y_j = x_{\cdot j} \in\mathbb{R}^{n} \qquad \text{or} \qquad y_j = c_{\cdot j} \in\mathbb{N}_0^{n}. \label{eq:ch07-feature-response} \end{equation}\]

The covariates, treatments, batches, and experimental factors are collected in the design matrix

\[\begin{equation} Z\in\mathbb{R}^{n\times r}. \label{eq:ch07-design-matrix} \end{equation}\]

In supervised regression or classification, the high-dimensional feature matrix is the predictor and an external response is the outcome:

\[\begin{equation} X\in\mathbb{R}^{n\times p}, \qquad y\in\mathbb{R}^{n} \quad \text{or} \quad a\in\{0,1\}^{n}. \label{eq:ch07-supervised-objects} \end{equation}\]

Here \(y\) is a continuous response, such as drug sensitivity or disease score, and \(a\) is a binary class label, such as case/control status. The same matrix \(X\) can therefore appear in two roles. In differential expression, columns of \(X\) or \(C\) are responses and \(Z\) explains them. In prediction, rows of \(X\) are predictor vectors and \(y\) or \(a\) is what we want to predict.

TipNotation convention

Notation follows the shared convention in the notation chapter. Vectors are columns by default, row vectors are written with transposes, and dimensions should be checked whenever matrices are multiplied, transformed, modeled, or decomposed.

NoteMain external references

For count regression, generalized linear models, DESeq2, contrasts, and multifactor designs, the main book reference is MSMB High-Throughput Count Data, especially the sections on linear models (MSMB Count Data Linear Models), generalized linear models (MSMB Count Data GLM), DESeq2 GLMs (MSMB Count Data GLM DESeq2), and two-factor designs (MSMB Count Data Two-Factor Analysis). For prediction, classification, objective functions, penalization, and cross-validation, see MSMB Supervised Learning, especially MSMB Supervised Data, MSMB Supervised Objective Functions, MSMB Supervised Penalization, and MSMB Supervised Cross-Validation. The corresponding computations for DESeq2-style testing, high-dimensional regression, lasso with glmnet, stability selection, multi-split inference, and ridge regression are practiced in Exercise Sheet 05 (Exercise Sheet 05 2026).

Learning Goals

After this chapter, students should be able to write a feature-wise linear model with a design matrix, interpret regression coefficients and contrasts, distinguish ordinary least squares from generalized linear models, formulate a negative-binomial regression model for count data, explain the offset in DESeq2-like models, derive Wald statistics for contrasts, distinguish regression from classification, write logistic regression as a Bernoulli GLM, explain why regularization is needed when \(p\) is large relative to \(n\), formulate ridge and lasso objectives, understand sparse logistic regression, and explain why log-contrast regression is a natural model for compositional predictors.

7.1 From Testing To Regression

The two-sample t-test from Chapter 06 is already a regression model in disguise. Suppose observations belong to two groups, \(g_i\in\{0,1\}\), and \(y_j=x_{\cdot j}\) is a transformed continuous feature. A two-group model can be written as

\[\begin{equation} y_{ij} = \beta_{0j} + g_i\beta_{1j} + \varepsilon_{ij}, \qquad \varepsilon_{ij}\sim\mathcal{N}(0,\sigma_j^2). \label{eq:ch07-two-group-regression} \end{equation}\]

The coefficient \(\beta_{0j}\) is the expected value in the reference group, and \(\beta_{1j}\) is the group difference for feature \(j\):

\[\begin{equation} \mathbb{E}(y_{ij}\mid g_i=0)=\beta_{0j}, \qquad \mathbb{E}(y_{ij}\mid g_i=1)=\beta_{0j}+\beta_{1j}. \label{eq:ch07-two-group-means} \end{equation}\]

Testing no group effect is therefore the regression hypothesis

\[\begin{equation} H_{0j}:\beta_{1j}=0. \label{eq:ch07-two-group-null} \end{equation}\]

This view matters because it immediately generalizes. Once the problem is written as a regression model, adding a batch effect, a sex effect, a continuous age covariate, a pairing variable, or an interaction is no longer a different testing problem. It is a different design matrix.

7.2 Feature-Wise Linear Models

For transformed continuous values, a feature-wise linear model uses

\[\begin{equation} y_j = Z\beta_j+\varepsilon_j, \qquad \varepsilon_j\sim\mathcal{N}(0,\sigma_j^2I_n). \label{eq:ch07-feature-wise-linear-model} \end{equation}\]

Here \(y_j\in\mathbb{R}^{n}\) is feature \(j\) across observations, \(Z\in\mathbb{R}^{n\times r}\) is the design matrix, \(\beta_j\in\mathbb{R}^{r}\) contains feature-specific coefficients, and \(\varepsilon_j\in\mathbb{R}^{n}\) is residual noise. The fitted mean vector is

\[\begin{equation} \mu_j = \mathbb{E}(y_j\mid Z) = Z\beta_j. \label{eq:ch07-linear-model-mean} \end{equation}\]

Ordinary least squares estimates \(\beta_j\) by minimizing the residual sum of squares:

\[\begin{equation} \hat{\beta}_j = \arg\min_{\beta\in\mathbb{R}^{r}} \left\| y_j-Z\beta \right\|_2^2. \label{eq:ch07-ols-objective} \end{equation}\]

If \(Z^\top Z\) is invertible, the solution is

\[\begin{equation} \hat{\beta}_j = (Z^\top Z)^{-1}Z^\top y_j. \label{eq:ch07-ols-solution} \end{equation}\]

This equation is not only a computational formula. It says that the coefficient estimates are determined by the geometry of the design matrix. If two columns of \(Z\) are nearly collinear, the corresponding effects are hard to separate. If a treatment and a batch are perfectly confounded, the design matrix loses rank:

\[\begin{equation} \operatorname{rank}(Z)<r. \label{eq:ch07-rank-deficient-design} \end{equation}\]

In that case, some coefficients are not identifiable. Regression does not repair a bad design; it makes the consequences of the design explicit.

The residual vector and residual variance estimate are

\[\begin{equation} \hat{\varepsilon}_j = y_j-Z\hat{\beta}_j, \qquad \hat{\sigma}_j^2 = \frac{\|\hat{\varepsilon}_j\|_2^2}{n-r}. \label{eq:ch07-residual-variance} \end{equation}\]

Under the Gaussian linear model, the covariance matrix of the coefficient estimator is

\[\begin{equation} \operatorname{Cov}(\hat{\beta}_j\mid Z) = \sigma_j^2 (Z^\top Z)^{-1}. \label{eq:ch07-ols-covariance} \end{equation}\]

In high-throughput biology, Equation \(\ref{eq:ch07-ols-covariance}\) also explains why small sample sizes are painful. With few observations and several design columns, the coefficient uncertainty can be large even when the fitted effect looks substantial.

7.3 Contrasts

A regression coefficient is interpreted relative to the design matrix. For example, suppose an experiment has two binary factors: siRNA treatment and drug treatment. Let

\[\begin{equation} z_{i1} = \mathbf{1}\{\text{observation }i\text{ receives siRNA}\}, \qquad z_{i2} = \mathbf{1}\{\text{observation }i\text{ receives drug}\}. \label{eq:ch07-binary-factor-coding} \end{equation}\]

With an intercept, two main effects, and an interaction, the design row is

\[\begin{equation} z_i^\top = \begin{pmatrix} 1 & z_{i1} & z_{i2} & z_{i1}z_{i2} \end{pmatrix}, \qquad Z\in\mathbb{R}^{n\times 4}. \label{eq:ch07-two-factor-design-row} \end{equation}\]

For feature \(j\), write

\[\begin{equation} \beta_j = \begin{pmatrix} \beta_{0j}\\ \beta_{\mathrm{siRNA},j}\\ \beta_{\mathrm{drug},j}\\ \beta_{\mathrm{siRNA:drug},j} \end{pmatrix}. \label{eq:ch07-two-factor-coefficients} \end{equation}\]

The expected transformed value is then

\[\begin{equation} \mathbb{E}(y_{ij}\mid z_i) = \beta_{0j} + z_{i1}\beta_{\mathrm{siRNA},j} + z_{i2}\beta_{\mathrm{drug},j} + z_{i1}z_{i2}\beta_{\mathrm{siRNA:drug},j}. \label{eq:ch07-two-factor-mean} \end{equation}\]

The interaction coefficient measures departure from additivity on the model scale. It is not the combined treatment effect itself. The combined treatment versus control effect is

\[\begin{equation} \beta_{\mathrm{siRNA},j} + \beta_{\mathrm{drug},j} + \beta_{\mathrm{siRNA:drug},j}. \label{eq:ch07-combined-treatment-effect} \end{equation}\]

Such quantities are contrasts. A contrast vector \(h\in\mathbb{R}^{r}\) defines the estimand

\[\begin{equation} \theta_j = h^\top\beta_j. \label{eq:ch07-contrast-definition} \end{equation}\]

For the two-factor model above, the drug effect among siRNA-treated observations is represented by

\[\begin{equation} h^\top = \begin{pmatrix} 0 & 0 & 1 & 1 \end{pmatrix}, \qquad h^\top\beta_j = \beta_{\mathrm{drug},j} + \beta_{\mathrm{siRNA:drug},j}. \label{eq:ch07-drug-effect-under-sirna} \end{equation}\]

This is the practical reason to care about design matrices. The design formula determines the coefficient vector. The scientific comparison is then a linear combination of those coefficients.

7.4 Wald Tests For Regression Coefficients

The testing machinery from Chapter 06 reappears once a coefficient or contrast has been estimated. For a contrast \(h^\top\beta_j\), the null hypothesis is

\[\begin{equation} H_{0j}:h^\top\beta_j=\theta_0. \label{eq:ch07-contrast-null} \end{equation}\]

The estimated contrast is \(h^\top\hat{\beta}_j\), and its standard error is

\[\begin{equation} \operatorname{SE}(h^\top\hat{\beta}_j) = \left[ h^\top \widehat{\operatorname{Cov}}(\hat{\beta}_j) h \right]^{1/2}. \label{eq:ch07-contrast-standard-error} \end{equation}\]

The Wald statistic is

\[\begin{equation} W_j = \frac{ h^\top\hat{\beta}_j-\theta_0 }{ \left[ h^\top \widehat{\operatorname{Cov}}(\hat{\beta}_j) h \right]^{1/2} }. \label{eq:ch07-wald-statistic} \end{equation}\]

With a large-sample normal approximation,

\[\begin{equation} W_j\mid H_{0j} \approx \mathcal{N}(0,1). \label{eq:ch07-wald-null} \end{equation}\]

The usual two-sided p-value is

\[\begin{equation} p_j = 2\left\{ 1-\Phi(|W_j|) \right\}, \label{eq:ch07-wald-pvalue} \end{equation}\]

where \(\Phi\) is the cumulative distribution function of the standard normal distribution. This is the same structure as in Chapter 06: a model, an effect, an uncertainty estimate, a test statistic, and then multiplicity correction over many tested features.

7.5 Generalized Linear Models

Linear regression assumes a continuous response with additive Gaussian noise. Generalized linear models keep the linear predictor but change the response distribution and the link function. For observation \(i\), write the conditional mean as

\[\begin{equation} \mu_i = \mathbb{E}(Y_i\mid z_i). \label{eq:ch07-glm-mean} \end{equation}\]

A GLM links the mean to the covariates through

\[\begin{equation} g(\mu_i) = z_i^\top\beta, \label{eq:ch07-glm-link} \end{equation}\]

where \(g\) is a link function. The inverse link gives

\[\begin{equation} \mu_i = g^{-1}(z_i^\top\beta). \label{eq:ch07-glm-inverse-link} \end{equation}\]

The same design matrix can therefore be used for continuous, binary, and count outcomes, but the sampling model changes. A Gaussian linear model uses an identity link. Logistic regression uses a logit link for binary data. Negative-binomial regression uses a log link for count data. This GLM perspective is the bridge between ordinary regression and RNA-seq differential expression (see MSMB Count Data GLM).

7.6 Negative-Binomial Regression For Count Data

For sequencing counts, the feature-wise response is \(y_j=c_{\cdot j}\in\mathbb{N}_0^n\). A DESeq2-like model uses a negative-binomial GLM for each feature \(j\):

\[\begin{equation} C_{ij} \sim \operatorname{NB}(\mu_{ij},\phi_j), \qquad \operatorname{Var}(C_{ij}\mid Z) = \mu_{ij} + \phi_j\mu_{ij}^2. \label{eq:ch07-nb-glm-count-model} \end{equation}\]

The mean is decomposed into an observation-specific size factor and an abundance-like quantity:

\[\begin{equation} \mu_{ij} = s_i q_{ij}, \qquad s_i>0. \label{eq:ch07-mean-size-factor} \end{equation}\]

The regression model is placed on the log scale:

\[\begin{equation} \log q_{ij} = z_i^\top\beta_j. \label{eq:ch07-log-abundance-linear-predictor} \end{equation}\]

Equivalently,

\[\begin{equation} \log\mu_{ij} = \log s_i + z_i^\top\beta_j. \label{eq:ch07-nb-glm-offset} \end{equation}\]

Thus \(\log s_i\) is an offset: it is part of the linear predictor, but it is known or estimated before the coefficient-fitting step and is not a coefficient of biological interest. The fitted mean can be written as

\[\begin{equation} \hat{\mu}_{ij} = \hat{s}_i \exp(z_i^\top\hat{\beta}_j). \label{eq:ch07-fitted-nb-mean} \end{equation}\]

For fixed dispersion \(\phi_j\), the coefficient estimate maximizes the negative-binomial log-likelihood:

\[\begin{equation} \hat{\beta}_j = \arg\max_{\beta\in\mathbb{R}^{r}} \sum_{i=1}^{n} \ell_{ij}(\beta;c_{ij},\hat{s}_i,z_i,\hat{\phi}_j). \label{eq:ch07-nb-glm-likelihood} \end{equation}\]

Equivalently, it minimizes the negative log-likelihood:

\[\begin{equation} \hat{\beta}_j = \arg\min_{\beta\in\mathbb{R}^{r}} \left\{ - \sum_{i=1}^{n} \ell_{ij}(\beta;c_{ij},\hat{s}_i,z_i,\hat{\phi}_j) \right\}. \label{eq:ch07-nb-glm-negative-log-likelihood} \end{equation}\]

This is the precise analogue of least squares. Ordinary least squares minimizes squared residuals because a Gaussian noise model implies a squared-error objective. Negative-binomial regression maximizes a count likelihood because the response is discrete, heteroscedastic, and overdispersed. DESeq2 additionally estimates and moderates feature-specific dispersions by borrowing information across genes (Love et al. 2014); see also MSMB Count Data GLM DESeq2 and MSMB Count Data Sharing Information.

For a fitted negative-binomial GLM, a large-sample approximation to the coefficient covariance has the form

\[\begin{equation} \widehat{\operatorname{Cov}}(\hat{\beta}_j) \approx (Z^\top \widehat{W}_j Z)^{-1}, \label{eq:ch07-nb-coefficient-covariance} \end{equation}\]

where \(\widehat{W}_j\in\mathbb{R}^{n\times n}\) is a diagonal weight matrix. Under the mean-dispersion parameterization used here, a typical working-weight form is

\[\begin{equation} \widehat{W}_j = \operatorname{diag} \left( \frac{\hat{\mu}_{1j}}{1+\hat{\phi}_j\hat{\mu}_{1j}}, \ldots, \frac{\hat{\mu}_{nj}}{1+\hat{\phi}_j\hat{\mu}_{nj}} \right). \label{eq:ch07-nb-working-weights} \end{equation}\]

The same contrast machinery then applies. A condition effect, batch-adjusted treatment effect, interaction effect, or combined-treatment effect can all be written as \(h^\top\beta_j\) and tested using Equation \(\ref{eq:ch07-wald-statistic}\).

7.7 Regression For Prediction

Regression is not always used for inference about a coefficient. Sometimes the goal is prediction. Let \(y_i\in\mathbb{R}\) be a continuous outcome and \(x_i^\top\in\mathbb{R}^{1\times p}\) the high-dimensional feature vector for observation \(i\). A linear prediction rule has the form

\[\begin{equation} \hat{y}_i = \beta_0 + x_i^\top\beta, \label{eq:ch07-linear-prediction-rule} \end{equation}\]

where \(\beta\in\mathbb{R}^{p}\) now contains predictor coefficients. The empirical squared prediction loss is

\[\begin{equation} \mathcal{L}_{\mathrm{sq}}(\beta_0,\beta) = \frac{1}{n} \sum_{i=1}^{n} \left( y_i-\beta_0-x_i^\top\beta \right)^2. \label{eq:ch07-squared-prediction-loss} \end{equation}\]

The distinction is now made by dimension and indexing. In feature-wise regression, \(\beta_j\) belongs to response feature \(j\) and has length \(r\), the number of design columns. In supervised prediction, \(\beta\) has length \(p\), the number of measured predictor features. When \(p\) is large, \(\beta\) is high-dimensional and ordinary least squares can overfit badly or may not even be identifiable.

The training error is

\[\begin{equation} \widehat{R}_{\mathrm{train}}(\hat{f}) = \frac{1}{n} \sum_{i=1}^{n} L(y_i,\hat{f}(x_i)), \label{eq:ch07-training-risk} \end{equation}\]

where \(L\) is a loss function. The scientific quantity we usually care about is the prediction error on new observations:

\[\begin{equation} R(\hat{f}) = \mathbb{E} \left[ L(Y_{\mathrm{new}},\hat{f}(X_{\mathrm{new}})) \right]. \label{eq:ch07-generalization-risk} \end{equation}\]

This expectation is over future data from the target population. In high-dimensional biology, training error is often much too optimistic, especially if feature selection, preprocessing, tuning, and model choice were performed using the same observations. This is the central supervised-learning warning in MSMB Supervised Learning and MSMB Supervised Cross-Validation.

7.8 Classification And Logistic Regression

For binary classification, the response is \(a_i\in\{0,1\}\). Logistic regression models the conditional probability of class 1:

\[\begin{equation} \rho_i = \Pr(a_i=1\mid x_i). \label{eq:ch07-class-probability} \end{equation}\]

The logit link is

\[\begin{equation} \log \left( \frac{\rho_i}{1-\rho_i} \right) = \beta_0+x_i^\top\beta. \label{eq:ch07-logistic-link} \end{equation}\]

Equivalently,

\[\begin{equation} \rho_i = \frac{ \exp(\beta_0+x_i^\top\beta) }{ 1+\exp(\beta_0+x_i^\top\beta) }. \label{eq:ch07-logistic-inverse-link} \end{equation}\]

The Bernoulli likelihood contribution is

\[\begin{equation} \Pr(a_i\mid x_i,\beta_0,\beta) = \rho_i^{a_i} (1-\rho_i)^{1-a_i}. \label{eq:ch07-bernoulli-logistic-likelihood} \end{equation}\]

The negative log-likelihood, averaged over observations, is

\[\begin{equation} \mathcal{L}_{\mathrm{log}}(\beta_0,\beta) = - \frac{1}{n} \sum_{i=1}^{n} \left[ a_i\log \rho_i + (1-a_i)\log(1-\rho_i) \right]. \label{eq:ch07-logistic-loss} \end{equation}\]

A class prediction is obtained by thresholding the fitted probability:

\[\begin{equation} \hat{a}_i = \mathbf{1}\{\hat{\rho}_i\geq t\}, \qquad 0<t<1. \label{eq:ch07-classification-threshold} \end{equation}\]

The default \(t=1/2\) is not always optimal. In medical classification, false negatives and false positives may have different consequences, and the decision threshold should reflect the scientific or clinical cost structure. The fitted probabilities and the final decisions are therefore different objects.

7.9 High-Dimensional Regression And Regularization

When \(p\) is large relative to \(n\), many coefficient vectors fit the training data almost equally well. Regularization makes the optimization problem better posed by penalizing model complexity. Ridge regression solves

\[\begin{equation} (\hat{\beta}_{0,\lambda}^{\mathrm{ridge}},\hat{\beta}_{\lambda}^{\mathrm{ridge}}) = \arg\min_{\beta_0,\beta} \left\{ \frac{1}{2n} \sum_{i=1}^{n} \left( y_i-\beta_0-x_i^\top\beta \right)^2 + \lambda \|\beta\|_2^2 \right\}. \label{eq:ch07-ridge-objective} \end{equation}\]

The penalty shrinks coefficients toward zero but usually does not set them exactly to zero. The lasso instead uses an \(\ell_1\) penalty:

\[\begin{equation} (\hat{\beta}_{0,\lambda}^{\mathrm{lasso}},\hat{\beta}_{\lambda}^{\mathrm{lasso}}) = \arg\min_{\beta_0,\beta} \left\{ \frac{1}{2n} \sum_{i=1}^{n} \left( y_i-\beta_0-x_i^\top\beta \right)^2 + \lambda \|\beta\|_1 \right\}. \label{eq:ch07-lasso-objective} \end{equation}\]

Here

\[\begin{equation} \|\beta\|_1 = \sum_{j=1}^{p} |\beta_j|. \label{eq:ch07-l1-norm} \end{equation}\]

The lasso can produce sparse estimates, meaning that many \(\hat{\beta}_{\lambda j}\) are exactly zero. This makes it attractive when the scientific hope is that only a subset of features is predictive, but sparsity is a modeling assumption, not a guarantee of biological causality (Tibshirani 1996). Penalization and model complexity are discussed in MSMB Supervised Penalization.

7.10 Model Selection For Penalized Regression

The tuning parameter \(\lambda\geq 0\) controls the strength of regularization. Choosing \(\lambda\) is therefore part of the statistical specification. In high-dimensional biology, there are three common questions that should be kept separate: which value predicts best, which value gives a parsimonious likelihood-based model, and which selected features are stable under perturbations of the data.

Cross-validation. Cross-validation estimates out-of-sample prediction error. Let \(\mathcal{I}_1,\ldots,\mathcal{I}_K\) be a partition of the observation indices into folds. If \(\hat{f}_{\lambda}^{(-k)}\) is trained without fold \(k\), the cross-validation estimate of prediction error is

\[\begin{equation} \widehat{R}_{\mathrm{CV}}(\lambda) = \frac{1}{K} \sum_{k=1}^{K} \frac{1}{|\mathcal{I}_k|} \sum_{i\in\mathcal{I}_k} L \left( y_i, \hat{f}_{\lambda}^{(-k)}(x_i) \right). \label{eq:ch07-cross-validation-risk} \end{equation}\]

The selected tuning parameter is often chosen as

\[\begin{equation} \hat{\lambda} = \arg\min_{\lambda\in\Lambda} \widehat{R}_{\mathrm{CV}}(\lambda), \label{eq:ch07-selected-lambda} \end{equation}\]

where \(\Lambda\) is a grid of candidate values. A more conservative variant, often called the one-standard-error rule, chooses the largest \(\lambda\) whose cross-validation error is within one estimated standard error of the minimum:

\[\begin{equation} \hat{\lambda}_{\mathrm{1SE}} = \max \left\{ \lambda\in\Lambda: \widehat{R}_{\mathrm{CV}}(\lambda) \leq \widehat{R}_{\mathrm{CV}}(\hat{\lambda}) + \widehat{\operatorname{SE}}_{\mathrm{CV}}(\hat{\lambda}) \right\}. \label{eq:ch07-one-se-rule} \end{equation}\]

This rule often gives a sparser model with essentially indistinguishable predictive performance. All preprocessing steps that use the outcome, including feature screening and tuning decisions, must be inside the cross-validation loop. Otherwise the estimated prediction error is biased downward (see MSMB Supervised Cross-Validation).

Information criteria. If the goal is model selection rather than prediction-error estimation, an information criterion can be used along the regularization path. For a fitted model at \(\lambda\), let \(\ell(\hat{\beta}_{0,\lambda},\hat{\beta}_{\lambda})\) be the unpenalized log-likelihood evaluated at the penalized estimate, and let \(\operatorname{df}(\lambda)\) denote an effective number of fitted parameters. The Akaike information criterion is

\[\begin{equation} \operatorname{AIC}(\lambda) = -2\ell(\hat{\beta}_{0,\lambda},\hat{\beta}_{\lambda}) + 2\operatorname{df}(\lambda). \label{eq:ch07-aic-penalized} \end{equation}\]

The Bayesian information criterion uses a stronger sample-size-dependent penalty:

\[\begin{equation} \operatorname{BIC}(\lambda) = -2\ell(\hat{\beta}_{0,\lambda},\hat{\beta}_{\lambda}) + \operatorname{df}(\lambda)\log n. \label{eq:ch07-bic-penalized} \end{equation}\]

For lasso linear regression, a common approximation is to count the intercept plus the number of nonzero coefficients,

\[\begin{equation} \operatorname{df}(\lambda) \approx 1+ \left| \left\{ j: \hat{\beta}_{\lambda j}\neq 0 \right\} \right|. \label{eq:ch07-lasso-degrees-of-freedom} \end{equation}\]

The selected tuning parameter is then, for example,

\[\begin{equation} \hat{\lambda}_{\mathrm{AIC}} = \arg\min_{\lambda\in\Lambda} \operatorname{AIC}(\lambda), \label{eq:ch07-aic-selected-lambda} \end{equation}\]

or

\[\begin{equation} \hat{\lambda}_{\mathrm{BIC}} = \arg\min_{\lambda\in\Lambda} \operatorname{BIC}(\lambda). \label{eq:ch07-bic-selected-lambda} \end{equation}\]

AIC is motivated by predictive information loss and often favors larger models, while BIC penalizes model size through \(\log n\) and therefore tends to favor more parsimonious models (Akaike 1974; Schwarz 1978). In high-dimensional settings, this interpretation is approximate: the effective degrees of freedom, correlated features, and the use of the same data for tuning all matter. Information criteria are useful for comparing models along a path, but they should not be mistaken for biological validation.

Stability selection. Sparse models can have similar prediction error but different selected feature sets. Stability selection asks how often each feature is selected under subsampling or resampling (Meinshausen and Bühlmann 2010). Let \(B\) subsamples be drawn, and let \(\hat{S}^{(b)}_{\lambda}\) be the active set selected at \(\lambda\) in subsample \(b\):

\[\begin{equation} \hat{S}^{(b)}_{\lambda} = \left\{ j: \hat{\beta}^{(b)}_{\lambda j}\neq 0 \right\}, \qquad b=1,\ldots,B. \label{eq:ch07-stability-active-set} \end{equation}\]

The selection probability of feature \(j\) is estimated by

\[\begin{equation} \hat{\pi}_{j}(\lambda) = \frac{1}{B} \sum_{b=1}^{B} \mathbf{1} \left\{ j\in\hat{S}^{(b)}_{\lambda} \right\}. \label{eq:ch07-selection-probability} \end{equation}\]

For a threshold \(\pi_{\mathrm{thr}}\in(1/2,1)\), the stable set is

\[\begin{equation} \hat{S}_{\mathrm{stable}}(\lambda,\pi_{\mathrm{thr}}) = \left\{ j: \hat{\pi}_{j}(\lambda) \geq \pi_{\mathrm{thr}} \right\}. \label{eq:ch07-stable-set} \end{equation}\]

Stability selection changes the question from “which features were selected once?” to “which features are repeatedly selected under reasonable perturbations of the data?” This is particularly important for omics data, where correlated genes, proteins, metabolites, or taxa can substitute for one another. The practical lasso workflow is therefore: fit a regularization path, select \(\lambda\) by cross-validation or an information criterion, inspect the selected variables, and then assess stability under resampling. This sequence is carried out for high-dimensional regression in Exercise Sheet 05 (Exercise Sheet 05 2026), and it is the same computational logic that will reappear in Chapter 08 when sparse regression is used to learn graph neighborhoods.

7.11 Sparse Logistic Regression

Sparse logistic regression combines the logistic loss with an \(\ell_1\) penalty:

\[\begin{equation} (\hat{\beta}_{0,\lambda},\hat{\beta}_{\lambda}) = \arg\min_{\beta_0,\beta} \left\{ \mathcal{L}_{\mathrm{log}}(\beta_0,\beta) + \lambda \|\beta\|_1 \right\}. \label{eq:ch07-sparse-logistic-objective} \end{equation}\]

The coefficient vector \(\beta\) has one entry per measured feature. In a gene-expression classifier, \(\beta_j\) is the contribution of feature \(j\) to the log-odds of class 1, conditional on the other selected features:

\[\begin{equation} \log \left( \frac{\rho_i}{1-\rho_i} \right) = \beta_0 + \sum_{j=1}^{p} x_{ij}\beta_j. \label{eq:ch07-sparse-logistic-linear-predictor} \end{equation}\]

For a fixed \(\lambda\), features with \(\hat{\beta}_{\lambda j}=0\) do not enter the fitted log-odds. This can make the classifier easier to inspect, but the selected set can be unstable when features are highly correlated. In omics, many genes, proteins, or taxa carry redundant information. Sparse prediction should therefore be assessed by held-out performance and by stability, not only by the apparent interpretability of one selected feature list.

7.12 Sparse Log-Contrast Regression For Compositions

Microbiome and other sequencing-derived abundance data are often compositional: the observed row carries relative information. If \(\pi_i\in\Delta^{p-1}\) is a continuous composition, a log-contrast model predicts an outcome from log components:

\[\begin{equation} \eta_i = \beta_0 + \sum_{j=1}^{p} \beta_j\log \pi_{ij}. \label{eq:ch07-log-contrast-linear-predictor} \end{equation}\]

The defining constraint is

\[\begin{equation} \sum_{j=1}^{p} \beta_j = 0. \label{eq:ch07-log-contrast-sum-zero} \end{equation}\]

This constraint makes the predictor scale-invariant. If all components are multiplied by the same positive constant before renormalization, the log-contrast does not change:

\[\begin{equation} \sum_{j=1}^{p} \beta_j\log(a\pi_{ij}) = \sum_{j=1}^{p} \beta_j\log\pi_{ij} + \log(a) \sum_{j=1}^{p} \beta_j = \sum_{j=1}^{p} \beta_j\log\pi_{ij}. \label{eq:ch07-log-contrast-scale-invariance} \end{equation}\]

This is the regression analogue of the log-ratio geometry introduced in Chapter 03. The model depends on ratios among components, not on their absolute scale. Equivalently, the same linear predictor can be written as a model in all pairwise log-ratios. If \(\mathbf{1}^\top\beta=0\), then

\[\begin{equation} \sum_{j=1}^{p}\beta_j\log\pi_{ij} = \frac{1}{p} \sum_{1\leq j<\ell\leq p} (\beta_j-\beta_\ell) \log\left(\frac{\pi_{ij}}{\pi_{i\ell}}\right). \label{eq:ch07-log-contrast-pairwise-ratios} \end{equation}\]

Thus a log-contrast regression is a constrained linear model on the complete set of pairwise log-ratios. The pairwise coefficient for \(\log(\pi_{ij}/\pi_{i\ell})\) is proportional to \(\beta_j-\beta_\ell\), so the model is identifiable through relative effects between components rather than through absolute effects of individual proportions. For a continuous response, sparse log-contrast regression can be written as

\[\begin{equation} (\hat{\beta}_{0,\lambda},\hat{\beta}_{\lambda}) = \arg\min_{\beta_0,\beta} \left\{ \frac{1}{2n} \sum_{i=1}^{n} \left( y_i-\beta_0-\sum_{j=1}^{p}\beta_j\log\pi_{ij} \right)^2 + \lambda\|\beta\|_1 \right\} \quad \text{subject to} \quad \mathbf{1}^\top\beta=0. \label{eq:ch07-sparse-log-contrast-objective} \end{equation}\]

For binary classification, the same log-contrast predictor can be inserted into the logistic model:

\[\begin{equation} \log \left( \frac{\rho_i}{1-\rho_i} \right) = \beta_0 + \sum_{j=1}^{p} \beta_j\log\pi_{ij}, \qquad \mathbf{1}^\top\beta=0. \label{eq:ch07-logistic-log-contrast} \end{equation}\]

Zeros require special care because \(\log 0\) is undefined. In practice, one must specify a zero-handling rule, such as filtering, pseudocounts, model-based replacement, or a method designed for sparse compositions. The important statistical point is not the pseudocount itself but the model scale: compositional predictors should be interpreted through log-ratios rather than ordinary changes in raw proportions. For the log-ratio foundation, see the compositional material used earlier (Egozcue and Tolosana-Delgado 2007) and Aitchison’s logistic-normal construction (Aitchison and Shen 1980).

7.13 What To Report

A regression analysis is interpretable only if the response, predictors, model family, link function, design matrix, contrasts, preprocessing, and validation procedure are stated. For feature-wise regression, one should report the tested contrast, the coefficient estimate, standard error, test statistic, p-value, adjusted p-value, and the model formula. For prediction, one should report the training data, the outcome, preprocessing, feature filtering, loss function, tuning procedure, whether \(\lambda\) was selected by cross-validation, AIC, BIC, or another criterion, any stability-selection threshold, the final model, and performance on held-out data.

For feature-wise models, the compact result object is

\[\begin{equation} \left( h^\top\hat{\beta}_j, \operatorname{SE}(h^\top\hat{\beta}_j), W_j, p_j, p_j^{\mathrm{adj}} \right), \qquad j=1,\ldots,m. \label{eq:ch07-feature-wise-result-object} \end{equation}\]

For prediction models, the compact result object is instead

\[\begin{equation} \left( \hat{f}, \hat{\lambda}, \widehat{R}_{\mathrm{CV}}(\hat{\lambda}), \widehat{R}_{\mathrm{test}}(\hat{f}) \right), \label{eq:ch07-prediction-result-object} \end{equation}\]

where \(\widehat{R}_{\mathrm{test}}\) is computed on data not used to fit or tune the model. Regression for inference and regression for prediction can use similar formulas, but they answer different scientific questions.

Summary

Regression is the common language behind covariate adjustment, differential expression, contrast testing, and supervised prediction. In feature-wise models, each molecular feature is a response and \(Z\) encodes the design. In supervised learning, rows of \(X\) are predictor vectors and the outcome is a phenotype, trait, or class label. Linear models are appropriate for transformed continuous responses. Negative-binomial GLMs model counts directly and explain the role of size-factor offsets in DESeq2-like workflows. Logistic regression models binary outcomes through log-odds. High-dimensional regression requires regularization, and sparse methods trade variance reduction and interpretability against instability and tuning dependence. For compositional predictors, log-contrast regression preserves the central log-ratio principle: only ratios are interpretable.

The chapter’s central message is that regression is not one method. It is a modeling grammar. The response type, design matrix, link function, likelihood, penalty, contrast, and validation scheme determine what the fitted coefficients and predictions mean.

References

Aitchison, J., and S. M. Shen. 1980. “Logistic-Normal Distributions: Some Properties and Uses.” Biometrika 67 (2): 261. https://doi.org/10.2307/2335470.
Akaike, Hirotugu. 1974. “A New Look at the Statistical Model Identification.” IEEE Transactions on Automatic Control 19 (6): 716–23. https://doi.org/10.1109/TAC.1974.1100705.
Egozcue, Juan José, and Raimon Tolosana-Delgado. 2007. Lecture Notes on Compositional Data Analysis. Universitat de Girona.
Exercise Sheet 05. 2026. Exercise Sheet 05 Solutions: Hypothesis Testing, Differential Abundance, PERMANOVA, and High-Dimensional Inference. https://bio-datascience.github.io/ahbd/assets/exercises/AHDBD_SS26_Exercise05_Solution.html.
Love, Michael I, Wolfgang Huber, and Simon Anders. 2014. “Moderated Estimation of Fold Change and Dispersion for RNA-Seq Data with DESeq2.” Genome Biology 15 (12). https://doi.org/10.1186/s13059-014-0550-8.
Meinshausen, Nicolai, and Peter Bühlmann. 2010. “Stability Selection.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 72 (4): 417–73. https://doi.org/10.1111/j.1467-9868.2010.00740.x.
Schwarz, Gideon. 1978. “Estimating the Dimension of a Model.” The Annals of Statistics 6 (2): 461–64. https://doi.org/10.1214/aos/1176344136.
Tibshirani, Robert. 1996. “Regression Shrinkage and Selection via the Lasso.” Journal of the Royal Statistical Society Series B: Statistical Methodology 58 (1): 267–88. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x.