6  Hypothesis Testing

Chapter Aim

The previous chapters were mainly exploratory. We transformed count data, defined distances, built embeddings, and proposed clusters. Chapter 06 introduces formal hypothesis testing: the language used when we want to quantify evidence against a specified null model. In high-dimensional biological data, this is the machinery behind differential expression, differential abundance, community-level testing, genome-wide association, and many feature-wise screening analyses.

Testing may be formulated on raw counts or on transformed feature values. For feature-wise testing, the response for feature \(j\) is written as

\[\begin{equation} y_j=x_{\cdot j}\in\mathbb{R}^{n} \qquad \text{or} \qquad y_j=c_{\cdot j}\in\mathbb{N}_0^{n}, \label{eq:ch06-feature-response} \end{equation}\]

depending on whether the test is applied to transformed continuous values or raw counts. Covariates and experimental design variables are collected in

\[\begin{equation} Z\in\mathbb{R}^{n\times r}. \label{eq:ch06-design-matrix} \end{equation}\]

This notation matters. In high-throughput data, testing is rarely just “run a test for each gene.” It is a model for a feature response, a design matrix, an effect of interest, and a multiplicity correction over many hypotheses.

TipNotation convention

Notation follows the shared convention in the notation chapter. Vectors are columns by default, row vectors are written with transposes, and dimensions should be checked whenever matrices are multiplied, transformed, modeled, or decomposed.

NoteMain external reference

The main book reference for this chapter is MSMB Testing. The most relevant subsections are the motivation for FDR in high-throughput settings (MSMB Section 6.1.3), the general testing workflow (MSMB Testing Five Steps), type I and type II errors (MSMB Testing Types Of Error), the t-test (MSMB Testing t-Test), FWER (MSMB Testing FWER), FDR (MSMB Testing FDR), and the Benjamini-Hochberg procedure (MSMB Testing BH). For RNA-seq count testing, see MSMB High-Throughput Count Data. For design, confounding, and replicates, see MSMB Experimental Design. Exercise Sheet 05 is the main practical companion: it covers PERMANOVA, univariate differential abundance, multiple testing, DESeq2 on the pasilla data, and high-dimensional inference examples (Exercise Sheet 05 2026).

Learning Goals

After this chapter, students should be able to formulate null and alternative hypotheses for a feature-wise test, compute and interpret a p-value, distinguish effect size from statistical evidence, explain type I error, type II error, power, FWER, and FDR, apply Bonferroni and Benjamini-Hochberg corrections, understand why ordinary t-tests are often unstable in small-sample high-dimensional data, explain empirical-Bayes variance moderation, write down a negative-binomial GLM for differential expression, describe GWAS as large-scale marginal association testing, and interpret PERMANOVA as a distance-based community-level test.

6.1 The Five-Step Logic Of Testing

A hypothesis test begins by choosing an effect and a statistic before looking at the evidence: specify the null model, define a statistic, understand its null distribution, compute the observed evidence, and only then make a decision (see MSMB Testing Five Steps). For a feature-wise comparison, the effect might be the difference in mean expression between treated and untreated samples, the log fold change of a gene, the differential abundance of a taxon, or the association between a SNP and a disease indicator. The null hypothesis specifies what “no effect” means.

For feature \(j\), a generic null and alternative can be written as

\[\begin{equation} H_{0j}:\theta_j=0, \qquad H_{1j}:\theta_j\neq 0. \label{eq:ch06-generic-hypotheses} \end{equation}\]

Here \(\theta_j\) is the feature-specific effect of interest. It might be a difference in means, a regression coefficient, a log fold change, or a group effect in a generalized linear model. A test statistic \(T_j\) summarizes the evidence against \(H_{0j}\):

\[\begin{equation} T_j = t_j(y_j,Z). \label{eq:ch06-test-statistic} \end{equation}\]

Under the null hypothesis, the statistic has a null distribution:

\[\begin{equation} T_j\mid H_{0j} \sim F_{0j}. \label{eq:ch06-null-distribution} \end{equation}\]

The p-value is the probability, under the null model, of observing a statistic at least as extreme as the one obtained. For a two-sided test with large absolute values indicating evidence against the null,

\[\begin{equation} p_j = \Pr_{H_{0j}} \left( |T_j^{0}|\geq |T_j^{\mathrm{obs}}| \right), \label{eq:ch06-pvalue-definition} \end{equation}\]

where \(T_j^{0}\) is a random statistic drawn from the null distribution and \(T_j^{\mathrm{obs}}\) is the observed value. A small p-value says that the observed statistic is unusual under the null model. It does not say that the null hypothesis has probability \(p_j\), and it does not measure the size or biological importance of the effect.

If the p-value is compared to a significance level \(\alpha\), the decision rule is

\[\begin{equation} \text{reject }H_{0j} \quad \text{if} \quad p_j\leq \alpha. \label{eq:ch06-single-test-decision} \end{equation}\]

For a valid continuous test under the null hypothesis, the p-value is uniformly distributed:

\[\begin{equation} p_j\mid H_{0j} \sim \operatorname{Uniform}(0,1). \label{eq:ch06-null-pvalue-uniform} \end{equation}\]

This property is the basis for multiple-testing corrections and p-value histograms. In high-throughput analyses, a roughly uniform bulk of p-values with an excess near zero is often interpreted as a mixture of null and non-null features.

6.2 Errors, Power, And Effect Sizes

Testing decisions have two possible error modes, and this distinction is the bridge from single tests to power and multiplicity (see MSMB Testing Types Of Error). If the null hypothesis is true but rejected, this is a type I error:

\[\begin{equation} \Pr(\text{reject }H_{0j}\mid H_{0j}\ \text{true}) = \alpha. \label{eq:ch06-type-i-error} \end{equation}\]

If the null hypothesis is false but not rejected, this is a type II error:

\[\begin{equation} \Pr(\text{do not reject }H_{0j}\mid H_{1j}\ \text{true}) = \beta_j. \label{eq:ch06-type-ii-error} \end{equation}\]

The power is the probability of rejecting a false null:

\[\begin{equation} \operatorname{Power}_j = 1-\beta_j. \label{eq:ch06-power} \end{equation}\]

Power depends on the effect size, sample size, noise level, model assumptions, and correction for multiple testing. This is why p-values should be reported alongside effect sizes. A tiny effect can be highly significant in a large study, and a large effect can be non-significant in a small noisy study.

For a two-group comparison on transformed data, a simple effect size is the difference in group means:

\[\begin{equation} \hat{\theta}_j = \bar{x}_{1j}-\bar{x}_{0j}. \label{eq:ch06-mean-difference-effect} \end{equation}\]

For count-based differential expression, the natural effect size is often a log fold change:

\[\begin{equation} \theta_j = \log_2 \left( \frac{q_{1j}}{q_{0j}} \right). \label{eq:ch06-log2-fold-change} \end{equation}\]

Here \(q_{1j}\) and \(q_{0j}\) are abundance-like quantities after accounting for exposure or size factors. A log2 fold change of 1 means a two-fold increase; a log2 fold change of \(-1\) means a two-fold decrease.

6.3 The Two-Sample t-Test

The two-sample t-test is the classical reference point for transformed continuous feature data (see MSMB Testing t-Test for the corresponding single-feature derivation). Suppose observations belong to two groups, \(g_i\in\{0,1\}\). Let \(n_0\) and \(n_1\) be the group sizes. For feature \(j\), the group means are

\[\begin{equation} \bar{x}_{gj} = \frac{1}{n_g} \sum_{i:g_i=g}x_{ij}, \qquad g\in\{0,1\}. \label{eq:ch06-group-means} \end{equation}\]

The unbiased group variances are

\[\begin{equation} s_{gj}^2 = \frac{1}{n_g-1} \sum_{i:g_i=g} (x_{ij}-\bar{x}_{gj})^2, \qquad g\in\{0,1\}. \label{eq:ch06-group-variances} \end{equation}\]

If the two groups are assumed to have equal variance, the pooled variance estimator is

\[\begin{equation} s_{pj}^2 = \frac{(n_0-1)s_{0j}^2+(n_1-1)s_{1j}^2}{n_0+n_1-2}. \label{eq:ch06-pooled-variance} \end{equation}\]

The classical two-sample t-statistic is

\[\begin{equation} T_j = \frac{\bar{x}_{1j}-\bar{x}_{0j}} {s_{pj}\sqrt{1/n_0+1/n_1}}. \label{eq:ch06-two-sample-t-statistic} \end{equation}\]

Under the null hypothesis of equal means, and under normality and equal-variance assumptions,

\[\begin{equation} T_j\mid H_{0j} \sim t_{n_0+n_1-2}. \label{eq:ch06-t-null-distribution} \end{equation}\]

The two-sided p-value is

\[\begin{equation} p_j = 2\Pr \left( t_{n_0+n_1-2}\geq |T_j| \right). \label{eq:ch06-t-pvalue} \end{equation}\]

The t-test is simple and interpretable, but the assumptions are often fragile in high-throughput biology. Small sample sizes make variance estimates noisy. Count data are discrete and heteroscedastic. Compositional data have relative constraints. Zero inflation and outliers are common. Transformations from Chapter 03 can make t-tests more reasonable, but they do not guarantee normality or equal variances.

Welch’s t-test relaxes the equal-variance assumption by using the standard error

\[\begin{equation} \operatorname{SE}_{\mathrm{Welch},j} = \left( \frac{s_{0j}^2}{n_0} + \frac{s_{1j}^2}{n_1} \right)^{1/2}. \label{eq:ch06-welch-standard-error} \end{equation}\]

It is often preferable to the pooled t-test when group variances differ, but it still operates on transformed continuous values and does not solve count-specific modeling issues.

6.4 Rank-Based Testing

For skewed or outlier-prone transformed abundances, a rank-based test can be more robust. Let \(R_{ij}\) be the rank of observation \(i\) among the \(n\) values of feature \(j\):

\[\begin{equation} R_{ij} = \operatorname{rank}(x_{ij}). \label{eq:ch06-rank} \end{equation}\]

For a two-group Wilcoxon rank-sum test, the rank-sum statistic for group 1 is

\[\begin{equation} W_j = \sum_{i:g_i=1} R_{ij}. \label{eq:ch06-wilcoxon-statistic} \end{equation}\]

Under the null hypothesis that the two groups have the same distribution, and ignoring ties for simplicity,

\[\begin{equation} \mathbb{E}_{0}(W_j) = \frac{n_1(n+1)}{2}, \qquad \operatorname{Var}_{0}(W_j) = \frac{n_0n_1(n+1)}{12}. \label{eq:ch06-wilcoxon-mean-variance} \end{equation}\]

Because the Wilcoxon test uses ranks, it is invariant under strictly increasing transformations:

\[\begin{equation} \operatorname{rank}(x_{ij}) = \operatorname{rank}\{h(x_{ij})\} \quad \text{for strictly increasing }h. \label{eq:ch06-rank-invariance} \end{equation}\]

Exercise Sheet 05 uses this property in the microbiome setting: the Wilcoxon test gives the same p-value for absolute abundances and for a monotone log-transformed version, while the t-test changes because it depends on the numerical scale (Exercise Sheet 05 2026).

6.5 Permutation Tests

Permutation tests construct the null distribution by randomizing labels in a way that is valid under the null hypothesis. For a two-group test, let \(\pi_b\) be a permutation of the group labels for permutation \(b=1,\ldots,B\). The permuted statistic is

\[\begin{equation} T_j^{(b)} = t_j(y_j,\pi_b(g)). \label{eq:ch06-permutation-statistic} \end{equation}\]

The permutation p-value can be estimated as

\[\begin{equation} \hat{p}_j = \frac{ 1+\sum_{b=1}^{B} \mathbf{1}\{|T_j^{(b)}|\geq |T_j^{\mathrm{obs}}|\} }{ B+1 }. \label{eq:ch06-permutation-pvalue} \end{equation}\]

The plus one in numerator and denominator avoids zero p-values and reflects that the observed labeling is one of the possible labelings. Permutation tests are attractive because they require fewer parametric assumptions. Their validity still depends on exchangeability: under the null, the labels being permuted must be interchangeable. If there are batches, paired samples, repeated measures, or time structure, unrestricted permutation can be invalid.

6.6 Borrowing Strength Across Features

In high-dimensional biology, the number of features is large but the number of observations is often small. A feature-specific variance estimate \(s_j^2\) can be highly unstable. Moderated testing methods stabilize variance estimates by borrowing information across features. This is the central idea of limma’s empirical-Bayes moderated t-statistic (Smyth 2004).

A simplified moderated variance estimator has the form

\[\begin{equation} \tilde{s}_j^2 = \frac{d_0s_0^2+d_js_j^2}{d_0+d_j}. \label{eq:ch06-moderated-variance} \end{equation}\]

Here \(s_j^2\) is the ordinary feature-specific variance estimate, \(d_j\) is its degrees of freedom, \(s_0^2\) is a prior variance estimated from all features, and \(d_0\) is a prior degrees-of-freedom parameter. The moderated t-statistic replaces \(s_j\) by \(\tilde{s}_j\):

\[\begin{equation} \tilde{T}_j = \frac{\hat{\theta}_j} {\tilde{s}_j a_j}. \label{eq:ch06-moderated-t} \end{equation}\]

The factor \(a_j\) depends on the design and contrast. The important idea is that noisy feature-wise variances are shrunk toward a common trend or prior value. This improves stability when \(n\) is small and \(p\) is large. The same statistical philosophy appears repeatedly in high-throughput biology: estimate many related quantities jointly rather than pretending each feature is an isolated experiment.

6.7 Count Models For Differential Expression

For RNA-seq and related count data, it is often better to test using a count model rather than a t-test on transformed values. A feature-wise negative-binomial model uses

\[\begin{equation} C_{ij}\sim \operatorname{NB}(\mu_{ij},\phi_j), \qquad \operatorname{Var}(C_{ij})=\mu_{ij}+\phi_j\mu_{ij}^2. \label{eq:ch06-nb-count-model} \end{equation}\]

The mean is linked to size factors and covariates by

\[\begin{equation} \log \mu_{ij} = \log s_i+z_i^\top\beta_j. \label{eq:ch06-nb-glm} \end{equation}\]

Here \(s_i\) is the size factor for observation \(i\), \(z_i^\top\) is row \(i\) of the design matrix \(Z\), and \(\beta_j\in\mathbb{R}^r\) contains feature-specific regression coefficients. If the coefficient of interest is the last component, the null hypothesis is

\[\begin{equation} H_{0j}:\beta_{jr}=0 \qquad \text{versus} \qquad H_{1j}:\beta_{jr}\neq 0. \label{eq:ch06-nb-coefficient-test} \end{equation}\]

A Wald statistic has the form

\[\begin{equation} W_j = \frac{\hat{\beta}_{jr}} {\operatorname{SE}(\hat{\beta}_{jr})}. \label{eq:ch06-wald-statistic} \end{equation}\]

For large samples, the null approximation is

\[\begin{equation} W_j\mid H_{0j} \approx \mathcal{N}(0,1). \label{eq:ch06-wald-null} \end{equation}\]

DESeq2 estimates size factors, dispersions, coefficients, test statistics, p-values, and adjusted p-values in this modeling framework (Love et al. 2014). Exercise Sheet 05 applies this workflow to the pasilla data and emphasizes that exploratory transformations for PCA and heatmaps are not the same as the count model used for formal testing (Exercise Sheet 05 2026).

Likelihood-ratio tests compare a full model and a reduced model. If \(\ell_{\mathrm{full},j}\) and \(\ell_{\mathrm{red},j}\) are the maximized log-likelihoods, then

\[\begin{equation} \Lambda_j = 2 \left( \ell_{\mathrm{full},j} - \ell_{\mathrm{red},j} \right). \label{eq:ch06-likelihood-ratio-statistic} \end{equation}\]

Under regularity conditions and the null hypothesis,

\[\begin{equation} \Lambda_j\mid H_{0j} \approx \chi^2_{\Delta r}, \label{eq:ch06-lrt-null} \end{equation}\]

where \(\Delta r\) is the difference in the number of parameters. This is useful when testing multi-parameter effects, such as whether a factor with several levels contributes to expression.

6.8 Genome-Wide Association Testing

Genome-wide association studies test many genetic variants for association with a phenotype. They are therefore another large-scale feature-wise testing problem, but the features are genotypes rather than genes, taxa, or proteins. If \(X_{ij}\) is the genotype of observation \(i\) at variant \(j\), and \(Y_i\) is a quantitative trait, a simple marginal model is

\[\begin{equation} Y_i = \alpha_j+\gamma_jX_{ij}+u_i^\top\delta_j+\varepsilon_{ij}, \label{eq:ch06-gwas-marginal-model} \end{equation}\]

where \(u_i\) contains covariates such as population-structure components. The feature-wise null is

\[\begin{equation} H_{0j}:\gamma_j=0. \label{eq:ch06-gwas-null} \end{equation}\]

GWAS often controls FWER rather than FDR because individual variant discoveries can have high downstream cost and because millions of tests are performed. GWAS is a useful example of why stringent thresholds are common and why sample size matters; see also the GWAS overview in the course paper list (Tam et al. 2019).

Rare-variant tests combine information across variants in a region. In a kernel association test such as SKAT, a set of variants is represented by a genotype matrix \(G\in\mathbb{R}^{n\times q}\) for one region, and a kernel matrix can be written as

\[\begin{equation} K = G W_G G^\top. \label{eq:ch06-skat-kernel} \end{equation}\]

Here \(W_G\in\mathbb{R}^{q\times q}\) is a variant-weight matrix. A simplified score statistic has the quadratic form

\[\begin{equation} Q = r^\top K r, \label{eq:ch06-skat-score} \end{equation}\]

where \(r\) is a residual vector from a null model with covariates but without the variant set. This is a multivariate association test: it asks whether a region, rather than a single variant, is associated with the phenotype (Wu et al. 2011).

6.9 Multiple Testing

The defining feature of high-dimensional testing is multiplicity. In omics, the object of interest is usually a list of discoveries, not a single isolated decision; this is why FDR becomes a natural error criterion (see MSMB Section 6.1.3). If we test \(m\) null hypotheses at level \(\alpha\), even when all null hypotheses are true, the expected number of false positives is

\[\begin{equation} \mathbb{E}(V) = m\alpha, \label{eq:ch06-expected-false-positives} \end{equation}\]

where \(V\) is the number of falsely rejected true null hypotheses. If the tests are independent, the probability of at least one false positive under the global null is

\[\begin{equation} \Pr(V>0) = 1-(1-\alpha)^m. \label{eq:ch06-global-null-at-least-one} \end{equation}\]

For \(m=20000\) tests and \(\alpha=0.05\), the expected number of false positives under the global null is

\[\begin{equation} m\alpha = 20000\cdot 0.05 = 1000. \label{eq:ch06-twenty-thousand-tests} \end{equation}\]

This simple calculation is the reason one-at-a-time testing is not enough when thousands of features are tested.

To define the relevant error criteria, it is useful to keep explicit bookkeeping for the relationship between the testing decision and the unknown truth. The following table adapts the notation of MSMB Table 6.2 (see MSMB Testing FDR).

Table 6.1: Multiple-testing outcomes, adapted from MSMB Table 6.2. Here \(m\) is the number of tested hypotheses, \(m_0\) is the number of true null hypotheses, \(R\) is the number of rejections, \(V\) is the number of false rejections, and \(S\) is the number of true discoveries.
Test decision Null hypothesis is true Null hypothesis is false Total
Rejected \(V\) \(S\) \(R\)
Not rejected \(U\) \(T\) \(m-R\)
Total \(m_0\) \(m-m_0\) \(m\)

The table implies the accounting identities

\[\begin{equation} R=V+S, \qquad m-R=U+T. \label{eq:ch06-multiple-testing-accounting} \end{equation}\]

The family-wise error rate is the probability of making at least one false rejection (see MSMB Testing FWER):

\[\begin{equation} \operatorname{FWER} = \Pr(V>0). \label{eq:ch06-fwer} \end{equation}\]

The false discovery proportion is

\[\begin{equation} \operatorname{FDP} = \frac{V}{\max(R,1)}. \label{eq:ch06-fdp} \end{equation}\]

The false discovery rate is the expectation of the false discovery proportion (see MSMB Testing FDR):

\[\begin{equation} \operatorname{FDR} = \mathbb{E} \left[ \frac{V}{\max(R,1)} \right]. \label{eq:ch06-fdr} \end{equation}\]

FWER control is stringent: it tries to avoid any false positive. FDR control is often more useful in omics: it allows some false positives but controls their expected fraction among discoveries.

Bonferroni control rejects hypothesis \(j\) if

\[\begin{equation} p_j \leq \frac{\alpha}{m}. \label{eq:ch06-bonferroni-rule} \end{equation}\]

Equivalently, the Bonferroni-adjusted p-value is

\[\begin{equation} p_j^{\mathrm{Bonf}} = \min(mp_j,1). \label{eq:ch06-bonferroni-adjusted-pvalue} \end{equation}\]

This controls FWER but can be conservative when \(m\) is large.

6.10 Benjamini-Hochberg FDR Control

The Benjamini-Hochberg procedure controls the FDR under independence and certain dependence conditions (Benjamini and Hochberg 1995); see also MSMB Testing BH. Sort the p-values:

\[\begin{equation} p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(m)}. \label{eq:ch06-ordered-pvalues} \end{equation}\]

For target FDR level \(\alpha\), define

\[\begin{equation} \hat{k} = \max \left\{ k: p_{(k)} \leq \frac{k}{m}\alpha \right\}. \label{eq:ch06-bh-k} \end{equation}\]

If such a \(\hat{k}\) exists, reject the hypotheses corresponding to \(p_{(1)},\ldots,p_{(\hat{k})}\). If not, reject none.

BH-adjusted p-values can be written as

\[\begin{equation} p_{(k)}^{\mathrm{BH}} = \min_{\ell\geq k} \left\{ \frac{m}{\ell}p_{(\ell)} \right\}, \label{eq:ch06-bh-adjusted-pvalue-raw} \end{equation}\]

with truncation at 1:

\[\begin{equation} p_{(k)}^{\mathrm{BH}} \leftarrow \min\left(p_{(k)}^{\mathrm{BH}},1\right). \label{eq:ch06-bh-adjusted-pvalue-truncated} \end{equation}\]

The adjusted p-value is the smallest FDR level at which the feature would be called significant by the BH procedure. In differential expression and differential abundance tables, this is often reported as padj, qvalue, or adjusted p-value.

6.11 P-Value Histograms And Independent Filtering

A p-value histogram is a diagnostic for whether the test calibration and signal structure look plausible (see MSMB Testing P-Value Histogram). If most null hypotheses are true and tests are well calibrated, null p-values should contribute an approximately uniform background. Non-null features create an enrichment near zero. This informal mixture view can be written as

\[\begin{equation} f(p) = \pi_0 f_0(p) + (1-\pi_0)f_1(p), \qquad f_0(p)=1\ \text{for }0\leq p\leq 1. \label{eq:ch06-pvalue-mixture} \end{equation}\]

Here \(\pi_0\) is the fraction of true null hypotheses. This equation is not required for BH, but it explains the intuition behind p-value histograms and local FDR methods (see MSMB Testing Local FDR for a fuller two-groups treatment).

Filtering can improve power if it removes features that have little chance of being significant without using the test statistic in a biased way. Let \(q_j\) be a filtering statistic, such as mean normalized count or prevalence. Independent filtering keeps features satisfying

\[\begin{equation} q_j\geq q_{\min}. \label{eq:ch06-independent-filtering} \end{equation}\]

The filter must be independent of the test statistic under the null. Filtering low-count genes before DESeq2 testing is an example used to improve power while preserving error control when done properly. The same principle can be extended by using informative covariates to weight hypotheses while preserving FDR control under the required independence conditions (see MSMB Testing IHW).

6.12 Design, Covariates, And Confounding

Many flawed testing analyses come from using the wrong design. Suppose \(z_i^\top\) contains treatment, batch, sequencing type, sex, age, donor, or other covariates. A feature-wise linear model on transformed data is

\[\begin{equation} y_j = Z\beta_j+\varepsilon_j, \qquad \varepsilon_j\sim\mathcal{N}(0,\sigma_j^2I_n). \label{eq:ch06-feature-linear-model} \end{equation}\]

If the treatment coefficient is \(\beta_{jr}\), the null is again

\[\begin{equation} H_{0j}:\beta_{jr}=0. \label{eq:ch06-linear-model-null} \end{equation}\]

This formulation makes the role of covariates explicit. A one-factor model and a two-factor model can answer different questions. Exercise Sheet 05 shows this with the pasilla data, where sequencing type enters the design before testing the condition effect (Exercise Sheet 05 2026).

Confounding occurs when the effect of interest is statistically entangled with another variable. In matrix terms, the design is problematic if columns of \(Z\) are nearly collinear. A simple diagnostic is the rank:

\[\begin{equation} \operatorname{rank}(Z) < r. \label{eq:ch06-rank-deficient-design} \end{equation}\]

If the design matrix is rank deficient, some coefficients are not identifiable. Even if \(Z\) has full rank, near-confounding can inflate standard errors and reduce interpretability. This is why experimental design belongs before data analysis, not after it.

6.13 Community-Level Testing With PERMANOVA

Feature-wise testing asks which features differ. Sometimes the scientific question is instead whether the whole community or multivariate profile differs between groups. PERMANOVA is a distance-based test for this purpose (Anderson 2001). It begins with a distance matrix

\[\begin{equation} D=(d_{ii'})_{i,i'=1}^{n}. \label{eq:ch06-permanova-distance-matrix} \end{equation}\]

For a grouping variable, PERMANOVA partitions distance-based variability into between-group and within-group components:

\[\begin{equation} SS_{\mathrm{total}} = SS_{\mathrm{between}} + SS_{\mathrm{within}}. \label{eq:ch06-permanova-ss-partition} \end{equation}\]

The effect size is often reported as

\[\begin{equation} R^2 = \frac{SS_{\mathrm{between}}}{SS_{\mathrm{total}}}. \label{eq:ch06-permanova-r2} \end{equation}\]

A pseudo-F statistic compares between-group and within-group variability:

\[\begin{equation} F_{\mathrm{PERM}} = \frac{SS_{\mathrm{between}}/(G-1)} {SS_{\mathrm{within}}/(n-G)}. \label{eq:ch06-permanova-f} \end{equation}\]

Here \(G\) is the number of groups. Significance is assessed by permuting the group labels and recomputing the statistic:

\[\begin{equation} \hat{p}_{\mathrm{PERM}} = \frac{ 1+\sum_{b=1}^{B} \mathbf{1}\{F_{\mathrm{PERM}}^{(b)}\geq F_{\mathrm{PERM}}^{\mathrm{obs}}\} }{ B+1 }. \label{eq:ch06-permanova-pvalue} \end{equation}\]

PERMANOVA is not a general replacement for careful modeling. It tests whether groups differ in the distance-based sense induced by \(D\). If the groups have different dispersions, a significant result may reflect different spread rather than only different group centers. Exercise Sheet 05 explicitly checks homogeneity of multivariate dispersions after a significant PERMANOVA result (Exercise Sheet 05 2026).

6.14 What To Report

A testing result is interpretable only if the full testing workflow is reported. At minimum, one should state the data object tested, the normalization and transformation, the features retained after filtering, the null and alternative hypotheses, the test statistic, the design matrix, the multiplicity correction, the significance threshold, the effect sizes, and the diagnostic checks.

For feature-wise testing, a compact result table often contains

\[\begin{equation} \left( \hat{\theta}_j, \operatorname{SE}(\hat{\theta}_j), T_j, p_j, p_j^{\mathrm{adj}} \right), \qquad j=1,\ldots,m. \label{eq:ch06-result-table} \end{equation}\]

The adjusted p-value controls the error criterion; the effect size tells us whether the result is biologically meaningful. Both are needed.

Differential abundance analysis in microbiome data is especially sensitive to preprocessing, compositionality, sparsity, and method choice. Nearing and colleagues’ comparison of differential abundance methods found that different tools can produce substantially different discoveries across datasets (Nearing et al. 2022). This is not a reason to avoid testing. It is a reason to report assumptions, perform sensitivity analyses, and avoid treating a single method as an oracle.

Summary

Hypothesis testing turns a biological question into a null model, a statistic, a p-value, and a decision rule. In high-dimensional biology, the main complications are small sample sizes, noisy variance estimates, count distributions, compositional constraints, confounding, and multiplicity. Ordinary t-tests and Wilcoxon tests are useful starting points for transformed feature-wise data. Moderated tests borrow strength across features. Negative-binomial GLMs test differential expression directly on counts, while GWAS applies the same feature-wise testing logic to genetic variants. Bonferroni controls the probability of any false positive, while Benjamini-Hochberg controls the expected fraction of false discoveries among rejected hypotheses. PERMANOVA then extends the testing discussion from feature-wise hypotheses to community-level differences defined by a distance matrix.

The chapter’s central message is that a significant result is not a standalone scientific conclusion. It is evidence against a specified null model under a specified workflow. The workflow, effect size, multiplicity correction, diagnostics, and biological validation determine what the result can mean.

References

Anderson, Marti J. 2001. “A New Method for Non-Parametric Multivariate Analysis of Variance.” Austral Ecology 26 (1): 32–46. https://doi.org/10.1046/j.1442-9993.2001.01070.x.
Benjamini, Yoav, and Yosef Hochberg. 1995. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society Series B: Statistical Methodology 57 (1): 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.
Exercise Sheet 05. 2026. Exercise Sheet 05 Solutions: Hypothesis Testing, Differential Abundance, PERMANOVA, and High-Dimensional Inference. https://bio-datascience.github.io/ahbd/assets/exercises/AHDBD_SS26_Exercise05_Solution.html.
Love, Michael I, Wolfgang Huber, and Simon Anders. 2014. “Moderated Estimation of Fold Change and Dispersion for RNA-Seq Data with DESeq2.” Genome Biology 15 (12). https://doi.org/10.1186/s13059-014-0550-8.
Nearing, Jacob T., Gavin M. Douglas, Molly G. Hayes, et al. 2022. “Microbiome Differential Abundance Methods Produce Different Results Across 38 Datasets.” Nature Communications 13 (1). https://doi.org/10.1038/s41467-022-28034-z.
Smyth, Gordon K. 2004. “Linear Models and Empirical Bayes Methods for Assessing Differential Expression in Microarray Experiments.” Statistical Applications in Genetics and Molecular Biology 3 (1): 1–25. https://doi.org/10.2202/1544-6115.1027.
Tam, Vivian, Nikunj Patel, Michelle Turcotte, Yohan Bossé, Guillaume Paré, and David Meyre. 2019. “Benefits and Limitations of Genome-Wide Association Studies.” Nature Reviews Genetics 20 (8): 467–84. https://doi.org/10.1038/s41576-019-0127-1.
Wu, Michael C., Seunggeun Lee, Tianxi Cai, Yun Li, Michael Boehnke, and Xihong Lin. 2011. “Rare-Variant Association Testing for Sequencing Data with the Sequence Kernel Association Test.” The American Journal of Human Genetics 89 (1): 82–93. https://doi.org/10.1016/j.ajhg.2011.05.029.