Analysis of high-dimensional biological data

Exercise sheet 5
Author
Published

June 29, 2026

Install packages required for this exercise sheet

Code
packages <- c("microbiome", "phyloseq", "vegan",
              "ggplot2", "dplyr", "tidyr", "patchwork",
              "reshape2", "DESeq2", "pasilla", "tidyverse",
              "hexbin", "viridis", "glmnet", "hdi")
not_installed <- packages[!(packages %in%
                              installed.packages()[, "Package"])]
if (length(not_installed) > 0) {
  BiocManager::install(not_installed)
}
  • R version: 4.5.1 (2025-06-13)
  • BioC version: 3.22

Exercise 1 (Community-level testing with PERMANOVA)

In Exercise sheet 4, we used distance matrices and hierarchical clustering to visually explore whether microbiome samples group by biological condition. PERMANOVA (Permutational Multivariate Analysis of Variance) provides a formal statistical framework to test such community-level differences. Unlike univariate tests that examine one taxon at a time, PERMANOVA tests whether the entire microbial composition differs significantly between groups, using the same distance matrices we have already encountered.

In this exercise, we use the dietswap data set from the microbiome R package. The data come from a cross-over diet study comparing the gut microbiome of African Americans (AAM) and rural Africans (AFR).

Code
library(microbiome)
library(phyloseq)
library(vegan)
library(ggplot2)
library(patchwork)

data(dietswap)

Let’s start with a brief overview of the dataset.

Code
dietswap
phyloseq-class experiment-level object
otu_table()   OTU Table:         [ 130 taxa and 222 samples ]
sample_data() Sample Data:       [ 222 samples by 8 sample variables ]
tax_table()   Taxonomy Table:    [ 130 taxa by 3 taxonomic ranks ]
Code
cat("Samples:", nsamples(dietswap), "\n")
Samples: 222 
Code
cat("Taxa:   ", ntaxa(dietswap), "\n")
Taxa:    130 
Code
cat("Sample variables:", paste(sample_variables(dietswap),
                               collapse = ", "), "\n")
Sample variables: subject, sex, nationality, group, sample, timepoint, timepoint.within.group, bmi_group 

The dataset contains 222 samples and 130 taxa across 8 sample variables. The key variables for this exercise are nationality (AAM vs. AFR) and group (the dietary intervention arm).

The dietswap data set comes from a controlled diet-swap study (O’Keefe et al., Nature Communications, 2015). Rural Africans (AFR) and African Americans (AAM) swapped diets for two weeks: the AFR group switched to a high-fat, low-fibre Western diet, while the AAM group adopted a high-fibre, low-fat traditional African diet. The study investigated how rapidly the gut microbiome and its metabolic outputs respond to dietary change. The nationality variable captures the population of origin, while group encodes the dietary intervention arm.

a) Data preparation and visualization

Transform the dietswap phyloseq object to compositional (relative) abundances using microbiome::transform(dietswap, "compositional"). Compute a Bray-Curtis distance matrix on the compositional data using phyloseq::distance(). Visualize the samples in two dimensions using NMDS (ordinate() with method "NMDS") and color the points by nationality. Do the two groups appear to separate?

First, we transform to relative abundances so that samples are comparable regardless of sequencing depth.

Code
pseq_rel <- microbiome::transform(dietswap, "compositional")

Compute the Bray-Curtis distance matrix. Bray-Curtis is a widely used dissimilarity measure for abundance data; it ranges from 0 (identical communities) to 1 (no shared taxa).

Code
bc_dist <- phyloseq::distance(pseq_rel, method = "bray")

Now we compute an NMDS ordination. NMDS is a rank-based method that preserves the ordering of pairwise distances rather than their absolute values — making it robust to non-linear relationships in the distance matrix. We encountered NMDS already in Exercise sheet 3.

Code
set.seed(42)
nmds_ord <- ordinate(pseq_rel, method = "NMDS", distance = bc_dist)
Code
cat("NMDS stress:", round(nmds_ord$stress, 4), "\n")
NMDS stress: 0.1429 

A stress value below 0.2 is generally considered acceptable. Here, the stress is 0.1429, indicating a reasonable 2D representation of the original high-dimensional distances. Let’s visualize:

Code
plot_ordination(pseq_rel, nmds_ord, color = "nationality") +
  geom_point(size = 2.5, alpha = 0.7) +
  scale_color_brewer(palette = "Set1") +
  labs(title = "NMDS on Bray-Curtis distance (dietswap)",
       caption = paste0("Stress = ",
                        round(nmds_ord$stress, 4))) +
  theme_bw()
Warning: `aes_string()` was deprecated in ggplot2 3.0.0.
ℹ Please use tidy evaluation idioms with `aes()`.
ℹ See also `vignette("ggplot2-in-packages")` for more information.
ℹ The deprecated feature was likely used in the phyloseq package.
  Please report the issue at <https://github.com/joey711/phyloseq/issues>.

The two nationality groups show partial but visible separation along the first NMDS axis, suggesting that population background is associated with differences in overall microbial composition. However, there is substantial overlap between the groups. This is expected given that each group also contains samples from different time points and dietary interventions. The visual impression motivates a formal statistical test — which is exactly what PERMANOVA provides.

b) PERMANOVA

Use vegan::adonis2() to test whether microbial community composition differs significantly between the two nationality groups. Use the Bray-Curtis distance and the formula ~ nationality. How many permutations are performed by default? Report the \(R^2\) and \(p\)-value. What does \(R^2\) represent in this context?

PERMANOVA (Anderson, 2001) tests whether microbial community composition differs significantly between groups using a distance matrix — without assuming any particular distribution of the data.

Step 1: Partition the total variability

The total sum of squared distances (SStotal) across all sample pairs is partitioned into two components — just like classical ANOVA, but operating on distances rather than raw observations:

\[SS_{total} = SS_{between} + SS_{within}\]

\(SS_{between}\) captures how different samples from different groups are from each other. \(SS_{within}\) captures how different samples from the same group are from each other. If the groups are truly distinct in composition, \(SS_{between}\) should be large relative to \(SS_{within}\).

Step 2: Compute the effect size

The proportion of total variability explained by the grouping variable is:

\[R^2 = \frac{SS_{between}}{SS_{total}}\]

\(R^2\) is the primary effect size measure in PERMANOVA. It ranges from 0 (the grouping explains nothing) to 1 (the grouping explains everything).

Step 3: Assess significance by permutation

Because Bray-Curtis distances do not follow a standard parametric distribution, significance cannot be assessed using an F-table. Instead, sample labels are shuffled randomly many times, and the pseudo-F statistic is recomputed for each permutation. The \(p\)-value is the fraction of permuted statistics that exceed the observed one. By default, adonis2() uses 999 permutations, giving a minimum possible \(p\)-value of 0.001.

Key properties:

  • PERMANOVA is robust to non-normality but assumes homogeneity of multivariate dispersions — i.e., groups should have similar within-group spread (tested in part c).
  • A significant \(p\)-value means the observed \(R^2\) is unlikely under the null hypothesis of no group difference.
  • \(R^2\) and \(p\)-value complement each other: \(R^2\) quantifies how much the groups differ, while the \(p\)-value assesses whether that difference exceeds chance.
Code
set.seed(42)
meta <- data.frame(sample_data(pseq_rel))
perm_res <- adonis2(bc_dist ~ nationality, data = meta,
                    permutations = 999)
perm_res
Permutation test for adonis under reduced model
Permutation: free
Number of permutations: 999

adonis2(formula = bc_dist ~ nationality, data = meta, permutations = 999)
          Df SumOfSqs      R2      F Pr(>F)    
Model      1    6.987 0.18985 51.556  0.001 ***
Residual 220   29.815 0.81015                  
Total    221   36.801 1.00000                  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The default number of permutations is 999. \(R^2\) indicates the fraction of total community-level variance attributable to the nationality grouping. In ecological terms, this is how much of the Bray-Curtis variability between all pairs of samples is explained by whether a sample comes from the AAM or AFR population.

Here, \(R^2 = 0.190\), meaning that nationality alone explains about 19% of the total community-level variation. The pseudo-F statistic is 51.6 with a permutation \(p\)-value of 0.001 (the smallest possible value given 999 permutations). This tells us that in none of the 999 random label shuffles did the pseudo-F exceed the observed value — strong evidence that the two nationality groups harbour significantly different microbial communities.

c) Checking the homogeneity of dispersions

PERMANOVA assumes that groups have similar multivariate dispersions (spread). Use vegan::betadisper() followed by permutest() to test this assumption. If the dispersions differ significantly, how would you interpret the PERMANOVA result? Is the observed difference driven by location (i.e., different group centroids) or by spread (i.e., different within-group variability)?

PERMANOVA can reject the null hypothesis for two distinct reasons: (1) the group centroids differ (a true location effect), or (2) one group is more spread out than the other (a dispersion effect). The betadisper() / permutest() combination isolates component (2). If dispersions differ significantly, a significant PERMANOVA result might be (partly) driven by unequal spread rather than a genuine shift in community composition.

Code
# Compute distances to group centroids
bd <- betadisper(bc_dist, meta$nationality)
bd

    Homogeneity of multivariate dispersions

Call: betadisper(d = bc_dist, group = meta$nationality)

No. of Positive Eigenvalues: 95
No. of Negative Eigenvalues: 126

Average distance to median:
   AAM    AFR 
0.3788 0.3097 

Eigenvalues for PCoA axes:
(Showing 8 of 221 eigenvalues)
  PCoA1   PCoA2   PCoA3   PCoA4   PCoA5   PCoA6   PCoA7   PCoA8 
17.8049  6.8119  2.6405  1.9419  1.3174  1.1974  1.0453  0.8576 
Code
# Permutation test for homogeneity of dispersions
set.seed(42)
permutest(bd, pairwise = TRUE)

Permutation test for homogeneity of multivariate dispersions
Permutation: free
Number of permutations: 999

Response: Distances
           Df  Sum Sq Mean Sq      F N.Perm Pr(>F)    
Groups      1 0.26155 0.26155 18.367    999  0.001 ***
Residuals 220 3.13284 0.01424                         
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Pairwise comparisons:
(Observed p-value below diagonal, permuted p-value above diagonal)
          AAM   AFR
AAM           0.001
AFR 2.725e-05      

Visualize the dispersions:

Code
boxplot(bd, main = "Distance to centroid by nationality",
        xlab = "Nationality", ylab = "Distance to centroid")

If the permutest() \(p\)-value is not significant (\(p > 0.05\)), the groups have comparable within-group variability, and the PERMANOVA result can be confidently attributed to a location effect (different centroids). If the \(p\)-value is significant, the groups differ in dispersion. In that case the PERMANOVA result is ambiguous: the rejection could reflect a difference in spread, in location, or both. Additional inspection (e.g. the ordination from part a) is needed to disentangle the two.

Here, the average distance to the group centroid is 0.379 for AAM and 0.310 for AFR, and permutest() yields \(p = 0.001\) — the dispersions differ significantly. This means the AAM group has substantially greater within-group variability than the AFR group. Consequently, the significant PERMANOVA result from part (b) cannot be attributed to a location effect alone — part of the signal may come from the unequal spread. However, the NMDS ordination from part (a) shows a clear spatial separation of group centroids, which supports a genuine compositional difference beyond the dispersion effect.

d) Adding covariates

Re-run adonis2() with the extended formula ~ group + nationality to account for the dietary intervention group. Does the significance of nationality change? What does this tell you about the relative importance of diet vs. population background in shaping the gut microbiome?

Code
set.seed(42)
perm_res2 <- adonis2(bc_dist ~ group + nationality, data = meta,
                     permutations = 999)
perm_res2
Permutation test for adonis under reduced model
Permutation: free
Number of permutations: 999

adonis2(formula = bc_dist ~ group + nationality, data = meta, permutations = 999)
          Df SumOfSqs    R2     F Pr(>F)    
Model      3    7.802 0.212 19.55  0.001 ***
Residual 218   29.000 0.788                 
Total    221   36.801 1.000                 
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

nationality is expected to remain highly significant even after accounting for the dietary group, suggesting that the population background effect on the microbiome is robust to the dietary intervention. The \(R^2\) values for group vs. nationality can be compared directly: a larger \(R^2\) for nationality indicates that population of origin explains more community-level variation than the diet swap intervention.

Note that adonis2() uses sequential (Type I) sums of squares by default: the variable listed first is fitted first, and the second variable explains the remaining variation. This means term order matters. To test both orderings, one can also pass by = "margin" for marginal (Type III) tests.

The marginal test below shows that nationality explains \(R^2 = 0.190\) of total variation (\(p = 0.001\)) while group explains only \(R^2 = 0.022\) (\(p = 0.003\)) — both after adjusting for the respective other variable. Nationality thus has a roughly 9 times larger effect than the dietary intervention on community composition, and both effects are independently significant.

Code
# Marginal test (Type III) — order-independent
set.seed(42)
adonis2(bc_dist ~ group + nationality, data = meta,
        permutations = 999, by = "margin")
Permutation test for adonis under reduced model
Marginal effects of terms
Permutation: free
Number of permutations: 999

adonis2(formula = bc_dist ~ group + nationality, data = meta, permutations = 999, by = "margin")
             Df SumOfSqs      R2       F Pr(>F)    
group         2    0.815 0.02214  3.0631  0.003 ** 
nationality   1    6.985 0.18980 52.5085  0.001 ***
Residual    218   29.000 0.78800                   
Total       221   36.801 1.00000                   
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

e) Bonus: Comparison with peerj32

(Bonus) Repeat parts (a)–(b) on the peerj32 data set (also from the microbiome package), testing for differences between probiotic treatment groups (~ group). Compare the PERMANOVA \(R^2\) values between the two data sets. What does this comparison tell you about the relative effect sizes of nationality vs. probiotic supplementation on overall community composition?

Code
data(peerj32)
pseq_peerj <- peerj32$phyloseq
pseq_peerj_rel <- microbiome::transform(pseq_peerj, "compositional")
bc_peerj <- phyloseq::distance(pseq_peerj_rel, method = "bray")
Code
set.seed(42)
nmds_peerj <- ordinate(pseq_peerj_rel, method = "NMDS",
                       distance = bc_peerj)
Code
plot_ordination(pseq_peerj_rel, nmds_peerj, color = "group") +
  geom_point(size = 2.5, alpha = 0.7) +
  scale_color_brewer(palette = "Set2") +
  labs(title = "NMDS on Bray-Curtis distance (peerj32)",
       caption = paste0("Stress = ",
                        round(nmds_peerj$stress, 4))) +
  theme_bw()

Code
set.seed(42)
meta_peerj <- data.frame(sample_data(pseq_peerj_rel))
perm_peerj <- adonis2(bc_peerj ~ group, data = meta_peerj,
                      permutations = 999)
perm_peerj
Permutation test for adonis under reduced model
Permutation: free
Number of permutations: 999

adonis2(formula = bc_peerj ~ group, data = meta_peerj, permutations = 999)
         Df SumOfSqs      R2      F Pr(>F)
Model     1  0.07374 0.02808 1.2135  0.265
Residual 42  2.55222 0.97192              
Total    43  2.62596 1.00000              

The PERMANOVA \(R^2\) for probiotic treatment on the peerj32 data is expected to be much smaller than the \(R^2\) for nationality on the dietswap data. This reflects a well-known pattern: population-level factors (genetics, long-term diet, geography) shape the gut microbiome much more strongly than short-term interventions like probiotic supplementation.

Concretely, the probiotic effect in peerj32 has \(R^2 = 0.028\) and \(p = 0.265\) — not significant, explaining less than 3% of community-level variation. By contrast, nationality in dietswap has \(R^2 = 0.190\) and \(p = 0.001\) — a nearly 7 times larger effect that is highly significant. The comparison illustrates why \(R^2\) (effect size) is often more informative than \(p\)-values alone: even a small and biologically negligible effect can be statistically significant with enough samples.


Exercise 2 (Differential abundance analysis)

PERMANOVA tells us whether microbial communities differ between groups but not which taxa drive that difference. In this exercise, we move from community-level to taxon-level analysis. We systematically compare univariate testing approaches — from simple \(t\)-tests and Wilcoxon tests to multiple-testing correction — and reflect on the broader question of how robust differential abundance results really are.

We continue with the dietswap data set comparing African Americans (AAM) and rural Africans (AFR).

Code
library(microbiome)
library(ggplot2)
library(dplyr)
library(patchwork)

data(dietswap)
d <- dietswap

a) Single-taxon comparison

Pick the taxon Dialister and construct a data frame containing its abundance and the nationality grouping variable. Compare the two groups using boxplots on the absolute scale and on the \(\log_{10}(1+x)\) scale (use patchwork to display them side by side). Why is the \(1+x\) offset necessary, and what does the log transformation achieve visually?

Code
taxa <- "Dialister"

df <- data.frame(
  Abundance = abundances(d)[taxa, ],
  Group     = meta(d)$nationality
)
df$Log10_Abundance <- log10(1 + df$Abundance)
Code
p1 <- ggplot(df, aes(x = Group, y = Abundance)) +
  geom_boxplot() +
  labs(title = "Absolute abundances",
       y = "Abundance\n(read count)") +
  theme_bw()

p2 <- ggplot(df, aes(x = Group, y = Log10_Abundance)) +
  geom_boxplot() +
  labs(title = "Log10 abundances",
       y = "Abundance\n(log10(1+x) read count)") +
  theme_bw()

p1 + p2

The \(1+x\) offset is necessary because the data contain zeros: \(\log_{10}(0)\) is undefined, so we shift all values by 1 before taking the log. This is a common (though imperfect) workaround for zero-inflated count data. On the absolute scale, the comparison is dominated by a few high-abundance outliers (some samples reach counts above 3000) and the boxplots are heavily compressed toward zero. The log transformation compresses the dynamic range and makes the distributional differences between groups much easier to see: on the log10 scale, the AFR group shows a clearly higher median abundance than the AAM group.

b) Univariate tests

Test the group difference for Dialister using both a \(t\)-test and a Wilcoxon rank-sum test (on the log-transformed data). Inspect the distributional assumption by plotting a density plot of the \(\log_{10}\) abundances, colored by group. Which test is more appropriate for these data? Verify that the Wilcoxon test yields the same \(p\)-value on the absolute abundances as on the log-transformed data. Why is this the case?

Code
cat("t-test p-value (log10):\n")
t-test p-value (log10):
Code
t.test(Log10_Abundance ~ Group, data = df)$p.value
[1] 0.02554997
Code
cat("\nWilcoxon p-value (log10):\n")

Wilcoxon p-value (log10):
Code
wilcox.test(Log10_Abundance ~ Group, data = df)$p.value
[1] 0.02979053

Now let’s inspect the distributional assumption with a density plot:

Code
ggplot(df, aes(fill = Group, x = Log10_Abundance)) +
  geom_density(alpha = 0.5) +
  scale_fill_brewer(palette = "Set1") +
  labs(title = "Density of log10(1+x) abundances for Dialister") +
  theme_bw()

The density plot reveals that the data are not Gaussian distributed — both groups show skewness and the AAM group has a pronounced spike at zero (many samples with no Dialister detected). The \(t\)-test assumes normality and is therefore less appropriate here; the Wilcoxon rank-sum test is the safer choice.

Both tests yield significant results at \(\alpha = 0.05\): the \(t\)-test gives \(p = 0.026\) and the Wilcoxon test gives \(p = 0.030\). While the results agree in this case, the Wilcoxon test is the more trustworthy one given the non-Gaussian distribution.

Now let’s verify that the Wilcoxon test gives the same \(p\)-value on absolute and log-transformed abundances:

Code
cat("Wilcoxon p-value (absolute):\n")
Wilcoxon p-value (absolute):
Code
wilcox.test(Abundance ~ Group, data = df)$p.value
[1] 0.02979053
Code
cat("\nWilcoxon p-value (log10):\n")

Wilcoxon p-value (log10):
Code
wilcox.test(Log10_Abundance ~ Group, data = df)$p.value
[1] 0.02979053

Both return \(p = 0.0298\) — the values are identical, confirming the invariance property.

The Wilcoxon rank-sum test operates on ranks, not on the actual values. Any monotone (strictly increasing) transformation — such as \(\log_{10}(1+x)\) — preserves the rank order of observations. Since the ranks are identical, the test statistic and \(p\)-value are unchanged. This is a key advantage of non-parametric tests: they are robust to the choice of transformation.

c) Multiple testing

Restrict the analysis to core taxa using core_members(d, prevalence = 20/100, detection = 3). For each core taxon, compute a \(t\)-test \(p\)-value (on \(\log_{10}\) data) and a Wilcoxon \(p\)-value (on absolute abundances). Apply Benjamini-Hochberg FDR correction using p.adjust().

Code
test.taxa <- core_members(d, prevalence = 20/100, detection = 3)
cat("Number of core taxa:", length(test.taxa), "\n")
Number of core taxa: 83 
Code
pvalue.ttest    <- c()
pvalue.wilcoxon <- c()

for (taxon in test.taxa) {
  df_loop <- data.frame(
    Abundance       = abundances(d)[taxon, ],
    Log10_Abundance = log10(1 + abundances(d)[taxon, ]),
    Group           = meta(d)$nationality
  )
  pvalue.ttest[[taxon]]    <- t.test(Log10_Abundance ~ Group,
                                     data = df_loop)$p.value
  pvalue.wilcoxon[[taxon]] <- wilcox.test(Abundance ~ Group,
                                          data = df_loop)$p.value
}

pvalues <- data.frame(
  taxon                     = test.taxa,
  pvalue.ttest              = pvalue.ttest,
  pvalue.wilcoxon           = as.numeric(pvalue.wilcoxon),
  pvalue.ttest.adjusted     = p.adjust(pvalue.ttest),
  pvalue.wilcoxon.adjusted  = p.adjust(pvalue.wilcoxon)
)

(i) Raw vs. adjusted p-value histograms

(i) Compare the distributions of raw vs. adjusted \(p\)-values using histograms (side by side with patchwork).

Code
p1 <- ggplot(pvalues, aes(x = pvalue.wilcoxon)) +
  geom_histogram(bins = 50, binwidth = 0.03) +
  labs(title = "Raw p-values (Wilcoxon)") +
  ylim(c(0, 80)) +
  theme_bw()

p2 <- ggplot(pvalues, aes(x = pvalue.wilcoxon.adjusted)) +
  geom_histogram(bins = 50, binwidth = 0.03) +
  labs(title = "Adjusted p-values (Wilcoxon, BH)") +
  ylim(c(0, 80)) +
  theme_bw()

p1 + p2

The raw \(p\)-value histogram often shows a spike near zero (truly differentially abundant taxa) and a roughly uniform distribution elsewhere (null taxa) — the classic “two-component” shape expected under a mixture of true and null hypotheses. Here, the first bin (near zero) contains roughly 57 of the 83 core taxa, indicating that a large fraction of core taxa show significant abundance differences between AAM and AFR before correction. After BH correction, the \(p\)-values are shifted upward: many previously “significant” taxa move above the 0.05 threshold, reflecting the cost of controlling the false discovery rate across 83 simultaneous tests. The rightmost bin at 1.0 in the adjusted histogram collects all taxa whose corrected \(p\)-values were capped at 1.

(ii) t-test vs. Wilcoxon scatter plot

(ii) Create a scatter plot of adjusted \(t\)-test \(p\)-values vs. adjusted Wilcoxon \(p\)-values. Add reference lines at \(p = 0.05\) on both axes. For which taxa do the two methods disagree?

Code
ggplot(pvalues, aes(x = pvalue.ttest.adjusted,
                    y = pvalue.wilcoxon.adjusted)) +
  geom_text(aes(label = taxon), size = 2, check_overlap = FALSE) +
  geom_abline(intercept = 0, slope = 1, lty = 2) +
  geom_hline(yintercept = 0.05, lty = 2, color = "red") +
  geom_vline(xintercept = 0.05, lty = 2, color = "red") +
  labs(x = "Adjusted p-value (t-test)",
       y = "Adjusted p-value (Wilcoxon)",
       title = "Comparison of adjusted p-values") +
  theme_bw()

Most taxa fall near the diagonal, indicating broad agreement between the two methods. Taxa in the off-diagonal quadrants (significant by one method but not the other) are the interesting cases: for example, Ruminococcus gnavus et rel. and Clostridium orbiscindens et rel. are significant by the \(t\)-test (adjusted \(p < 0.05\)) but not by Wilcoxon, while Dialister and Ruminococcus bromii et rel. are not significant by either method after BH correction despite having small raw \(p\)-values (see part b). In general, the \(t\)-test may give smaller \(p\)-values for taxa with approximately Gaussian residuals and larger \(p\)-values for taxa with heavy tails or many zeros — where the Wilcoxon test is more powerful.

d) (Optional) Reading: How robust are differential abundance results?

In part (c), you observed that even two simple methods — the \(t\)-test and the Wilcoxon test — can disagree on which taxa are differentially abundant. This is not a quirk of our particular comparison: the problem extends to the entire landscape of differential abundance tools used in microbiome research.

Read the following paper and answer the questions below:

Nearing, J.T., Douglas, G.M., Hayes, M.G. et al. Microbiome differential abundance methods produce different results across 38 datasets. Nat Commun 13, 342 (2022). https://doi.org/10.1038/s41467-022-28034-z

Nearing et al. benchmark 14 differential abundance (DA) methods — including DESeq2, edgeR, LEfSe, ALDEx2, ANCOM-II, MaAsLin2, limma voom, corncob, metagenomeSeq, and various Wilcoxon/t-test variants — on 38 real 16S rRNA gene datasets with two-group comparisons.

Key findings:

  • The tools identified drastically different numbers and sets of significant ASVs on the same data. Results also depended on whether rare taxa were filtered out beforehand.
  • ALDEx2 and ANCOM-II were the most conservative methods (fewest significant hits), but their hits were most consistently recovered by other tools — suggesting higher precision.
  • edgeR and LEfSe exhibited the highest false discovery rates and are explicitly not recommended for 16S data.
  • limma voom and Wilcoxon (CLR) performed poorly on unfiltered, highly sparse data.
  • The authors recommend a consensus approach: run multiple DA methods and focus on taxa identified by most tools, rather than relying on any single method.

(i) The authors compare 14 DA methods. Into which broad categories can these methods be grouped (distribution-based, compositional, non-parametric, etc.)? Where do the \(t\)-test and Wilcoxon test from part (c) fall?

The paper groups methods roughly as follows: distribution-based methods that assume a specific parametric model for the counts (DESeq2, edgeR, corncob, metagenomeSeq, limma voom), compositional methods that account for the relative nature of sequencing data (ALDEx2, ANCOM-II, Wilcoxon on CLR-transformed data), and non-parametric or simple parametric tests (Wilcoxon, \(t\)-test, LEfSe, MaAsLin2). The \(t\)-test and Wilcoxon test from part (c) fall into the last category — they make no microbiome-specific distributional assumptions and do not account for compositionality.

(ii) Inspect Figure 1 of the paper. Which tools tend to call the most significant ASVs? Which tools are the most conservative? How does prevalence filtering (10%) change the picture?

Figure 1 shows that limma voom, Wilcoxon (CLR), edgeR, and LEfSe tend to identify the largest proportions of significant ASVs, while ALDEx2 and ANCOM-II consistently identify the fewest. After 10% prevalence filtering, the range across tools narrows somewhat and the limma voom methods become less erratic, but the overall ranking is preserved.

(iii) The authors recommend a “consensus approach.” Based on your experience in part (c), where the \(t\)-test and Wilcoxon test already disagreed on some taxa: what are the practical advantages and limitations of requiring agreement across multiple methods?

Advantages: A consensus approach increases precision — taxa flagged by multiple methods with different assumptions are more likely to be true positives. It also protects against method-specific artifacts (e.g., edgeR’s inflated FDR, limma voom’s instability on sparse data).

Limitations: It reduces sensitivity — taxa that are genuinely differentially abundant but detectable only by one class of methods (e.g., compositional tools for compositionally driven effects) may be missed. It also introduces ambiguity: there is no principled threshold for “how many tools must agree,” and the result depends on which tools are included in the consensus set.

(iv) Relate the paper’s findings to the DESeq2 pipeline you used in Exercise 3 (pasilla data). DESeq2 was originally designed for RNA-seq. According to the paper, what are the potential pitfalls of applying RNA-seq tools like DESeq2 or edgeR to microbiome data?

The key issue is sparsity: microbiome 16S data is far more zero-inflated than typical RNA-seq data. RNA-seq tools like DESeq2 and edgeR assume a Negative Binomial distribution, which can be a reasonable fit for RNA-seq but may not adequately handle the extreme zero-inflation and compositionality of microbiome count tables. The paper shows that edgeR in particular has an unacceptably high FDR on microbiome data. DESeq2 performs better but still identifies ASVs that are less consistently recovered across other methods compared to compositionally-aware tools like ALDEx2 or ANCOM-II.


Exercise 3 (DESeq2 — Pasilla data analysis)

In the lecture, we have discussed in detail the individual steps in RNA-Seq and microbiome differential expression/abundance analysis. In Chapter 8 of the MSMB book, the practical analysis centers around the pasilla data set. In this exercise, the task is to reproduce the analysis in the book.

a) First look at the Pasilla data

Start by re-reading Chapter 8.3. Install the pasilla package and explore the associated count data.

These data are from an experiment on Drosophila melanogaster cell cultures that investigated the effect of RNAi knock-down of the splicing factor pasilla (Brooks et al. 2011) on the cells’ transcriptome.

Source: YouTube video RNAi: Gene Knock-down!

We already explored this data set briefly on earlier exercise sheets (normalization via size factors and VST). Here, we load it from scratch and take a closer look at the raw counts.

Code
library(DESeq2)
library(pasilla)
library(tidyverse)
library(hexbin)
Code
fn <- system.file("extdata", "pasilla_gene_counts.tsv",
                  package = "pasilla", mustWork = TRUE)

counts <- as.matrix(read.csv(fn, sep = "\t", row.names = "gene_id"))

dim(counts)
[1] 14599     7
Code
head(counts)
            untreated1 untreated2 untreated3 untreated4 treated1 treated2
FBgn0000003          0          0          0          0        0        0
FBgn0000008         92        161         76         70      140       88
FBgn0000014          5          1          0          0        4        0
FBgn0000015          0          2          1          2        1        0
FBgn0000017       4664       8714       3564       3150     6205     3072
FBgn0000018        583        761        245        310      722      299
            treated3
FBgn0000003        1
FBgn0000008       70
FBgn0000014        0
FBgn0000015        0
FBgn0000017     3334
FBgn0000018      308

The matrix tallies the number of reads seen for each gene in each sample. It contains 7 samples and 14599 genes.

b) Normalization

Answer Question 8.1 in the book and reproduce Figure 8.2 and 8.3.

CautionQuestion 8.1

For the example dataset count of Section 8.3, how does the output of DESeq2’s estimateSizeFactorsForMatrix compare to what you get by simply taking the column sums?

The most important systematic bias stems from variations in the total number of reads in each sample. If we have more reads for one library than in another, then we might assume that — everything else being equal — the counts are proportional to each other with some proportionality factor s.

Let’s take a look at the bar plots of the original and normalized counts for a small subset of the data. Here, we simply normalize by total sum scaling.

Code
library(viridis)
Code
# Subset 10 genes
counts_sub <- counts[21:30, ]

# Observed counts
barplot(as.matrix(counts_sub), col = viridis(12)[12:3], las = 2)

Code
# Normalized counts
counts_norm <- apply(counts_sub, 2, function(x) x / sum(x))
barplot(as.matrix(counts_norm), col = viridis(12)[12:3], las = 2)

The library sizes differ considerably, so that the counts of the same gene differ extremely across samples. When looking at the fractions, there are only slight differences.

Instead of simply dividing by the column sums, DESeq2 provides a different approach: the “median of ratios method”, which is implemented by the function estimateSizeFactorsForMatrix.

Let’s look at a tiny example to understand this approach.

Code
m <- matrix(c(60, 230, 40, 180, 1, 3, 1000, 5, 10, 40, 20, 100),
            byrow = TRUE, ncol = 2)
rownames(m) <- paste0("gene", 1:6)
colnames(m) <- c("sample1", "sample2")
counts_tmp <- as.data.frame(m)
counts_tmp
      sample1 sample2
gene1      60     230
gene2      40     180
gene3       1       3
gene4    1000       5
gene5      10      40
gene6      20     100
Code
# Common scale normalization:
counts_norm_tmp <- apply(counts_tmp, 2, function(x) x / sum(x))
css_counts <- as.data.frame(
  counts_norm_tmp * min(colSums(counts_tmp[, 1:2]))
)
css_counts
          sample1 sample2
gene1  29.6021220     230
gene2  19.7347480     180
gene3   0.4933687       3
gene4 493.3687003       5
gene5   4.9336870      40
gene6   9.8673740     100
Code
colSums(css_counts)
sample1 sample2 
    558     558 

Median of ratios method:

Step 1: Create a pseudo-reference sample (row-wise geometric mean) For each gene, a pseudo-reference sample is created that is equal to the geometric mean across all samples.

Code
counts_tmp$pseudo_ref <- round(
  sqrt(counts_tmp[, 1] * counts_tmp[, 2]), 1
)
counts_tmp
      sample1 sample2 pseudo_ref
gene1      60     230      117.5
gene2      40     180       84.9
gene3       1       3        1.7
gene4    1000       5       70.7
gene5      10      40       20.0
gene6      20     100       44.7

Step 2: Calculate ratio of each sample to the reference For every gene in a sample, the ratios (sample/ref) are calculated (as shown below). This is performed for each sample in the dataset. Since the majority of genes are not differentially expressed, the majority of genes in each sample should have similar ratios within the sample.

Code
counts_tmp$samp1_ratio <- counts_tmp$sample1 / counts_tmp$pseudo_ref
counts_tmp$samp2_ratio <- counts_tmp$sample2 / counts_tmp$pseudo_ref
counts_tmp
      sample1 sample2 pseudo_ref samp1_ratio samp2_ratio
gene1      60     230      117.5   0.5106383  1.95744681
gene2      40     180       84.9   0.4711425  2.12014134
gene3       1       3        1.7   0.5882353  1.76470588
gene4    1000       5       70.7  14.1442716  0.07072136
gene5      10      40       20.0   0.5000000  2.00000000
gene6      20     100       44.7   0.4474273  2.23713647

We notice that — except for gene 4 — all ratios of sample 1 are smaller than 1 and all ratios of sample 2 are larger than one. Taking the median of all genes gives us the size factor for this sample.

Step 3: Calculate the normalization factor for each sample (size factor) The median value (column-wise for the above table) of all ratios for a given sample is taken as the normalization factor (size factor) for that sample, as calculated below. Notice that the differentially expressed genes should not affect the median value:

Code
(size_factors <- apply(counts_tmp[, 4:5], 2, median))
samp1_ratio samp2_ratio 
  0.5053191   1.9787234 

Samples with a lower sum of read counts have a lower size factor so that the mean of the Negative Binomial can be modeled correctly.

Let’s compare the two normalization methods:

Code
counts_tmp <- counts_tmp[, 1:2]
sized_counts <- as.data.frame(
  sapply(1:ncol(counts_tmp),
         function(i) counts_tmp[, i] / size_factors[i])
)
dimnames(sized_counts) <- dimnames(css_counts)
sized_counts
          sample1    sample2
gene1  118.736842 116.236559
gene2   79.157895  90.967742
gene3    1.978947   1.516129
gene4 1978.947368   2.526882
gene5   19.789474  20.215054
gene6   39.578947  50.537634
Code
colSums(sized_counts)
 sample1  sample2 
2238.189  282.000 

Now let’s apply the estimateSizeFactorsForMatrix function to the actual count matrix to get the size factors for our data.

Code
size_factors <- estimateSizeFactorsForMatrix(counts)

factor_comparison <- data.frame(
  "size_factors" = size_factors,
  "colsums" = colSums(counts)
)

ggplot(factor_comparison, aes(x = colsums, y = size_factors)) +
  geom_point() +
  ggtitle("Size Factors vs Library Size") +
  labs(x = "Library Size", y = "Size Factors")

This method is preferable to total sum scaling because it’s robust to imbalance in up-/down-regulation and therefore to differentially expressed genes. This is nicely explained here: StatQuest: DESeq2, part 1, Library Normalization.

Here, there is not much difference, the results are nearly proportional.

We normalize the counts by library size by dividing by the size factors.

Code
sized_counts <- sapply(1:ncol(counts),
                       function(i) counts[, i] / size_factors[i])
colnames(sized_counts) <- colnames(counts)

head(sized_counts)
             untreated1   untreated2  untreated3  untreated4     treated1
FBgn0000003    0.000000    0.0000000    0.000000    0.000000    0.0000000
FBgn0000008   80.824908   89.7936241  117.004615   93.123591   85.5968034
FBgn0000014    4.392658    0.5577244    0.000000    0.000000    2.4456230
FBgn0000015    0.000000    1.1154487    1.539534    2.660674    0.6114057
FBgn0000017 4097.471407 4860.0101913 5486.900612 4190.561607 3793.7726082
FBgn0000018  512.183926  424.4282483  377.185929  412.404476  441.4349433
             treated2    treated3
FBgn0000003    0.0000    1.200981
FBgn0000008  115.5963   84.068670
FBgn0000014    0.0000    0.000000
FBgn0000015    0.0000    0.000000
FBgn0000017 4035.3632 4004.070676
FBgn0000018  392.7648  369.902150

We can then look at the mean-variance relationship for the untreated samples. The green line is what we expect if the variance equals the mean. The red line corresponds to the quadratic mean-variance relationship.

Code
untreated_counts <- sized_counts[, grep("untreated",
                                        colnames(sized_counts)),
                                 drop = FALSE]

untreated_data <- data.frame(
  log_mean = log2(rowMeans(untreated_counts)),
  log_var  = log2(MatrixGenerics::rowVars(untreated_counts))
)

ggplot(untreated_data, aes(x = log_mean, y = log_var)) +
  geom_hex() +
  coord_fixed() +
  geom_abline(slope = 1:2, color = c("forestgreen", "red")) +
  labs(x = "log(mean)", y = "log(var)",
       title = "Variance versus mean for the size factor\nadjusted counts data")

The green line (slope 1) is what we expect if the variance (v) equals the mean (m), as is the case for a Poisson-distributed random variable: v = m. We see that this approximately fits the data in the lower range. The red line (slope 2) corresponds to the quadratic mean-variance relationship v = m². We can see that in the upper range of the data, the quadratic relationship approximately fits the data.

c) Annotation data

Read Chapter 8.5.1 and explore the annotation data associated with the pasilla dataset.

The overall dataset was produced in two batches, the first one consisting of three sequencing libraries that were subjected to single end sequencing, the second batch consisting of four libraries for which paired end sequencing was used.

In single-end reading, the sequencer reads a fragment from only one end to the other, generating the sequence of base pairs. In paired-end reading it starts at one read, finishes this direction at the specified read length, and then starts another round of reading from the opposite end of the fragment. Paired-end reading improves the ability to identify the relative positions of various reads in the genome, making it much more effective than single-end reading in resolving structural rearrangements such as gene insertions, deletions, or inversions. It can also improve the assembly of repetitive regions. This degree of accuracy may not be required for all experiments, however, and paired-end reads are more expensive and time-consuming to perform than single-end reads.

Code
annotationFile <- system.file("extdata",
                              "pasilla_sample_annotation.csv",
                              package = "pasilla",
                              mustWork = TRUE)

pasillaSampleAnno <- read_csv(annotationFile,
                              show_col_types = FALSE) %>%
  mutate(
    condition = factor(condition,
                       levels = c("untreated", "treated")),
    type = factor(sub("-.*", "", type),
                  levels = c("single", "paired"))
  )

mt <- match(colnames(counts),
            sub("fb$", "", pasillaSampleAnno$file))

pasillaSampleAnno <- pasillaSampleAnno[mt, ]

pasillaSampleAnno
# A tibble: 7 × 6
  file    condition type  `number of lanes` total number of read…¹ `exon counts`
  <chr>   <fct>     <fct>             <dbl> <chr>                          <dbl>
1 untrea… untreated sing…                 2 17812866                    14924838
2 untrea… untreated sing…                 6 34284521                    20764558
3 untrea… untreated pair…                 2 10542625 (x2)               10283129
4 untrea… untreated pair…                 2 12214974 (x2)               11653031
5 treate… treated   sing…                 5 35158667                    15679615
6 treate… treated   pair…                 2 12242535 (x2)               15620018
7 treate… treated   pair…                 2 12443664 (x2)               12733865
# ℹ abbreviated name: ¹​`total number of reads`

Exon Counts refer to the number of reads (or sequences) that map to the exons of a gene in RNA sequencing (RNA-seq) data. Exons are the coding regions of a gene that are transcribed into mRNA and ultimately translated into proteins. Counting reads that map to exons is crucial for gene expression analysis because it provides information on how much of each gene is being expressed in the sample.

d) DESeq workflow

We are now ready to apply DESeq2 to the data set. Follow the steps in Chapter 8.5.2. Which conceptual steps does the command DESeq2 entail?

Before actually using the DESeq2 command, it makes sense to remove genes with low counts.

Code
counts_filt <- counts[rowSums(counts) > 10, ]

Now, the DESeq function is called because of its inner functions: estimateSizeFactors for normalization, estimateDispersions for dispersion estimation, and nbinomWaldTest for hypothesis tests for differential abundance.

Code
dds <- DESeqDataSetFromMatrix(countData = counts_filt,
                              colData = pasillaSampleAnno,
                              design = ~ condition)

## Differential expression analysis
(dds <- DESeq(dds))
estimating size factors
estimating dispersions
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing
class: DESeqDataSet 
dim: 9819 7 
metadata(1): version
assays(4): counts mu H cooks
rownames(9819): FBgn0000008 FBgn0000017 ... FBgn0261574 FBgn0261575
rowData names(22): baseMean baseVar ... deviance maxCooks
colnames(7): untreated1 untreated2 ... treated2 treated3
colData names(7): file condition ... exon counts sizeFactor

results extracts a result table from a DESeq analysis giving base means across samples, log2 fold changes, standard errors, test statistics, p-values and adjusted p-values.

Code
res <- results(dds)

head(res)
log2 fold change (MLE): condition treated vs untreated 
Wald test p-value: condition treated vs untreated 
DataFrame with 6 rows and 6 columns
              baseMean log2FoldChange     lfcSE        stat    pvalue      padj
             <numeric>      <numeric> <numeric>   <numeric> <numeric> <numeric>
FBgn0000008   95.13988     0.00213202  0.223785  0.00952711 0.9923986  0.996999
FBgn0000017 4352.33093    -0.24007395  0.126292 -1.90094528 0.0573092  0.289101
FBgn0000018  418.58606    -0.10482397  0.148425 -0.70624066 0.4800385  0.828966
FBgn0000024    6.40581     0.21074346  0.689680  0.30556690 0.7599344  0.944235
FBgn0000032  989.66163    -0.09194043  0.147720 -0.62239859 0.5336798  0.855013
FBgn0000037   14.09366     0.46292340  0.490630  0.94352858 0.3454106  0.745997

Interpretation of the columns: baseMean is the average of normalized counts in all samples, log2FoldChange is the fold change of a gene in the treated condition, lfcSE is the standard error of the log2FoldChange, stat is the test statistic of the Wald test, and pvalue / padj are the unadjusted and adjusted p-values.

The log2 fold change (LFC) quantifies how much the expression of a gene changes between two conditions. It is defined as:

\[\text{LFC}_i = \log_2\left(\frac{\text{expression in treated}}{\text{expression in untreated}}\right) = \beta_{i1}\]

Why log2? A simple ratio is asymmetric: a 2-fold increase gives 2, but a 2-fold decrease gives 0.5 — not −2. On the log2 scale, these become +1 and −1, making up- and downregulation symmetric around zero. Additionally, log2 turns multiplicative biological effects into additive model terms, enabling the use of standard linear model machinery.

Interpreting LFC values:

LFC Fold change Interpretation
0 1× No change
1 2× Doubled in treated
2 4× Quadrupled in treated
−1 0.5× Halved in treated
−2 0.25× Quarter of untreated

LFC and the Wald test: DESeq2 estimates the LFC as the coefficient \(\beta_{i1}\) in the generalized linear model. The Wald test then asks: is this coefficient significantly different from zero? A gene is called differentially expressed when both the LFC (effect size) and the Wald test p-value (statistical evidence) support a real difference.

Note that a large LFC alone is not sufficient — a gene with very few counts may show a large LFC simply due to random variation. Conversely, a gene with thousands of counts may show a small but highly significant LFC. Both pieces of information are visible in the MA plot.

Code
summary(res)

out of 9819 with nonzero total read count
adjusted p-value < 0.1
LFC > 0 (up)       : 515, 5.2%
LFC < 0 (down)     : 534, 5.4%
outliers [1]       : 1, 0.01%
low counts [2]     : 1333, 14%
(mean count < 6)
[1] see 'cooksCutoff' argument of ?results
[2] see 'independentFiltering' argument of ?results

We can change the significance level to 0.01 to be more conservative:

Code
res <- results(dds, alpha = 0.01)

summary(res)

out of 9819 with nonzero total read count
adjusted p-value < 0.01
LFC > 0 (up)       : 272, 2.8%
LFC < 0 (down)     : 303, 3.1%
outliers [1]       : 1, 0.01%
low counts [2]     : 1523, 16%
(mean count < 7)
[1] see 'cooksCutoff' argument of ?results
[2] see 'independentFiltering' argument of ?results

e) Visualization of the results

Explore the results using the visualization techniques outlined in 8.5.3.

P-values

We first look at the histogram of p-values.

Code
ggplot(as(res, "data.frame"), aes(x = pvalue)) +
  geom_histogram(binwidth = 0.01, fill = "Royalblue",
                 boundary = 0) +
  geom_hline(yintercept = 100, color = "red") +
  ggtitle("Distribution of p-values")
Warning: Removed 1 row containing non-finite outside the scale range
(`stat_bin()`).

The distribution displays two main components: a uniform background with values between 0 and 1, and a peak of small p-values at the left. The uniform background corresponds to the non-differentially expressed genes. The leftmost bin contains all p-values between 0 and 0.01. The background level is at around 100, so the FDR associated with calling all genes in the leftmost bin would be around 10% (100 out of 1000 counts).

MA plot

Fold change versus mean of size-factor normalized counts.

Code
plotMA(dds)

Code
plotMA(dds, alpha = 0.01)

Data points with extreme values along the y-axis represent the genes with highly differential expression levels (although, not necessarily differentially expressed). Blue points indicate genes that are differentially expressed in the treatment group.

PCA

While dividing the counts by the size factors adjusts for differences in sequencing depth, it does not stabilize the variance across different expression levels, which is needed for a PCA. Therefore, we use a variance-stabilizing transformation here.

The rlogTransformation function applies a “regularized log” transformation. It produces a similar variance stabilizing effect as varianceStabilizingTransformation, though rlog is more robust in the case when the size factors vary widely.

Code
pas_rlog <- rlogTransformation(dds)

plotPCA(pas_rlog, intgroup = c("condition", "type"))
using ntop=500 top features by variance

This type of plot is useful for visualizing the overall effect of experimental covariates and/or to detect batch effects. Here, the first principal axis, PC1, is mostly aligned with the experimental covariate of interest (untreated / treated), while the second axis is roughly aligned with the sequencing protocol (single / paired).

f) Two-factor analysis

Re-do the differential expression analysis with a factorial design. Follow the steps in Chapter 8.9 and reproduce Figure 8.9. How would you interpret the comparative results? Have you seen other examples in the lecture where sequencing technology played a pivotal role?

Method used to examine the effects of two different factors on a dependent variable. In the context of RNA-Seq data and DESeq2, it is used to analyze the influence of two different experimental conditions (factors) on gene expression levels.

We saw in the exploratory analysis that sequencing type has a considerable systematic effect on the data. We will now incorporate this information into our differential expression analysis. The variable of primary interest (here “condition”) should be at the end of the formula, which helps interpreting the results.

Code
dds_two <- dds

design(dds_two) <- formula(~ type + condition)
dds_two <- DESeq(dds_two)
using pre-existing size factors
estimating dispersions
found already estimated dispersions, replacing these
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing
Code
res2 <- results(dds_two, alpha = 0.01)
head(res2)
log2 fold change (MLE): condition treated vs untreated 
Wald test p-value: condition treated vs untreated 
DataFrame with 6 rows and 6 columns
              baseMean log2FoldChange     lfcSE      stat    pvalue      padj
             <numeric>      <numeric> <numeric> <numeric> <numeric> <numeric>
FBgn0000008   95.13988     -0.0407290  0.219875 -0.185237 0.8530429  0.950124
FBgn0000017 4352.33093     -0.2562416  0.112259 -2.282585 0.0224548  0.128778
FBgn0000018  418.58606     -0.0647999  0.131322 -0.493442 0.6217001  0.857050
FBgn0000024    6.40581      0.3086048  0.755108  0.408690 0.6827674  0.886778
FBgn0000032  989.66163     -0.0485676  0.120861 -0.401848 0.6877962  0.889081
FBgn0000037   14.09366      0.4531415  0.526110  0.861306 0.3890695  0.707900

Comparison p-values (on a log-transformed scale):

The following transformation is used: \((-\log_{10}p)^{\frac{1}{6}}\). Thus, smaller p-values are transformed into higher values.

Code
trsf <- function(x) ifelse(is.na(x), 0, (-log10(x))^(1/6))

pval_comparison <- data.frame(pOne = trsf(res$pvalue),
                              pTwo = trsf(res2$pvalue))

ggplot(pval_comparison, aes(x = pOne, y = pTwo)) +
  geom_hex(bins = 75) +
  labs(x = "Single factor analysis (condition)",
       y = "Two factor analysis (type + condition)") +
  geom_abline(col = "orange")

The p-values in the two-factor analysis are similar to those from the one-factor analysis, but are generally smaller. The more sophisticated analysis has led to an, albeit modest, increase in power. We can also see this by counting the number of genes that pass a certain significance threshold in each case.

Code
compareRes <- table(
  `simple analysis` = res$padj < 0.01,
  `two factor`      = res2$padj < 0.01
)

addmargins(compareRes)
               two factor
simple analysis FALSE TRUE  Sum
          FALSE  7547  173 7720
          TRUE      9  566  575
          Sum    7556  739 8295

The two-factor analysis detects more genes as differentially expressed than the one-factor analysis, illustrating the benefit of accounting for known batch effects in the experimental design.


Exercise 4 (Sparse regression on B. subtilis data)

In the lecture, we detailed sparse (linear) regression techniques to learn parsimonious models from biological data in the p > n setting. This exercise will go through the individual steps with a special focus on model selection. The basis for this exercise is the review article:

P. Bühlmann, M. Kalisch, and L. Meier, High-Dimensional Statistics with a View Toward Applications in Biology, Annu. Rev. Stat. Its Appl., vol. 1, no. 1, pp. 255–278, Jan. 2014.

The article is also available on the Moodle course page.

Our analysis will center around the riboflavin data set which is part of the hdi R package.

Code
library(glmnet)
library(hdi)
data("riboflavin")

Riboflavin (Vitamin B) is a small molecule that can be produced by the Bacterium Bacillus subtilis. The data set comprises the log-production rate of riboflavin for n = 71 different strains of B. subtilis as measured in separate bioreactors. For each of the 71 strains, we have p = 4088 gene expression measurements. Unfortunately, it is not known which genes are responsible for efficient riboflavin production. Given the genomic data, the task is to find a small set of genes that can accurately predict the measured log-production rate.

a) GLM with lasso regularization

As a warm-up read the first two sections of the paper. At the end of Section 2 you see the archetypical Lasso workflow on the riboflavin data using glmnet. Perform this analysis on your computer.

Can you reproduce the reported results? Which model selection scheme is used here?

Now change the random seed to set.seed(23). How many genes do we select now? Re-do this analysis with different random seeds. What is the range of number of genes that you typically select?

Code
# Gene expression matrix
gene_exp <- riboflavin[, -1]

# Log-production rate of riboflavin
rib_prod <- riboflavin[, 1]

dim(gene_exp)
[1]   71 4088
Code
length(rib_prod)
[1] 71

Lasso regression with glmnet:

Code
set.seed(42)
fit <- glmnet(x = gene_exp, y = rib_prod)
plot(fit, xvar = "lambda")

Each integer value at the top of the plot corresponds to the number of genes that have non-zero coefficients at a particular value of λ. This count decreases as λ increases (moving from right to left on the plot) because stronger regularization tends to shrink more coefficients to zero. Note that there cannot be more non-zero coefficients than samples (here 71).

Select the optimal lambda (via cross-validation)

Code
# Perform 10-fold cross-validation
fit.cv <- cv.glmnet(x = gene_exp, y = rib_prod)

## Visualize cross-validation error-path
plot(fit.cv)

Code
fit.cv

Call:  cv.glmnet(x = gene_exp, y = rib_prod) 

Measure: Mean-Squared Error 

     Lambda Index Measure      SE Nonzero
min 0.02886    66  0.2159 0.06388      40
1se 0.08812    42  0.2776 0.08427      27

The “1SE” method prefers the lambda which is one standard error away from the minimum lambda. This is a common approach as we prefer sparser models.

Code
## Get selected genes
b <- as.matrix(coef(fit.cv))
rownames(b)[b != 0]
 [1] "(Intercept)" "ARGF_at"     "DNAJ_at"     "GAPB_at"     "LYSC_at"    
 [6] "PCKA_at"     "PKSA_at"     "SPOIISA_at"  "SPOVAA_at"   "XHLB_at"    
[11] "XKDS_at"     "XTRA_at"     "YBFI_at"     "YCGO_at"     "YCKE_at"    
[16] "YCLB_at"     "YCLF_at"     "YDDK_at"     "YEBC_at"     "YEZB_at"    
[21] "YFHE_r_at"   "YHDS_r_at"   "YOAB_at"     "YRVJ_at"     "YURQ_at"    
[26] "YXLD_at"     "YXLE_at"     "YYDA_at"    

The model selects 27 genes.

Let’s do the same thing for 100 different seeds and see how many genes get selected in each model.

Code
test_seeds <- 1:100

fit_models <- function(seed, x = gene_exp, y = rib_prod) {
  set.seed(seed)
  fit.cv <- cv.glmnet(x = x, y = y)
  b <- as.matrix(coef(fit.cv))
  n_sig <- length(b[b != 0]) - 1
  return(n_sig)
}

ngenes <- sapply(test_seeds, fit_models)
Code
boxplot(ngenes, ylab = "Nb. selected genes")

For most of the seeds around 28 genes get selected. But the results vary quite a lot. There’s even a random seed for which only 15 genes get selected. This shows that cross-validation doesn’t deliver robust results in the data-scarce regime!

b) Introduction to the hdi package

In the lecture, we have seen the concept of stability selection as model selection procedure in several settings. We can apply this idea also in sparse regression tasks. Section 3.3 of the paper covers this approach using the riboflavin data. Read the Section and familiarize yourself with the hdi package.

c) Stability selection

Perform now stability selection with the package. Note that the interface slightly changed compared to the code shown in the paper in 3.3.1. Can you reproduce the results in the paper? How many genes do you select using the default setting?

Step 1: Subsampling

  • Generate multiple random subsamples \(I^*\) of size \(\lfloor n/2 \rfloor\) from the original data set \(\{1,...,n\}\).

Step 2: Variable selection on subsamples

  • Apply the variable selection algorithm (e.g., Lasso) to each subsample \(I^*\) to obtain a set of selected variables \(\hat{S}(I^*)\).

Step 3: Compute selection frequencies:

  • For each variable \(j\), compute the selection frequency \(\hat{\pi}_j\), which is the proportion of times variable \(j\) is selected across all subsamples.
  • This is approximated by: \[\hat{\pi}_j\approx\frac{1}{B}\sum_{b=1}^{B}I(j\in\hat{S}(I^*(b))),\] where \(B\) is the total number of subsamples, and \(I(\cdot)\) is an indicator function that is 1 if the condition is true and 0 otherwise.

Step 4: Determine stable variables

  • Choose a threshold \(\pi_{thresh}\) (e.g., 0.6).
  • Define the set of stable variables as those with selection frequencies \(\hat{\pi}_j\) greater than or equal to the threshold: \[\hat{S}_{stable}=\{j:\hat{\pi}_j\ge\pi_{thresh}\}.\]
Code
set.seed(37)
fit.stab <- hdi::stability(x = gene_exp, y = rib_prod,
                           B = 500,
                           EV = 1,
                           threshold = 0.55)
Warning in .resolve_control(control = control, nvars = nvars, deprecated =
list(thresh = if (!missing(thresh)) thresh, : Passing 'dfmax' to glmnet() is
deprecated. Use control = list(dfmax = ...) instead.
Code
fit.stab
Selected predictors:
--------------------
LYSC_at YOAB_at YXLD_at 
    624    2564    4003 
--------------------
Expected number of false positives controlled at level 1 

Stability selection selects only three genes.

d) Multi-split and Ridge regression

(Bonus) Read Section 3.1 and 3.2 and review again the concept of FWER that we learned in the lecture. Perform the hypothesis testing framework for univariate and sparse regression outlined in the Sections using the hdi package.

We are interested in whether our estimated coefficients are significantly different from zero. So, the hypotheses are:

\[H_{0,j}: \beta_j = 0 \quad\text{vs.}\quad H_{1,j}: \beta_j \neq 0\]

When testing many hypotheses simultaneously — for example, whether each of 4088 genes has a non-zero effect on riboflavin production — the probability of making at least one false positive increases rapidly with the number of tests. This is the multiple testing problem.

The Family-Wise Error Rate (FWER) is the probability of making at least one false positive across all tests:

\[\text{FWER} = P(\text{at least one } H_{0,j} \text{ falsely rejected})\]

For \(m\) independent tests each at level \(\alpha\), the FWER is:

\[\text{FWER} = 1 - (1 - \alpha)^m\]

For example, with \(m = 4088\) tests at \(\alpha = 0.05\):

\[\text{FWER} = 1 - (1 - 0.05)^{4088} \approx 1\]

Without correction, a false discovery is virtually guaranteed.

Controlling the FWER means ensuring that the probability of any false positive stays below a chosen threshold (typically 0.05). The simplest approach is Bonferroni correction: multiply each p-value by the total number of tests \(m\), then threshold at \(\alpha\):

\[P_{\text{corr},j} = \min(p_j \cdot m,\ 1)\]

This is very conservative — it protects strongly against false positives, but at the cost of reduced power to detect true effects.

FWER vs. FDR: The Benjamini-Hochberg correction used in Exercise 2 controls the False Discovery Rate (FDR) instead — the expected proportion of false positives among all rejected hypotheses. FDR control is less conservative than FWER control and therefore more powerful, but provides a weaker guarantee: it allows some false positives as long as their fraction stays small.

Step 1: Split the Sample

  • Randomly split the dataset \(\{1,...,n\}\) into two subsets \(I_1\) and \(I_2\).
  • \(I_1\) and \(I_2\) are of equal size if possible, i.e., \(|I_1|\approx n/2\) and \(|I_2|\approx n/2\)

Step 2: Variable Selection on \(I_1\)

  • Perform variable selection on the training set \(I_1\). This could involve methods like Lasso, forward selection, or any other variable selection technique.
  • Denote the selected variables as \(\hat{S}\)

Step 3: Inference on test set \(I_2\)

  • Using the test set \(I_2\), fit the regression model including only the variables in \(\hat{S}\).
  • The model can be written as: \[Y_{I_2} = X_{\hat{S}, I_2}\beta_{\hat{S}}+\epsilon\] where \(\beta_{\hat{S}}\) are the coefficients for the selected variables, and \(\epsilon\) is the error term.
  • Compute the p-values \(P_j\) for each variable \(j\in\hat{S}\) using t-tests from the least squares estimates.
  • The test statistics are: \(t_j = \frac{\hat{\beta}_j}{SE(\hat{\beta}_j)}\)
  • For variables not in \(\hat{S}\), set \(P_j=1\).

Step 4: Adjust for Multiple Testing

  • To control the FWER, adjust the p-values using the formula \(P_{corr,j}=\min(P_j\cdot|\hat{S}|,1)\)
  • This adjustment ensures that the probability of making one or more false discoveries is controlled.

Ridge Projection (Bühlmann 2013) provides a way to compute valid p-values for individual hypotheses \(H_{0,j}: \beta_j = 0\) in the \(p \gg n\) setting — without splitting the data and without requiring a \(\beta_{\min}\) assumption on the size of the true coefficients.

The core problem: In the \(p \gg n\) setting, OLS is undefined. Ridge regression provides a unique solution by adding an \(\ell_2\) penalty, but the resulting estimates are biased — \(\mathbb{E}[\hat{\beta}_{\text{Ridge}}] \neq \beta\). Biased estimates cannot be used directly for hypothesis testing.

Step 1: Construct a linear estimator

For each variable \(j\), define a score vector \(\mathbf{Z}^{(j)}\) based on a Ridge regression of \(X^{(j)}\) against all other variables. Use it to form the initial estimator:

\[\hat{b}_j = \frac{(\mathbf{Z}^{(j)})^T \mathbf{Y}}{(\mathbf{Z}^{(j)})^T \mathbf{X}^{(j)}}\]

This estimator is consistent but biased, because the other variables \(\{X^{(k)}; k \neq j\}\) contribute a systematic error.

Step 2: Debias using the Lasso

Subtract the estimated contribution of all other variables to remove the bias:

\[\hat{\beta}_{\text{corr},j} = \hat{b}_j - \sum_{k \neq j} P_{jk} \hat{\beta}_{\text{Lasso},k}, \quad P_{jk} = \frac{(\mathbf{Z}^{(j)})^T \mathbf{X}^{(k)}}{(\mathbf{Z}^{(j)})^T \mathbf{X}^{(j)}}\]

Step 3: Hypothesis test

The debiased estimator is approximately normally distributed:

\[\sqrt{n}(\hat{\beta}_{\text{corr},j} - \beta_j) / \hat{\sigma}_j \rightarrow \mathcal{N}(0, 1)\]

This yields a p-value for each variable \(j = 1, \ldots, p\).

Step 4: FWER correction

Since all \(p\) variables are tested simultaneously, Bonferroni correction is applied across all \(p\) tests — making this approach very conservative in the \(p \gg n\) regime.

Key difference from multisample splitting: Ridge Projection operates on the full dataset without any random splits — the debiasing step handles the selection-inference problem analytically rather than by data separation. As a consequence, it does not require a \(\beta_{\min}\) assumption, but typically has lower power than multisample splitting because the FWER correction covers all \(p\) variables rather than only the small selected set \(\hat{S}\).

Code
x <- riboflavin[, -1]
y <- riboflavin[, 1]
Code
## Multi-split p-values
set.seed(12)
fit.multi <- hdi(x, y, method = "multi-split", B = 100)

## Ridge p-values
fit.ridge <- ridge.proj(x, y)
Code
fit.ridge$pval[fit.ridge$pval <= 0.05]
named numeric(0)
Code
fit.ridge$pval.corr[fit.ridge$pval.corr <= 0.05]
named numeric(0)
Code
fit.multi$pval.corr[fit.multi$pval.corr <= 0.05]
   YXLD_at 
0.03920298 

Ridge Projection finds no significant genes after FWER correction, while multisample splitting identifies one significant gene (YXLD_at, \(p_{\text{corr}} = 0.039\)). This is not surprising: Ridge Projection tests all \(p = 4{,}088\) genes simultaneously and pays the full Bonferroni penalty, leaving insufficient power to detect any signal at \(n = 71\). Multisample splitting avoids this by first reducing the candidate set to \(|\hat{S}| \approx 20\) genes via Lasso on \(I_1\), so the FWER correction in step 4 multiplies by \(|\hat{S}|\) rather than \(p\) — a roughly 200-fold smaller penalty. Notably, YXLD_at also appears among the three genes selected by stability selection in part (c), providing convergent evidence across two independent inference frameworks that this gene has a genuine association with riboflavin production.