packages <-c("tidyverse", "DESeq2", "pasilla", "vsn", "phyloseq", "SummarizedExperiment", "Rtsne", "devtools")# Find uninstalled packagesnot_installed <- packages[!(packages %in%installed.packages()[,"Package"])]# Install the missing packagesif (length(not_installed) >0) {# We use the BiocManager here, which can also install CRAN packages BiocManager::install(not_installed)}
R version: 4.5.1 (2025-06-13)
BioC version: 3.22
Exercise 01: Normalization and low dimensional representation of 16s sequencing
For these exercises, we will be using microbiome 16s-sequencing data gathered during the American Gut Project. You can gather the necessary data from SpiecEasi R package.
TipTypes of omics data
Source: Chen, C., Wang, J., Pan, D., Wang, X., Xu, Y., Yan, J., Wang, L., Yang, X., Yang, M., & Liu, G. P. (2023). Applications of multi-omics analysis in human diseases. MedComm, 4(4), e315. https://doi.org/10.1002/MCO2.315
Code
if (!require(SpiecEasi)) { devtools::install_github("zdk123/SpiecEasi")}library(SpiecEasi)library(phyloseq)library(tidyverse)library(DESeq2)library(vsn)data(amgut2.filt.phy)
a) OTU table extraction and sample filtering
Extract the OTU table as p x n matrix from the phyloseq object and keep samples only with sufficient sequencing depth, e.g., where number of reads > 100. Apply your normalization/trans- formations from the previous exercise to the data and plot mean-variance relationship. What do you observe?
In microbiome analysis, the p × n orientation (OTUs × samples) is conventional in phyloseq and matches the NGS count matrix convention from RNA-seq. Be mindful when passing data to functions that expect the transposed (n × p, samples × features) layout — for example, prcomp() or dist().
There are 296 samples (columns) and 138 OTUs (rows).
Keep in mind that this is just a sample of the complete data set. One difference between this and the RNA-seq example is that we should remove lowly sequenced samples.
Code
# Filter out low sequenced sampleskeep <-colSums(otu_counts) >=100otu_counts <- otu_counts[,keep]sample_info <- sample_info[colnames(otu_counts),]cat("Samples remaining after depth filter:", ncol(otu_counts), "\n")
Samples remaining after depth filter: 261
Code
cat("OTUs retained: ", nrow(otu_counts), "\n")
OTUs retained: 138
After filtering, there are 261 samples (columns) and 138 OTUs (rows) remaining.
Very few samples are removed, which tells us that the dataset already comes pre-filtered at a reasonable sequencing depth — as one might expect from a published, quality-controlled cohort like the American Gut Project.
b) Normalization and mean–variance relationship
Apply the following normalization/transformations to the data and plot the mean–variance relationship. What do you observe?
We first define a helper for the geometric mean (needed by CLR) and then implement the transformNGS function exactly as provided on the exercise sheet.
We apply all five strategies. For VST and rlog we need a minimal metadata frame to satisfy DESeqDataSetFromMatrix.
Code
ts_otu <-transformNGS(otu_counts, norm_strategy ="total_sum")cs_otu <-transformNGS(otu_counts, norm_strategy ="common_scale")clr_otu <-transformNGS(otu_counts, norm_strategy ="clr")# Minimal metadata — DESeq2 requires a ~condition design even if we do not# use it for the VST itselfmeta_df <-data.frame("sample"=row.names(sample_info),"condition"= sample_info$SEX) %>%column_to_rownames("sample")vst_otu <-transformNGS(otu_counts, norm_strategy ="vst",metadata.df = meta_df)rlog_otu <-transformNGS(otu_counts, norm_strategy ="rlog",metadata.df = meta_df)
Now we inspect the mean–variance relationship for each transformation.
Code
meanSdPlot(as.matrix(otu_counts), main ="Raw counts")
Warning: `aes_string()` was deprecated in ggplot2 3.0.0.
ℹ Please use tidy evaluation idioms with `aes()`.
ℹ See also `vignette("ggplot2-in-packages")` for more information.
ℹ The deprecated feature was likely used in the vsn package.
Please report the issue to the authors.
Warning in geom_hex(bins = bins, ...): Ignoring unknown parameters: `main`
Code
meanSdPlot(as.matrix(ts_otu), main ="Total-sum normalization")
Warning in geom_hex(bins = bins, ...): Ignoring unknown parameters: `main`
Code
meanSdPlot(as.matrix(cs_otu), main ="Common-scale normalization")
Warning in geom_hex(bins = bins, ...): Ignoring unknown parameters: `main`
Code
meanSdPlot(as.matrix(clr_otu), main ="CLR transformation")
Warning in geom_hex(bins = bins, ...): Ignoring unknown parameters: `main`
Code
meanSdPlot(as.matrix(vst_otu), main ="VST (DESeq2)")
Warning in geom_hex(bins = bins, ...): Ignoring unknown parameters: `main`
Code
meanSdPlot(as.matrix(rlog_otu), main ="rlog (DESeq2)")
Warning in geom_hex(bins = bins, ...): Ignoring unknown parameters: `main`
TipWhat to look for in a meanSdPlot
The red line shows the running median of the standard deviation as a function of the rank of the mean. A flat red line indicates that variance no longer depends on mean expression — the ideal outcome of a variance-stabilizing transformation.
Observations:
Raw counts show the characteristic heteroscedastic pattern: standard deviation rises steeply with the mean, reflecting the count-data mean–variance relationship \(\text{Var}(X) \propto \mu^2\) typical of overdispersed microbiome data.
Total-sum and common-scale normalization correct for library size differences but leave the mean–variance trend largely intact. The relative abundances still exhibit strong heteroscedasticity because the nonlinear count structure is not addressed.
CLR substantially flattens the relationship. The log transformation compresses the dynamic range and the centering by the geometric mean accounts for compositionality, bringing the running median much closer to horizontal.
VST and rlog provide the strongest stabilization. The DESeq2 methods explicitly model the negative binomial dispersion–mean relationship and produce a near-flat trend across the full range of mean expression. As in the RNA-seq setting, rlog applies additional shrinkage towards zero for low-count OTUs, which can further reduce noise at the low end.
For downstream ordination and network inference on microbiome data, CLR remains the most principled choice because it respects the compositional nature of the data — VST and rlog were designed for absolute count data and treat the OTU table as if it were RNA-seq, which is a pragmatic approximation rather than a theoretically grounded one.
c) Aitchison distance and MDS
Apply the CLR transformation to the OTU table and compute the pairwise Aitchison distance matrix between samples. Then perform both a non-metric MDS (vegan::metaMDS()) and a metric MDS / PCoA (vegan::wcmdscale()) on the resulting distance matrix.
TipAitchison distance
The Aitchison distance is the natural distance measure on the simplex (compositional data). It is defined as the Euclidean distance after CLR transformation:
Because the CLR maps compositions into an unconstrained Euclidean space, ordinary Euclidean distance applied to CLR-transformed data is exactly the Aitchison distance — no further correction is needed.
i) Aitchison distance matrix
The dist() function expects an n × p matrix (samples in rows), so we transpose the p × n CLR matrix before computing distances.
Code
# clr_otu is p x n; dist() needs n x pclr_t <-t(clr_otu) # n x paitchison_dist <-dist(clr_t, method ="euclidean")# Quick sanity checkcat("Distance matrix dimensions: ", attr(aitchison_dist, "Size"), "x",attr(aitchison_dist, "Size"), "\n")
Distance matrix dimensions: 261 x 261
ii) Non-metric MDS with metaMDS
Non-metric MDS (NMDS) finds a low-dimensional configuration that preserves the rank order of dissimilarities rather than their absolute values. The goodness-of-fit is measured by stress — values below 0.2 are generally considered acceptable.
Code
set.seed(42)nmds_res <- vegan::metaMDS(aitchison_dist, k =2, autotransform =FALSE)
Metric MDS (also called Principal Coordinates Analysis, PCoA) finds the configuration that best preserves the actual dissimilarity values in least-squares sense. Setting eig = TRUE allows us to inspect how much variance each axis explains.
Code
pcoa_res <- vegan::wcmdscale(aitchison_dist, k =2, eig =TRUE)# Proportion of variance explained by each axiseig_vals <- pcoa_res$eigpct_var <-round(100* eig_vals[1:2] /sum(eig_vals[eig_vals >0]), 1)cat("PCoA axis 1:", pct_var[1], "% variance explained\n")
Both embeddings place samples in a two-dimensional space derived from the same Aitchison distance matrix, but they pursue different objectives:
NMDS
PCoA
Optimises
rank order of distances (stress)
actual distance values (least squares)
Axes
arbitrary — no variance interpretation
ordered by variance explained
Sensitivity to outliers
robust
more sensitive
Interpretation
topological structure
proportion of total variance
In this dataset the two embeddings reveal a broadly similar cloud of samples without strong discrete clustering, consistent with the continuous nature of gut microbiome variation across healthy donors. The PCoA axes provide the additional benefit of showing that the first two principal coordinates account for only a moderate fraction of total variation — a reminder that microbiome beta-diversity is inherently high-dimensional and cannot be fully captured in two axes.
Exercise 02: Visualization of microbiome data with phyloseq
In the lecture, we learned about distance calculation and Multi-Dimensional Scaling (MDS) for visualizing samples from next-generation sequencing data. In this exercise, we will learn how to perform such a workflow for microbiome data.
One early claim in microbiome research was that human gut microbiome compositions can be separated into three different classes or types, so called “enterotypes”. We will analyze this claim using human gut microbiome data and distance embeddings.
TipEnterotypes
Source: Cheng & Ning, 2019, “Stereotypes About Enterotype: the Old and New Ideas”
a) Distance functions provided by the phyloseq package
The phyloseq package incorporates many distance functions. Familiarize yourself with the available distance function using the command ?distance. How many distance functions are available? Print the list of distances in the command window. Which distances do you recognize from the lecture?
Code
# Available distance measuresdist_methods <-unlist(distanceMethodList)dist_methods
The phyloseq package includes several example datasets. We will use the “enterotype” dataset. Load the data set, and remove the OTUs that included all unassigned sequences. Compute the Bray-Curtis distance among all the samples, and apply Multi-Dimensional Scaling.
phyloseq-class experiment-level object
otu_table() OTU Table: [ 553 taxa and 280 samples ]
sample_data() Sample Data: [ 280 samples by 9 sample variables ]
tax_table() Taxonomy Table: [ 553 taxa by 1 taxonomic ranks ]
Code
# Remove taxa without a labelenterotype <-subset_taxa(enterotype, Genus !="-1")# Apply MDS on bray curtis distanceenterotype_mds <-ordinate(enterotype, method ="MDS", distance ="bray")
c) MDS plots
Plot the resulting MDS embedding in two dimensions using phyloseq function plot_ordination. Do not use any covariates for coloring the different samples. How many apparent clusters do you see?
Re-do the plot and color the samples by sequencing technology (“SeqTech”). What do you observe?
First a plot without coloring metadata.
Code
plot_ordination(enterotype, enterotype_mds, title ="MDS of enterotypes")
One could identify two main clusters, separated by the first axis. Maybe a third cluster can be formed on the left side of Axis.1 along Axis.2
Let’s color according to sequencing technology.
Code
plot_ordination(enterotype, enterotype_mds, color="SeqTech",title ="MDS of enterotype colored by sequencing technology")
Indeed, sequencing technology separates samples along Axis.1. Illumina and Sanger seem to cluster together, this might be due to the similarity in the sequencing technique.
d) Application of distance functions
Now apply all available distance functions to the data, compute the corresponding MDS embeddings, and plot the resulting embeddings.
Three distance functions (“UniFrac1”, “UniFrac2”, “DPCoA”) require a tree as additional input. Remove these distances as well as the userdefined “designdist” from your distance list prior to computation. Overall, there should be 43 distances available.
Now we can apply these distance metrics to our data.
Code
gather_ordination_data <-function(d_method, phylo_obj){# Calculate the MDS matrix using the distance ordinate_obj <-ordinate(phylo_obj, method ="MDS", distance = d_method)# Get the data for plotting plot_data <-plot_ordination(phylo_obj, ordinate_obj)$data# Add a column for distance plot_data$distance <- d_methodreturn(plot_data)}mds_plot_data <-bind_rows(lapply(dist_methods_filt, gather_ordination_data,phylo_obj=enterotype))head(mds_plot_data)
Axis.1 Axis.2 Enterotype Sample_ID SeqTech SampleID
AM.AD.1...1 0.090933362 -0.17953105 <NA> AM.AD.1 Sanger AM.AD.1
AM.AD.2...2 -0.007543528 -0.13104447 <NA> AM.AD.2 Sanger AM.AD.2
AM.F10.T1...3 0.039414739 -0.01436821 <NA> AM.F10.T1 Sanger AM.F10.T1
AM.F10.T2...4 -0.004891984 -0.05159793 3 AM.F10.T2 Sanger AM.F10.T2
DA.AD.1...5 0.211396571 -0.17160996 2 DA.AD.1 Sanger DA.AD.1
DA.AD.1T...6 0.192034295 -0.16828348 <NA> DA.AD.1T Sanger <NA>
Project Nationality Gender Age ClinicalStatus distance
AM.AD.1...1 gill06 american F 28 healthy jsd
AM.AD.2...2 gill06 american M 37 healthy jsd
AM.F10.T1...3 turnbaugh09 american F NA obese jsd
AM.F10.T2...4 turnbaugh09 american F NA obese jsd
DA.AD.1...5 MetaHIT danish F 59 healthy jsd
DA.AD.1T...6 <NA> <NA> <NA> NA <NA> jsd
In each plot, color the samples by “Enteroype”. Do you find an embedding that can discriminate the enterotypes?
Code
mds.enterotype <-ggplot(mds_plot_data, aes(x=Axis.1, y=Axis.2,color=Enterotype)) +geom_point() +facet_wrap(~ distance, scales ="free") +labs(x="Axis.1",y="Axis.2",title="Separation of samples according to difference distances")ggsave("figures/Exercise03/mds_enterotype.png", mds.enterotype, width =13.91, height =9)
It seems to me that maybe altGower, minkowski and euclidean can separate the samples by enterotype.
e) Different colors and shapes
Now color the different samples by “SeqTech”. Use different shapes for the “Enterotype” covariate. What do you observe? Discuss if/how you can ameliorate the observed effect.
However, if now color by sequencing technology we again notice that it is the main determinant of the variance between samples.
Code
mds.seqtech <-ggplot(mds_plot_data, aes(x=Axis.1, y=Axis.2, color=SeqTech)) +geom_point() +facet_wrap(~ distance, scales ="free") +labs(x="Axis.1",y="Axis.2",title="Separation of samples according to difference distances")ggsave("figures/Exercise03/mds_seqtech.png", mds.seqtech, width =13.91, height =9)
Exercise 03 (Non-linear dimensionality reduction for single-cell data)
In the lecture, we briefly touched on non-linear dimensionality reduction methods beyond classical MDS. Here, we will take a closer look at one prominent example: t-distributed Stochastic Neighborhood Embedding (t-SNE). The MSMB book provides an excellent workflow on the Moignard cell data.
NoteAbout the Moignard et al. (2015) dataset
This dataset comes from Moignard et al. (2015), Nature Biotechnology and profiles early blood development (haematopoiesis) in the mouse embryo. It contains single-cell qPCR expression measurements of 18 transcription factors across 3,934 cells with blood-forming potential, collected at four developmental time points (E7.0–E8.5). The cells capture the transition from mesoderm to blood progenitors and span several known populations (primitive streak, head fold, GFP⁺/GFP⁻ sorted fractions). Because the gene panel is small and biologically curated, the dataset is well suited for learning trajectory inference and network reconstruction methods without the overhead of full-transcriptome preprocessing.
Read and reproduce all steps in Chapter 9.5.1 and 9.5.2 of the MSMB book. Here, we start with classical MDS and then extend the analysis to t-SNE embeddings.
NoteBackground: Metric vs. Non-metric MDS
Multidimensional Scaling (MDS) embeds objects into a low-dimensional space such that pairwise distances in the embedding approximate the original dissimilarities. Metric MDS (including PCoA) preserves the numerical values of the dissimilarities — it assumes the input distances carry quantitative meaning. Non-metric MDS (NMDS) only preserves their rank order, fitting a monotone transformation via isotonic regression. This makes NMDS more robust when the relationship between ecological dissimilarity and the chosen distance measure is potentially non-linear, which is why it has become the default ordination method in community ecology.
ce1Mds <-cmdscale(dist_l1, k =20, eig =TRUE)ce2Mds <-cmdscale(dist_euclid, k =20, eig =TRUE)
Code
# Variance of each componentperc1 <-round(100*sum(ce1Mds$eig[1:2]) /sum(ce1Mds$eig))perc2 <-round(100*sum(ce2Mds$eig[1:2]) /sum(ce2Mds$eig))
Code
as_tibble(ce1Mds$points[, 1:2]) |>setNames(c("PCo1", "PCo2")) |>ggplot(aes(x = PCo1, y = PCo2, color = cellt)) +geom_point(alpha =0.6) +scale_colour_manual(values = colsn) +labs(title =paste0("MDS with Manhattan distance (", perc1, "% variance in PCo1+2)"),color ="Cell type") +theme_bw()
Warning: The `x` argument of `as_tibble.matrix()` must have unique column names if
`.name_repair` is omitted as of tibble 2.0.0.
ℹ Using compatibility `.name_repair`.
Code
as_tibble(ce2Mds$points[, 1:2]) |>setNames(c("PCo1", "PCo2")) |>ggplot(aes(x = PCo1, y = PCo2, color = cellt)) +geom_point(alpha =0.6) +scale_colour_manual(values = colsn) +labs(title =paste0("MDS with Euclidean distance (", perc2, "% variance in PCo1+2)"),color ="Cell type") +theme_bw()
Both MDS embeddings reveal a similar underlying structure: The most striking feature is the clear isolation of 4SG (red) in the upper left, well separated from all other populations. The remaining four populations form a curved, arc-like structure with considerable overlap — PS and NP cluster together in the lower left, while HF and 4SFG- spread broadly to the right without clean boundaries between them. A complete separation of all five populations in 2D is not achieved.
ii) t-SNE
Tipt-SNE — core idea
t-SNE models pairwise similarities in the high-dimensional space as a probability distribution (Gaussian kernel) and seeks a low-dimensional embedding whose similarity distribution (Student-t kernel) matches it as closely as possible. The heavy-tailed t-distribution in the low-dimensional space addresses the crowding problem: dissimilar points are pushed further apart than a Gaussian would allow.
The cost function is the KL-divergence between the two distributions, minimised via gradient descent.
Compared to MDS, t-SNE produces sharper local cluster structure. Most notably, 4SG (red) forms two compact, well-isolated islands in the upper left — a clearer separation than in the MDS embedding. However, the remaining four populations (4SFG-, HF, NP, PS) still overlap considerably in the large right-hand cloud, suggesting that these cell states are not cleanly separable in 2D regardless of the method. This reflects the continuous nature of the differentiation trajectory: PS → NP → HF → 4SFG- represents a gradual transition without sharp boundaries, while 4SG represents a more distinct terminal state.
b) The perplexity parameter
The t-SNE comprises a parameter called perplexity. What is the effect of this parameter? Is there a visually optimal parameter setting?
TipWhat is perplexity?
The perplexity \(\text{Perp}\) controls the effective number of neighbours each point considers. Formally it is defined as \(2^{H(P_i)}\), where \(H(P_i)\) is the Shannon entropy of the conditional probability distribution over the neighbours of point \(i\). The bandwidth of the Gaussian kernel is adjusted per point so that the target perplexity is achieved.
Low perplexity → narrow kernel → only the immediate neighbourhood matters → local structure dominates, global structure is lost
High perplexity → wide kernel → many neighbours contribute → more global structure is captured, local detail blurs
The choice of perplexity depends largely on the dataset size:
For smaller datasets, a smaller perplexity is typically sufficient (e.g., 5-30).
For larger datasets, a larger perplexity may be more appropriate to capture more of the global structure (e.g., 30-50 or higher).
Perplexity 5: The embedding is fragmented and noisy — the neighbourhood is so narrow that the global structure breaks down into scattered, unreliable clusters.
Perplexity 15–30: Good balance between local and global structure. 4SG (red) and PS (blue) are clearly separated from the remaining populations, while HF, NP and 4SFG- remain partially overlapping — consistent with their gradual differentiation trajectory.
Perplexity 50–70: The embedding becomes more compact overall, but the main cluster boundaries are preserved. Contrary to common expectation, very high perplexity does not dissolve structure here — it simply compresses the layout.
Perplexity 15–30 appears most interpretable for this dataset, offering a stable separation of the distinct terminal state 4SG while honestly representing the continuous overlap among the remaining populations.
c) UMAP with different distance metrics
Revisit the Seurat UMAP pipeline from the previous exercise sheet. Rerun the FindNeighbors() step with annoy.metric = "cosine" and annoy.metric = "manhattan" and recompute the UMAP embedding for each. Plot the three embeddings side by side, coloring cells by cluster identity.
We start from the pbmc3k dataset and reproduce the standard Seurat preprocessing pipeline before computing UMAP embeddings under three different distance metrics.
UMAP first constructs a k-nearest-neighbour graph in the high-dimensional space. The choice of distance metric determines which points are considered neighbours — and therefore the topology of the graph that UMAP subsequently maps to 2D.
Euclidean: Sensitive to the absolute magnitude of features; the default in Seurat.
Cosine: Measures only the angle between vectors, ignoring magnitude. Robust to residual sequencing depth differences — often preferred for scRNA-seq.
Manhattan: Sums absolute differences; more robust to individual highly expressed genes than Euclidean distance.
We now wrap the FindNeighbors + RunUMAP steps into a small helper function so we can cleanly apply all three metrics without code repetition.
The global topology — which clusters are adjacent and how many exist — remains broadly consistent across all three metrics. Cluster 3 (green) is isolated at the top and cluster 1 (orange) appears on the right in all three embeddings.
Euclidean and Manhattan yield the most similar layouts: both produce an elongated structure with well-separated, compact clusters. This is expected since both metrics measure absolute distances and differ only in how they aggregate differences across dimensions.
Cosine deviates most strongly — clusters are more compressed and the overall structure appears more condensed. By ignoring vector magnitude and measuring only the angle between cells, cosine treats cells with similar relative expression profiles as close neighbours regardless of their total expression level, which fundamentally alters the neighbourhood structure compared to the magnitude-based metrics.
d) t-SNE vs. UMAP — a comparison
Compute a t-SNE embedding of the PBMC dataset using Seurat’s function. Next, compare this result with the UMAP embedding generated in . How do the two methods differ visually in terms of cluster separation and the preservation of global structure?”
Cluster separation: Both methods recover the same well-known PBMC cell types with comparable clarity. t-SNE produces slightly more compact clusters with more whitespace between them, while UMAP allows a bit more scatter at cluster boundaries — but the differences are minor. Both methods clearly separate B cells, NK cells, monocytes, and T cell subsets.
T cell overlap: Neither method cleanly separates Naive CD4 T from Memory CD4 T cells. This is not a limitation of the embedding. These populations genuinely overlap in transcriptomic space and are difficult to distinguish even with marker-based gating.
Global layout: UMAP places the two monocyte subsets (CD14+, FCGR3A+) closer together and keeps the T cell populations in a connected region, which is consistent with lineage relationships. t-SNE spreads these populations further apart, but the overall topology is similar. The global structure differences are subtle and should not be over-interpreted.
Distance metric effect (UMAP only): The three UMAP distance metrics produce near-identical embeddings. This is expected when the input is PCA space, where the components are already orthogonal and variance-scaled. For this dataset, metric choice is essentially irrelevant.
Practical takeaway: For well-separated populations like PBMCs, UMAP and t-SNE give very similar results. The methodological differences between them become more consequential for datasets with continuous differentiation trajectories or subtle population boundaries — neither of which applies here.