Exercise 1 (Normalization and variance-stabilizing transformations of sequencing data)
In the lecture, we learned about different normalization and variance-stabilizing transformation schemes for next-generation sequencing data. Here, we implement different normalization schemes and apply them to RNA-seq data from the pasilla package.
Take a quick look at the pasilla dataset. How many samples are there? How many genes? Anything about the count values that will be important to consider down the line? Compare the library sizes (total counts) across samples. Is there a need for normalization?
TipPasilla dataset
These data are from an experiment on Drosophila melanogaster cell cultures investigating the effect of RNAi knock-down of the splicing factor pasilla (Brooks et al. 2011) on the transcriptome.
The dataset contains 14599 genes and 7 samples — a classic high-dimensional setting with p >> n. An important consideration is the high fraction of zero counts (~28 %), which will matter for log-transformations and require a pseudo-count.
Code
lib_sizes <-colSums(counts)lib_df <-data.frame(sample =names(lib_sizes), lib_size = lib_sizes)ggplot(lib_df, aes(x = sample, y = lib_size /1e6, fill = sample)) +geom_col(show.legend =FALSE, width =0.65, color ="grey30") +geom_hline(yintercept =mean(lib_sizes) /1e6,linetype ="dashed", color ="firebrick", linewidth =0.8) +annotate("text", x =0.6, y =mean(lib_sizes) /1e6+0.3,label ="mean", color ="firebrick", hjust =0, size =3.5) +labs(title ="Library sizes per sample",x =NULL, y ="Total counts (millions)") +theme_bw(base_size =13) +theme(axis.text.x =element_text(angle =45, hjust =1))
Library sizes differ up to ~2-fold across samples (e.g., untreated2 is sequenced considerably more deeply than untreated3). Normalization is therefore necessary before any cross-sample comparison.
b) Mean–variance relationship
What is the relationship between a gene’s average number of counts and its variance? (See meanSdPlot vignette in the vsn package.)
Warning: `aes_string()` was deprecated in ggplot2 3.0.0.
ℹ Please use tidy evaluation idioms with `aes()`.
ℹ See also `vignette("ggplot2-in-packages")` for more information.
ℹ The deprecated feature was likely used in the vsn package.
Please report the issue to the authors.
Warning in geom_hex(bins = bins, ...): Ignoring unknown parameters: `main`
The standard deviation increases strongly with the mean — a hallmark of overdispersed count data following a Negative Binomial distribution. The variance is not constant; highly expressed genes dominate purely due to their large absolute variance. Variance stabilization is needed before PCA or clustering.
c) Median of ratios normalization (DESeq2)
Perform normalization using the median of ratios method as implemented in DESeq2. Extract and plot the size factors. What do you notice? Which samples needed the strongest correction?
sf_df <-data.frame(sample =names(sf), size_factor = sf)ggplot(sf_df, aes(x = sample, y = size_factor, fill = sample)) +geom_col(show.legend =FALSE, width =0.65, color ="grey30") +geom_hline(yintercept =1, linetype ="dashed",color ="firebrick", linewidth =0.8) +labs(title ="DESeq2 size factors (median of ratios)",x =NULL, y ="Size factor") +theme_bw(base_size =13) +theme(axis.text.x =element_text(angle =45, hjust =1))
Size factors mirror the library sizes observed in (a). untreated2 has the largest size factor (> 1, counts scaled down) and untreated3 the smallest (< 1, counts scaled up), making them the samples that required the strongest correction. treated3 also deviates noticeably from 1.
Code
counts_norm <-counts(dds, normalized =TRUE)
d) Heatmaps: raw vs. normalized counts
Use pheatmap to create heatmaps of the 50 most variable genes (by row variance), once for raw counts and once for median-of-ratios normalized counts. How do the two heatmaps differ?
Take a close look at these two heatmaps to understand why data preprocessing is the backbone of bioinformatics. In the first heatmap, we visualize the top 50 most variable genes using raw counts. At a superficial glance, the data appears “clean”: there are distinct red and blue blocks and clear-cut clusters. However, you will notice that the samples are primarily clustering by sequencing type (cyan = paired-end, purple = single-read), rather than the actual biological condition. Treated and untreated samples are scattered inconsistently across the clusters. This is a textbook example of a batch effect; the single biggest source of variation in this raw data is a technical artifact of the library preparation, not the biology of the Pasilla knockdown.
The situation changes dramatically after applying size-factor normalization via DESeq2. In the second heatmap, the clustering structure is reorganized. While the technical “batch” (sequencing type) still exerts a strong influence on the top-level hierarchy, the samples now group much more coherently by their biological condition (treated vs. untreated) within those technical constraints. Additionally, the parreterns are far more heterogenous as the color scale has widened and individual genes now show condition-dependent patters that cut both sequencing types. By correcting for library size, the dominant technical noise is reduced, allowing the true biological signal of the experiment to emerge.
e) Variance-stabilizing transformations in DESeq2
Extend your normalization pipeline by applying variance-stabilizing transformations (VSTs) from DESeq2. What are the different available options?
DESeq2 offers two VST strategies:
Function
Method
Notes
vst()
Parametric VST
Fast; recommended for n ≥ 30
varianceStabilizingTransformation()
Same, slower full model
More accurate for small n
rlog()
Regularized log transformation
Better for n < 30; slower; applies shrinkage to low-count genes
Re-examine the mean–variance relationship in VST-normalized data. How has it changed compared to (b)?
Code
par(mfrow =c(1, 2))meanSdPlot(assay(vsd), ranks =TRUE, main ="Mean-SD: VST")
Warning in geom_hex(bins = bins, ...): Ignoring unknown parameters: `main`
Code
meanSdPlot(assay(rld), ranks =TRUE, main ="Mean-SD: rlog")
Warning in geom_hex(bins = bins, ...): Ignoring unknown parameters: `main`
Code
par(mfrow =c(1, 1))
The strong mean-dependent trend in the standard deviation is largely removed. The SD is now approximately flat across the expression range — the desired variance-stabilized behaviour. The rlog additionally shrinks low-count genes towards a global prior, which can slightly over-shrink in very low-expression regimes.
g) Heatmap of VST-transformed data
Create a heatmap of the 50 most variable genes using the VST-transformed data. How does this compare to (d)?
Finally, we arrive at the third stage of our analysis: the Variance Stabilizing Transformation (VST). While size-factor normalization corrected for library size, it did not address the heteroskedasticity inherent in RNA-Seq data. After applying the VST, the clustering hierarchy undergoes a definitive transformation: For the first time, biological condition (treated vs. untreated) replaces technical artifacts as the primary driver of the top-level clusters. All three treated samples are now grouped together on one major branch, while all four untreated samples reside on the other.
By stabilizing the variance across the entire dynamic range, we have effectively leveled the playing field for every gene, regardless of its expression level. The “batch effect” of the sequencing type is no longer strong enough to disrupt the biological signal. This visualization confirms that our preprocessing has been successful: we have moved beyond measuring the settings of the sequencing machine and are now accurately looking at the transcriptional changes caused by the Pasilla knockdown. This refined data is now ready for high-level statistical testing.
h) PCA: raw vs. VST-normalized counts
Perform a PCA using un-normalized counts and compare it to the PCA using VST-normalized counts. How are the samples separated in PC1 and PC2? According to which meta-variables?
We can see that when we do a PCA of the raw counts allows us to see that most of the variance (separation along PC1) is explained by the sequencing type.
The transformation is striking when we look at the VST-normalized PCA. After variance stabilization, the “biological map” finally aligns with our experimental design:
PC1: The X-axis now separates the samples by condition. All blue (untreated) samples have migrated to the left, while all red (treated) samples have shifted to the right. This confirms that the treatment effect of the Pasilla knockdown is now the primary source of variation in our dataset.
PC2: The Y-axis now captures the remaining technical variation. The “triangles” are at the top and the “circles” are at the bottom.
By using VST, we have effectively “rotated” the perspective of our data. We have promoted the biological signal to the primary axis (PC1) and relegated the technical artifacts to the secondary axis (PC2). This PCA provides the visual proof that our data is now clean, consistent, and ready for further analysis.
Exercise 2 (Variance Stabilizing Transformation in Single-Cell RNA-seq)
We use the PBMC3k dataset (~2,700 peripheral blood mononuclear cells, 10x Genomics) and the Seurat framework to compare log1p normalization, a manual data-driven VST, and SCTransform() — Seurat’s implementation of a VST based on a regularized Negative Binomial model.
An object of class Seurat
13714 features across 2700 samples within 1 assay
Active assay: RNA (13714 features, 0 variable features)
2 layers present: counts, data
a) Mean–variance relationship in raw counts
Compute per-gene mean and variance from the raw count matrix and plot them on a log–log scale. What do you observe? Why is this problematic for PCA or clustering?
Code
X <- pbmc3k@assays$RNA@countsgene_means <-rowMeans(X)gene_vars <-apply(X, 1, var)keep <- gene_means >0gene_means_filt <- gene_means[keep]gene_vars_filt <- gene_vars[keep]cat("Genes with mean > 0:", sum(keep), "out of", length(keep), "\n")
Variance increases with the mean and faster than linearly (log–log slope > 1). This overdispersion means that highly expressed genes dominate PCA and clustering purely due to their large variance, even if they carry no meaningful biological signal relative to lowly expressed genes.
b) Negative Binomial trend lines
The NB distribution models overdispersed count data via Var(Y) = μ + αμ². Overlay NB trend lines for α ∈ {0.1, 0.5, 1.0} onto your plot from (a). Which α best describes the data? What does α = 0 correspond to?
The bulk of the point cloud lies between the α = 0.1 and α = 0.5 curves, with α ≈ 0.1–0.5 describing the data best for moderately expressed genes; highly expressed genes scatter above all curves, suggesting α > 1 for some. α = 0 corresponds to the Poisson distribution (Var = μ), which clearly underestimates the observed overdispersion.
c) log1p normalization
Apply Seurat’s standard NormalizeData() pipeline (LogNormalize, scale factor 10,000) and repeat the mean–variance plot. Is the mean–variance dependence resolved? What does the fixed pseudo-count of 1 imply about the assumed overdispersion α?
The log1p transformation reduces the extreme scaling but does not fully resolve the mean–variance dependence — variance still rises for lowly expressed genes and then falls. The VST formula gives pseudo-count y₀ = 1/(4α); setting y₀ = 1 (as in log1p) implies α = 1/4 = 0.25 — a single global overdispersion assumed equal for all genes, which is too coarse.
d) Manual data-driven VST
Define size factors sᵢ = ∑g yᵢg / 10,000 and relative expression qg = ∑i yᵢg / ∑i sᵢ. Estimate gene-specific dispersion θ̂g via fitdistr, then apply the gene-specific and global VST.
NoteWhy does µ_ig = s_i · q_g serve as the expected count?
To estimate the dispersion parameter α, we need a baseline expectation for each count y_ig that captures only technical variation, not biological signal. We construct this by decomposing the expected count into two components: a cell-specific size factor s_i, which reflects how deeply cell i was sequenced, and a gene-specific relative expression q_g, which captures how abundant gene g is on average across all cells.
Under the assumption that all cells express each gene at the same underlying rate, any difference in observed counts between cells is purely due to differences in sequencing depth. The expected count µ_ig = s_i · q_g is therefore the count we would predict for gene g in cell i if there were no biological variation — only a proportional scaling by library size.
This is deliberately a null model: by fixing µ_ig and estimating only θ_g = 1/α_g from the negative binomial distribution, we ask how much the observed counts deviate from this baseline expectation. A gene with high dispersion is one whose counts vary across cells more than sequencing depth alone can explain — which is precisely the biological signal we want to capture with the VST.
i) Sequencing depths and size factors
Code
Y <-t(as.matrix(pbmc3k@assays$RNA@counts)) # cells x genessequencing_depths <-rowSums(Y)size_factors <- sequencing_depths /1e4cat("Median sequencing depth:", median(sequencing_depths), "\n")
hist(size_factors, breaks =60, col ="steelblue", border ="white",main ="Distribution of size factors across cells",xlab ="Size factor sᵢ", ylab ="Number of cells")abline(v =median(size_factors), col ="firebrick", lwd =2, lty =2)
The median depth is around 2,000 counts/cell and size factors are concentrated near 0.2. The right skew reflects a small population of deeply sequenced cells.
ii) Gene-specific VST via fitdistr
Code
# Expected counts: outer product of size factors and relative expressionq_g <-colSums(Y) /sum(size_factors)keep <- q_g >0Y_filt <- Y[, keep]mu <- size_factors %o% q_g[keep] # cells x genes# Estimate per-gene theta = 1/alpha via fitdistr (fixed mu)n_genes <-ncol(Y_filt)n_cores <-max(1, parallel::detectCores() -1)cat("Fitting NB dispersion for", n_genes, "genes on", n_cores, "cores...\n")
Fitting NB dispersion for 13714 genes on 11 cores...
p_log + p_vst_specific +plot_annotation(title ="Mean–Variance: log1p vs. Gene-Specific VST",theme =theme(plot.title =element_text(size =14, face ="bold")))
`geom_smooth()` using formula = 'y ~ x'
`geom_smooth()` using formula = 'y ~ x'
The gene-specific VST substantially flattens the mean–variance curve, with the loess trend hugging the dashed reference line (Var = 1) much more closely than after log1p.
X_vst_global <-log(sweep(Y_gs, 1, sf_v, "/") + y0_global)p_vst_global <-ggplot(data.frame(mean =colMeans(X_vst_global),var =apply(X_vst_global, 2, var)),aes(x = mean, y = var)) +geom_point(size =0.8, alpha =0.35, color ="darkorange") +geom_smooth(method ="loess", color ="firebrick", linewidth =1.2) +geom_hline(yintercept =1, linetype ="dashed", color ="grey40") +labs(title ="Global VST",subtitle =paste0("y₀ = median(θ̂_g)/4 = ", round(y0_global, 3)),x ="Mean after VST", y ="Variance after VST") +theme_bw(base_size =13)p_log + p_vst_specific + p_vst_global +plot_annotation(title ="Mean–Variance: log1p | Gene-Specific VST | Global VST",theme =theme(plot.title =element_text(size =13, face ="bold")) )
`geom_smooth()` using formula = 'y ~ x'
`geom_smooth()` using formula = 'y ~ x'
`geom_smooth()` using formula = 'y ~ x'
Replacing gene-specific pseudo-counts with a single global median θ̂/4 partially sacrifices the stabilization for genes whose true dispersion deviates strongly from the median. Highly expressed genes (large θ in the data) tend to be over-shrunk; rare lowly-expressed genes are under-shrunk. The global VST is simpler but less accurate than the gene-specific version.
e) SCTransform
Apply SCTransform() to the PBMC dataset. Plot the relationship between geometric mean of raw counts and Pearson residual variance, and place it alongside the plots from (c) and (d-ii).
# Use gene-level summary statistics stored by SCTransform:# gmean = geometric mean of raw counts per gene# residual_variance = variance of Pearson residuals per genesct_attrs <- pbmc_vst@assays$SCT@SCTModel.list[[1]]@feature.attributesp_sct <-ggplot(sct_attrs, aes(x =log10(gmean), y = residual_variance)) +geom_point(size =0.8, alpha =0.3, color ="black") +geom_hline(yintercept =1, linetype ="dashed", color ="grey40") +geom_density_2d(color ="steelblue", linewidth =0.4, na.rm =TRUE) +labs(title ="SCTransform (Pearson residuals)",subtitle ="Geometric mean of raw counts vs. residual variance",x ="log10(geometric mean raw counts)",y ="Variance of Pearson residuals") +theme_bw(base_size =13)(p_log + p_vst_specific) / p_sct +plot_annotation(title ="Mean–Variance Relationship: Three Normalization Strategies",caption ="Note: x-axes are not comparable — log1p/VST show transformed means; SCTransform shows log10(geometric mean of raw counts).",theme =theme(plot.title =element_text(size =13, face ="bold"),plot.caption =element_text(size =9, color ="grey40")) )
`geom_smooth()` using formula = 'y ~ x'
`geom_smooth()` using formula = 'y ~ x'
i) Characterization of mean–variance across panels
In a successfully variance-stabilized dataset, Var(zᵢg) ≈ 1 across all values of the geometric mean ḡg. The three panels show a progression: log1p leaves a pronounced mean-dependent trend; the gene-specific VST largely flattens it; SCTransform achieves the tightest stabilization with the trend closest to the dashed reference line Var = 1, because it uses gene-specific regularized NB regression.
ggplot(fa_clean, aes(x = y0_g)) +geom_histogram(bins =80, fill ="steelblue", color ="white", alpha =0.8) +geom_vline(xintercept = y0_global, color ="firebrick",linewidth =1.2, linetype ="dashed") +annotate("text", x = y0_global, y =Inf, vjust =2, hjust =-0.15,label =paste0("Global ŷ₀ = ", round(y0_global, 3)),color ="firebrick", size =4) +scale_x_log10() +labs(title ="Distribution of Gene-Specific Pseudo-counts (SCTransform)",subtitle ="y₀_g = θ̂_g / 4; dashed line = global MLE estimate",x ="Pseudo-count y₀_g (log10 scale)", y ="Number of genes") +theme_bw(base_size =13)
SCTransform applies kernel smoothing to regularize per-gene θ̂g estimates as a function of mean expression. This means that lowly expressed genes (noisy individual estimates) are pooled towards the global trend, preventing extreme pseudo-counts driven by estimation noise. The SCTransform estimates differ most from the unregularized per-gene estimates for lowly expressed genes, precisely where individual MLE fits are most unreliable due to the small number of non-zero counts.
f) UMAP comparison across normalization strategies
Run the full Seurat pipeline (PCA → FindNeighbors → FindClusters (resolution 0.5) → RunUMAP) for all three approaches and compare the UMAPs side by side.
Warning: The default method for RunUMAP has changed from calling Python UMAP via reticulate to the R-native UWOT using the cosine metric
To use Python UMAP via reticulate, set umap.method to 'umap-learn' and metric to 'correlation'
This message will be shown once per session
log1p normalization (left) produces a compressed embedding with poorly separated clusters — several cell types overlap in the center, and only clusters 2 and 7 are clearly isolated. Manual gene-specific VST (middle), applying a per-gene variance-stabilizing transformation with estimated θ₀, substantially improves cluster separation: all 9 clusters are well-resolved and more evenly distributed across the UMAP space. SCTransform (right) yields a similar improvement, with clearly separated and compact clusters, broadly consistent with the manual VST result. Overall, both VST-based approaches outperform log1p normalization in resolving distinct cell populations, highlighting the importance of appropriate variance stabilization for count data prior to dimensionality reduction.
Kabsch alignment finds the optimal rotation matrix that minimizes the Root Mean Square Deviation (RMSD) between two sets of paired points. Applied to UMAP embeddings, it rotates one embedding to best match the other in a shared coordinate space — allowing direct visual comparison of cluster structures across normalization methods without artifacts from arbitrary UMAP orientation.
Following Kabsch alignment, the log1p (blue) and SCTransform (red) embeddings are superimposed in a common coordinate space, allowing direct visual comparison of their cluster structures. Where the two methods agree on cell population identity, blue and red points co-localize tightly — this is visible for several clusters in the lower-left region and the upper-center area. However, notable mismatches remain: some clusters are spatially offset between the two methods, and several SCTransform clusters appear more compact and better separated than their log1p counterparts. A few isolated red points with no corresponding blue neighbors (e.g., the outlier near UMAP1 = 5, UMAP2 = 8) suggest cells whose neighborhood structure differs substantially between normalizations. Overall, the alignment reveals that while the two methods recover broadly similar global topology, SCTransform produces tighter, more internally consistent clusters, consistent with its superior variance stabilization.