Analysis of high-dimensional biological data

Exercise sheet 2 — Solutions
Author
Published

May 19, 2026

Install packages required for this exercise sheet

Code
packages <- c("DESeq2", "pasilla", "vsn", "pheatmap",
              "dplyr", "ggplot2", "tidyr", "ggrepel",
              "Seurat", "SeuratData", "patchwork", "Matrix",
              "scales", "MASS")

not_installed <- packages[!(packages %in% installed.packages()[, "Package"])]
if (length(not_installed) > 0) {
  BiocManager::install(not_installed)
}
  • R version: 4.4.2 (2024-10-31)
  • BioC version: 3.20

Exercise 1 (Normalization and variance-stabilizing transformations of sequencing data)

In the lecture, we learned about different normalization and variance-stabilizing transformation schemes for next-generation sequencing data. Here, we implement different normalization schemes and apply them to RNA-seq data from the pasilla package.

Code
library(DESeq2)
library(pasilla)
library(vsn)
library(pheatmap)
library(dplyr)
library(ggplot2)
library(ggrepel)
library(tidyr)

# Resolve namespace conflicts (MASS::select, Seurat::select, etc.)
select <- dplyr::select
filter <- dplyr::filter

a) The pasilla dataset

Take a quick look at the pasilla dataset. How many samples are there? How many genes? Anything about the count values that will be important to consider down the line? Compare the library sizes (total counts) across samples. Is there a need for normalization?

TipPasilla dataset

These data are from an experiment on Drosophila melanogaster cell cultures investigating the effect of RNAi knock-down of the splicing factor pasilla (Brooks et al. 2011) on the transcriptome.

Code
fn <- system.file("extdata", "pasilla_gene_counts.tsv",
                  package = "pasilla", mustWork = TRUE)
counts <- read.csv(fn, sep = "\t", row.names = "gene_id")

cat("Dimensions (genes x samples):", nrow(counts), "x", ncol(counts), "\n")
Dimensions (genes x samples): 14599 x 7 
Code
cat("Fraction of zero counts:", round(mean(counts == 0) * 100, 1), "%\n")
Fraction of zero counts: 27.6 %
Code
head(counts)
            untreated1 untreated2 untreated3 untreated4 treated1 treated2
FBgn0000003          0          0          0          0        0        0
FBgn0000008         92        161         76         70      140       88
FBgn0000014          5          1          0          0        4        0
FBgn0000015          0          2          1          2        1        0
FBgn0000017       4664       8714       3564       3150     6205     3072
FBgn0000018        583        761        245        310      722      299
            treated3
FBgn0000003        1
FBgn0000008       70
FBgn0000014        0
FBgn0000015        0
FBgn0000017     3334
FBgn0000018      308

The dataset contains 14599 genes and 7 samples — a classic high-dimensional setting with p >> n. An important consideration is the high fraction of zero counts (~28 %), which will matter for log-transformations and require a pseudo-count.

Code
lib_sizes <- colSums(counts)
lib_df    <- data.frame(sample = names(lib_sizes), lib_size = lib_sizes)

ggplot(lib_df, aes(x = sample, y = lib_size / 1e6, fill = sample)) +
  geom_col(show.legend = FALSE, width = 0.65, color = "grey30") +
  geom_hline(yintercept = mean(lib_sizes) / 1e6,
             linetype = "dashed", color = "firebrick", linewidth = 0.8) +
  annotate("text", x = 0.6, y = mean(lib_sizes) / 1e6 + 0.3,
           label = "mean", color = "firebrick", hjust = 0, size = 3.5) +
  labs(title = "Library sizes per sample",
       x = NULL, y = "Total counts (millions)") +
  theme_bw(base_size = 13) +
  theme(axis.text.x = element_text(angle = 45, hjust = 1))

Library sizes differ up to ~2-fold across samples (e.g., untreated2 is sequenced considerably more deeply than untreated3). Normalization is therefore necessary before any cross-sample comparison.


b) Mean–variance relationship

What is the relationship between a gene’s average number of counts and its variance? (See meanSdPlot vignette in the vsn package.)

Code
meanSdPlot(as.matrix(counts + 1), ranks = TRUE,
           main = "Mean-SD plot — raw counts (+ 1)")
Warning: `aes_string()` was deprecated in ggplot2 3.0.0.
ℹ Please use tidy evaluation idioms with `aes()`.
ℹ See also `vignette("ggplot2-in-packages")` for more information.
ℹ The deprecated feature was likely used in the vsn package.
  Please report the issue to the authors.
Warning in geom_hex(bins = bins, ...): Ignoring unknown parameters: `main`

The standard deviation increases strongly with the mean — a hallmark of overdispersed count data following a Negative Binomial distribution. The variance is not constant; highly expressed genes dominate purely due to their large absolute variance. Variance stabilization is needed before PCA or clustering.


c) Median of ratios normalization (DESeq2)

Perform normalization using the median of ratios method as implemented in DESeq2. Extract and plot the size factors. What do you notice? Which samples needed the strongest correction?

Code
annotationFile <- system.file("extdata", "pasilla_sample_annotation.csv",
                               package = "pasilla", mustWork = TRUE)
pasillaSampleAnno <- read.csv(annotationFile) %>%
  mutate(file = gsub("fb$", "", file)) %>%
  select(file, condition, type)

rownames(pasillaSampleAnno) <- pasillaSampleAnno$file
pasillaSampleAnno           <- pasillaSampleAnno[colnames(counts), ]
pasillaSampleAnno
                 file condition        type
untreated1 untreated1 untreated single-read
untreated2 untreated2 untreated single-read
untreated3 untreated3 untreated  paired-end
untreated4 untreated4 untreated  paired-end
treated1     treated1   treated single-read
treated2     treated2   treated  paired-end
treated3     treated3   treated  paired-end
Code
dds <- DESeqDataSetFromMatrix(
  countData = counts,
  colData   = pasillaSampleAnno,
  design    = ~ condition
)
Warning in DESeqDataSet(se, design = design, ignoreRank): some variables in
design formula are characters, converting to factors
Code
dds <- estimateSizeFactors(dds)

sf <- sizeFactors(dds)
cat("Size factors:\n"); print(round(sf, 4))
Size factors:
untreated1 untreated2 untreated3 untreated4   treated1   treated2   treated3 
    1.1383     1.7930     0.6495     0.7517     1.6356     0.7613     0.8327 
Code
sf_df <- data.frame(sample = names(sf), size_factor = sf)

ggplot(sf_df, aes(x = sample, y = size_factor, fill = sample)) +
  geom_col(show.legend = FALSE, width = 0.65, color = "grey30") +
  geom_hline(yintercept = 1, linetype = "dashed",
             color = "firebrick", linewidth = 0.8) +
  labs(title = "DESeq2 size factors (median of ratios)",
       x = NULL, y = "Size factor") +
  theme_bw(base_size = 13) +
  theme(axis.text.x = element_text(angle = 45, hjust = 1))

Size factors mirror the library sizes observed in (a). untreated2 has the largest size factor (> 1, counts scaled down) and untreated3 the smallest (< 1, counts scaled up), making them the samples that required the strongest correction. treated3 also deviates noticeably from 1.

Code
counts_norm <- counts(dds, normalized = TRUE)

d) Heatmaps: raw vs. normalized counts

Use pheatmap to create heatmaps of the 50 most variable genes (by row variance), once for raw counts and once for median-of-ratios normalized counts. How do the two heatmaps differ?

Code
select_top50 <- function(mat) {
  rv  <- rowVars(as.matrix(mat))
  idx <- order(rv, decreasing = TRUE)[1:50]
  mat[idx, , drop = FALSE]
}

top50_raw  <- select_top50(counts)
top50_norm <- select_top50(counts_norm)

anno_col <- pasillaSampleAnno %>% select(condition, type)
Code
pheatmap(
  log2(top50_raw + 1),
  annotation_col = anno_col,
  scale          = "row",
  show_rownames  = FALSE,
  cluster_cols   = TRUE,
  cluster_rows   = TRUE,
  color          = colorRampPalette(c("navy", "white", "firebrick3"))(100),
  main           = "Top 50 variable genes — raw counts (log2+1, row-scaled)",
  fontsize       = 10,
  border_color   = NA
)

Code
pheatmap(
  log2(top50_norm + 1),
  annotation_col = anno_col,
  scale          = "row",
  show_rownames  = FALSE,
  cluster_cols   = TRUE,
  cluster_rows   = TRUE,
  color          = colorRampPalette(c("navy", "white", "firebrick3"))(100),
  main           = "Top 50 variable genes — normalized counts (log2+1, row-scaled)",
  fontsize       = 10,
  border_color   = NA
)

Take a close look at these two heatmaps to understand why data preprocessing is the backbone of bioinformatics. In the first heatmap, we visualize the top 50 most variable genes using raw counts. At a superficial glance, the data appears “clean”: there are distinct red and blue blocks and clear-cut clusters. However, you will notice that the samples are primarily clustering by sequencing type (cyan = paired-end, purple = single-read), rather than the actual biological condition. Treated and untreated samples are scattered inconsistently across the clusters. This is a textbook example of a batch effect; the single biggest source of variation in this raw data is a technical artifact of the library preparation, not the biology of the Pasilla knockdown.

The situation changes dramatically after applying size-factor normalization via DESeq2. In the second heatmap, the clustering structure is reorganized. While the technical “batch” (sequencing type) still exerts a strong influence on the top-level hierarchy, the samples now group much more coherently by their biological condition (treated vs. untreated) within those technical constraints. Additionally, the parreterns are far more heterogenous as the color scale has widened and individual genes now show condition-dependent patters that cut both sequencing types. By correcting for library size, the dominant technical noise is reduced, allowing the true biological signal of the experiment to emerge.


e) Variance-stabilizing transformations in DESeq2

Extend your normalization pipeline by applying variance-stabilizing transformations (VSTs) from DESeq2. What are the different available options?

DESeq2 offers two VST strategies:

Function Method Notes
vst() Parametric VST Fast; recommended for n ≥ 30
varianceStabilizingTransformation() Same, slower full model More accurate for small n
rlog() Regularized log transformation Better for n < 30; slower; applies shrinkage to low-count genes
Code
vsd <- vst(dds,  blind = TRUE)   # blind=TRUE: ignore design (QC / exploration)
rld <- rlog(dds, blind = TRUE)

cat("VST assay dimensions:", dim(assay(vsd)), "\n")
VST assay dimensions: 14599 7 

f) Mean–variance relationship after VST

Re-examine the mean–variance relationship in VST-normalized data. How has it changed compared to (b)?

Code
par(mfrow = c(1, 2))
meanSdPlot(assay(vsd), ranks = TRUE, main = "Mean-SD: VST")
Warning in geom_hex(bins = bins, ...): Ignoring unknown parameters: `main`

Code
meanSdPlot(assay(rld), ranks = TRUE, main = "Mean-SD: rlog")
Warning in geom_hex(bins = bins, ...): Ignoring unknown parameters: `main`

Code
par(mfrow = c(1, 1))

The strong mean-dependent trend in the standard deviation is largely removed. The SD is now approximately flat across the expression range — the desired variance-stabilized behaviour. The rlog additionally shrinks low-count genes towards a global prior, which can slightly over-shrink in very low-expression regimes.


g) Heatmap of VST-transformed data

Create a heatmap of the 50 most variable genes using the VST-transformed data. How does this compare to (d)?

Code
top50_vst <- select_top50(assay(vsd))

pheatmap(
  top50_vst,
  annotation_col = anno_col,
  scale          = "row",
  show_rownames  = FALSE,
  cluster_cols   = TRUE,
  cluster_rows   = TRUE,
  color          = colorRampPalette(c("navy", "white", "firebrick3"))(100),
  main           = "Top 50 variable genes — VST-transformed (row-scaled)",
  fontsize       = 10,
  border_color   = NA
)

Finally, we arrive at the third stage of our analysis: the Variance Stabilizing Transformation (VST). While size-factor normalization corrected for library size, it did not address the heteroskedasticity inherent in RNA-Seq data. After applying the VST, the clustering hierarchy undergoes a definitive transformation: For the first time, biological condition (treated vs. untreated) replaces technical artifacts as the primary driver of the top-level clusters. All three treated samples are now grouped together on one major branch, while all four untreated samples reside on the other.

By stabilizing the variance across the entire dynamic range, we have effectively leveled the playing field for every gene, regardless of its expression level. The “batch effect” of the sequencing type is no longer strong enough to disrupt the biological signal. This visualization confirms that our preprocessing has been successful: we have moved beyond measuring the settings of the sequencing machine and are now accurately looking at the transcriptional changes caused by the Pasilla knockdown. This refined data is now ready for high-level statistical testing.


h) PCA: raw vs. VST-normalized counts

Perform a PCA using un-normalized counts and compare it to the PCA using VST-normalized counts. How are the samples separated in PC1 and PC2? According to which meta-variables?

Code
# Remove zero-variance genes (all-zero rows become constant after log+1)
counts_pca <- counts[rowSums(counts) > 0, ]
pca_raw <- prcomp(t(log2(counts_pca + 1)), scale. = TRUE)

pca_raw_df  <- as.data.frame(pca_raw$x[, 1:2]) %>%
  tibble::rownames_to_column("sample") %>%
  left_join(pasillaSampleAnno %>% tibble::rownames_to_column("sample"),
            by = "sample")

var_exp_raw <- round(summary(pca_raw)$importance[2, 1:2] * 100, 1)

ggplot(pca_raw_df, aes(x = PC1, y = PC2, color = condition,
                        shape = type, label = sample)) +
  geom_point(size = 4) +
  geom_text_repel(size = 3, show.legend = FALSE) +
  labs(title = "PCA — raw counts (log2+1)",
       x = paste0("PC1 (", var_exp_raw[1], "%)"),
       y = paste0("PC2 (", var_exp_raw[2], "%)")) +
  theme_bw(base_size = 13) +
  scale_color_brewer(palette = "Set1")

We can see that when we do a PCA of the raw counts allows us to see that most of the variance (separation along PC1) is explained by the sequencing type.

Can normalization address this?

Code
p_vst_pca  <- plotPCA(vsd, intgroup = c("condition", "type"), returnData = TRUE)
using ntop=500 top features by variance
Code
var_exp_vst <- round(attr(p_vst_pca, "percentVar") * 100, 1)

ggplot(p_vst_pca, aes(x = PC1, y = PC2, color = condition,
                       shape = type, label = name)) +
  geom_point(size = 4) +
  geom_text_repel(size = 3, show.legend = FALSE) +
  labs(title = "PCA — VST-normalized counts",
       x = paste0("PC1 (", var_exp_vst[1], "%)"),
       y = paste0("PC2 (", var_exp_vst[2], "%)")) +
  theme_bw(base_size = 13) +
  scale_color_brewer(palette = "Set1")

The transformation is striking when we look at the VST-normalized PCA. After variance stabilization, the “biological map” finally aligns with our experimental design:

PC1: The X-axis now separates the samples by condition. All blue (untreated) samples have migrated to the left, while all red (treated) samples have shifted to the right. This confirms that the treatment effect of the Pasilla knockdown is now the primary source of variation in our dataset.

PC2: The Y-axis now captures the remaining technical variation. The “triangles” are at the top and the “circles” are at the bottom.

By using VST, we have effectively “rotated” the perspective of our data. We have promoted the biological signal to the primary axis (PC1) and relegated the technical artifacts to the secondary axis (PC2). This PCA provides the visual proof that our data is now clean, consistent, and ready for further analysis.


Exercise 2 (Variance Stabilizing Transformation in Single-Cell RNA-seq)

We use the PBMC3k dataset (~2,700 peripheral blood mononuclear cells, 10x Genomics) and the Seurat framework to compare log1p normalization, a manual data-driven VST, and SCTransform() — Seurat’s implementation of a VST based on a regularized Negative Binomial model.

Code
library(Seurat)
library(SeuratData)
library(Matrix)
library(MASS)
library(patchwork)
library(scales)

InstallData("pbmc3k")
data("pbmc3k")
pbmc3k <- UpdateSeuratObject(pbmc3k)
pbmc3k
An object of class Seurat 
13714 features across 2700 samples within 1 assay 
Active assay: RNA (13714 features, 0 variable features)
 2 layers present: counts, data

a) Mean–variance relationship in raw counts

Compute per-gene mean and variance from the raw count matrix and plot them on a log–log scale. What do you observe? Why is this problematic for PCA or clustering?

Code
X <- pbmc3k@assays$RNA@counts

gene_means <- rowMeans(X)
gene_vars  <- apply(X, 1, var)

keep            <- gene_means > 0
gene_means_filt <- gene_means[keep]
gene_vars_filt  <- gene_vars[keep]

cat("Genes with mean > 0:", sum(keep), "out of", length(keep), "\n")
Genes with mean > 0: 13714 out of 13714 
Code
df_raw <- data.frame(mean = gene_means_filt, var = gene_vars_filt)

p_raw <- ggplot(df_raw, aes(x = mean, y = var)) +
  geom_point(size = 0.8, alpha = 0.3, color = "steelblue") +
  scale_x_log10(labels = label_log()) +
  scale_y_log10(labels = label_log()) +
  labs(title = "Mean–Variance (raw counts)",
       x = "Mean expression (log10)",
       y = "Variance (log10)") +
  theme_bw(base_size = 13) +
  annotation_logticks()

p_raw

Variance increases with the mean and faster than linearly (log–log slope > 1). This overdispersion means that highly expressed genes dominate PCA and clustering purely due to their large variance, even if they carry no meaningful biological signal relative to lowly expressed genes.


b) Negative Binomial trend lines

The NB distribution models overdispersed count data via Var(Y) = μ + αμ². Overlay NB trend lines for α ∈ {0.1, 0.5, 1.0} onto your plot from (a). Which α best describes the data? What does α = 0 correspond to?

Code
mu_grid <- 10^seq(log10(min(gene_means_filt)),
                   log10(max(gene_means_filt)), length.out = 300)

df_nb <- bind_rows(
  data.frame(mu = mu_grid, nb_var = mu_grid + 0.1 * mu_grid^2, alpha = "α = 0.1"),
  data.frame(mu = mu_grid, nb_var = mu_grid + 0.5 * mu_grid^2, alpha = "α = 0.5"),
  data.frame(mu = mu_grid, nb_var = mu_grid + 1.0 * mu_grid^2, alpha = "α = 1.0")
)

ggplot(df_raw, aes(x = mean, y = var)) +
  geom_point(size = 0.8, alpha = 0.25, color = "steelblue") +
  geom_line(data = df_nb, aes(x = mu, y = nb_var, color = alpha), linewidth = 1.2) +
  scale_x_log10(labels = label_log()) +
  scale_y_log10(labels = label_log()) +
  scale_color_manual(values = c("darkorange", "firebrick", "darkgreen")) +
  labs(title  = "Mean–Variance with NB Trend Lines",
       x      = "Mean expression (log10)",
       y      = "Variance (log10)",
       color  = "Dispersion") +
  theme_bw(base_size = 13) +
  annotation_logticks() +
  theme(legend.position = "bottom")

The bulk of the point cloud lies between the α = 0.1 and α = 0.5 curves, with α ≈ 0.1–0.5 describing the data best for moderately expressed genes; highly expressed genes scatter above all curves, suggesting α > 1 for some. α = 0 corresponds to the Poisson distribution (Var = μ), which clearly underestimates the observed overdispersion.


c) log1p normalization

Apply Seurat’s standard NormalizeData() pipeline (LogNormalize, scale factor 10,000) and repeat the mean–variance plot. Is the mean–variance dependence resolved? What does the fixed pseudo-count of 1 imply about the assumed overdispersion α?

Code
pbmc_log <- pbmc3k
pbmc_log <- NormalizeData(pbmc_log,
                           normalization.method = "LogNormalize",
                           scale.factor          = 1e4)

X_log <- pbmc_log@assays$RNA@data

log_means <- rowMeans(X_log)
log_vars  <- apply(X_log, 1, var)

keep_log       <- log_means > 0
log_means_filt <- log_means[keep_log]
log_vars_filt  <- log_vars[keep_log]

df_log <- data.frame(mean = log_means_filt, var = log_vars_filt)

p_log <- ggplot(df_log, aes(x = mean, y = var)) +
  geom_point(size = 0.8, alpha = 0.3, color = "darkorange") +
  geom_smooth(method = "loess", color = "firebrick",
              se = TRUE, linewidth = 1.2) +
  labs(title = "Mean–Variance: log1p normalization",
       x = "Mean log-normalized expression",
       y = "Variance") +
  theme_bw(base_size = 13)

p_log
`geom_smooth()` using formula = 'y ~ x'

The log1p transformation reduces the extreme scaling but does not fully resolve the mean–variance dependence — variance still rises for lowly expressed genes and then falls. The VST formula gives pseudo-count y₀ = 1/(4α); setting y₀ = 1 (as in log1p) implies α = 1/4 = 0.25 — a single global overdispersion assumed equal for all genes, which is too coarse.


d) Manual data-driven VST

Define size factors sᵢ = ∑g yᵢg / 10,000 and relative expression qg = ∑i yᵢg / ∑i sᵢ. Estimate gene-specific dispersion θ̂g via fitdistr, then apply the gene-specific and global VST.

To estimate the dispersion parameter α, we need a baseline expectation for each count y_ig that captures only technical variation, not biological signal. We construct this by decomposing the expected count into two components: a cell-specific size factor s_i, which reflects how deeply cell i was sequenced, and a gene-specific relative expression q_g, which captures how abundant gene g is on average across all cells.

Under the assumption that all cells express each gene at the same underlying rate, any difference in observed counts between cells is purely due to differences in sequencing depth. The expected count µ_ig = s_i · q_g is therefore the count we would predict for gene g in cell i if there were no biological variation — only a proportional scaling by library size.

This is deliberately a null model: by fixing µ_ig and estimating only θ_g = 1/α_g from the negative binomial distribution, we ask how much the observed counts deviate from this baseline expectation. A gene with high dispersion is one whose counts vary across cells more than sequencing depth alone can explain — which is precisely the biological signal we want to capture with the VST.

i) Sequencing depths and size factors

Code
Y <- t(as.matrix(pbmc3k@assays$RNA@counts))  # cells x genes

sequencing_depths <- rowSums(Y)
size_factors      <- sequencing_depths / 1e4

cat("Median sequencing depth:", median(sequencing_depths), "\n")
Median sequencing depth: 2196 
Code
cat("Mean sequencing depth:  ", round(mean(sequencing_depths), 1), "\n")
Mean sequencing depth:   2365.4 
Code
hist(size_factors, breaks = 60, col = "steelblue", border = "white",
     main = "Distribution of size factors across cells",
     xlab = "Size factor sᵢ", ylab = "Number of cells")
abline(v = median(size_factors), col = "firebrick", lwd = 2, lty = 2)

The median depth is around 2,000 counts/cell and size factors are concentrated near 0.2. The right skew reflects a small population of deeply sequenced cells.

ii) Gene-specific VST via fitdistr

Code
# Expected counts: outer product of size factors and relative expression
q_g    <- colSums(Y) / sum(size_factors)
keep   <- q_g > 0
Y_filt <- Y[, keep]
mu     <- size_factors %o% q_g[keep]   # cells x genes

# Estimate per-gene theta = 1/alpha via fitdistr (fixed mu)
n_genes <- ncol(Y_filt)
n_cores <- max(1, parallel::detectCores() - 1)
cat("Fitting NB dispersion for", n_genes, "genes on", n_cores, "cores...\n")
Fitting NB dispersion for 13714 genes on 11 cores...
Code
theta_estimates <- parallel::mclapply(seq_len(n_genes), function(g) {
  tryCatch(
    fitdistr(Y_filt[, g], "negative binomial",
             start = list(size = 1), mu = mu[, g])$estimate["size"],
    error = function(e) NA_real_
  )
}, mc.cores = n_cores)

theta_vec <- unlist(theta_estimates)
theta_vec <- theta_vec[is.finite(theta_vec) & theta_vec > 0]

cat("Genes fitted:  ", length(theta_vec),  "\n")
Genes fitted:   13650 
Code
cat("Genes failed:  ", n_genes - length(theta_vec), "\n")
Genes failed:   64 
Code
cat("θ median:", round(median(theta_vec), 3),
    "| mean:", round(mean(theta_vec), 3),
    "| range:", round(range(theta_vec), 3), "\n")
θ median: 1.738 | mean: 27517.61 | range: 0.002 838861.7 
Code
# Fitted gene names and gene-specific pseudo-counts y0_g = theta_g / 4
fitted_genes <- colnames(Y_filt)[seq_len(n_genes)][!is.na(unlist(theta_estimates)) &
                                                     unlist(theta_estimates) > 0 &
                                                     is.finite(unlist(theta_estimates))]
y0_per_gene  <- theta_vec / 4

# Gene-specific VST: f(y_ig) = log(y_ig / s_i + y0_g)
Y_gs  <- Y_filt[, fitted_genes]
sf_v  <- size_factors

X_vst_specific <- log(
  sweep(Y_gs, 1, sf_v, "/") +
  matrix(y0_per_gene, nrow = nrow(Y_gs),
         ncol = length(y0_per_gene), byrow = TRUE)
)

means_specific <- colMeans(X_vst_specific)
vars_specific  <- apply(X_vst_specific, 2, var)

p_vst_specific <- ggplot(data.frame(mean = means_specific, var = vars_specific),
                          aes(x = mean, y = var)) +
  geom_point(size = 0.8, alpha = 0.35, color = "steelblue") +
  geom_smooth(method = "loess", color = "firebrick", linewidth = 1.2) +
  geom_hline(yintercept = 1, linetype = "dashed", color = "grey40") +
  labs(title    = "Gene-specific VST",
       subtitle = "y0_g = θ̂_g / 4 per gene",
       x = "Mean after VST", y = "Variance after VST") +
  theme_bw(base_size = 13)
Code
theta_plot <- theta_vec[theta_vec <= 100]

ggplot(data.frame(theta = theta_plot), aes(x = theta)) +
  geom_histogram(bins = 60, fill = "#2d7d8e", color = "white", linewidth = 0.2) +
  scale_x_log10(
    breaks = 10^(-2:2),
    labels = scales::trans_format("log10", scales::math_format(10^.x))
  ) +
  annotation_logticks(sides = "b") +
  geom_vline(xintercept = median(theta_plot), linetype = "dashed",
             color = "firebrick", linewidth = 0.8) +
  labs(
    x = expression(hat(theta)[g] ~ "(NB size, log scale)"),
    y = "Number of genes",
    title = expression("Distribution of " * hat(theta)[g]),
    subtitle = paste0(sum(theta_vec <= 100), " genes shown; ",
                      sum(theta_vec > 100), 
                      " quasi-Poisson genes (θ > 100) excluded")
  ) +
  theme_classic(base_size = 13)

Code
p_log + p_vst_specific +
  plot_annotation(title = "Mean–Variance: log1p vs. Gene-Specific VST",
                  theme = theme(plot.title = element_text(size = 14, face = "bold")))
`geom_smooth()` using formula = 'y ~ x'
`geom_smooth()` using formula = 'y ~ x'

The gene-specific VST substantially flattens the mean–variance curve, with the loess trend hugging the dashed reference line (Var = 1) much more closely than after log1p.

iii) Global VST

Code
theta_global <- median(theta_vec)
y0_global    <- theta_global / 4
cat("Global θ̂ (median):", round(theta_global, 4), "\n")
Global θ̂ (median): 1.7375 
Code
cat("Global y₀ = θ̂/4: ", round(y0_global,    4), "\n")
Global y₀ = θ̂/4:  0.4344 
Code
X_vst_global <- log(sweep(Y_gs, 1, sf_v, "/") + y0_global)

p_vst_global <- ggplot(data.frame(mean = colMeans(X_vst_global),
                                   var  = apply(X_vst_global, 2, var)),
                        aes(x = mean, y = var)) +
  geom_point(size = 0.8, alpha = 0.35, color = "darkorange") +
  geom_smooth(method = "loess", color = "firebrick", linewidth = 1.2) +
  geom_hline(yintercept = 1, linetype = "dashed", color = "grey40") +
  labs(title    = "Global VST",
       subtitle = paste0("y₀ = median(θ̂_g)/4 = ", round(y0_global, 3)),
       x = "Mean after VST", y = "Variance after VST") +
  theme_bw(base_size = 13)

p_log + p_vst_specific + p_vst_global +
  plot_annotation(
    title = "Mean–Variance: log1p | Gene-Specific VST | Global VST",
    theme = theme(plot.title = element_text(size = 13, face = "bold"))
  )
`geom_smooth()` using formula = 'y ~ x'
`geom_smooth()` using formula = 'y ~ x'
`geom_smooth()` using formula = 'y ~ x'

Replacing gene-specific pseudo-counts with a single global median θ̂/4 partially sacrifices the stabilization for genes whose true dispersion deviates strongly from the median. Highly expressed genes (large θ in the data) tend to be over-shrunk; rare lowly-expressed genes are under-shrunk. The global VST is simpler but less accurate than the gene-specific version.


e) SCTransform

Apply SCTransform() to the PBMC dataset. Plot the relationship between geometric mean of raw counts and Pearson residual variance, and place it alongside the plots from (c) and (d-ii).

Code
pbmc_vst <- pbmc3k
pbmc_vst <- SCTransform(pbmc_vst, verbose = FALSE)
Code
# Use gene-level summary statistics stored by SCTransform:
# gmean = geometric mean of raw counts per gene
# residual_variance = variance of Pearson residuals per gene
sct_attrs <- pbmc_vst@assays$SCT@SCTModel.list[[1]]@feature.attributes

p_sct <- ggplot(sct_attrs, aes(x = log10(gmean), y = residual_variance)) +
  geom_point(size = 0.8, alpha = 0.3, color = "black") +
  geom_hline(yintercept = 1, linetype = "dashed", color = "grey40") +
  geom_density_2d(color = "steelblue", linewidth = 0.4, na.rm = TRUE) +
  labs(title    = "SCTransform (Pearson residuals)",
       subtitle = "Geometric mean of raw counts vs. residual variance",
       x = "log10(geometric mean raw counts)",
       y = "Variance of Pearson residuals") +
  theme_bw(base_size = 13)

(p_log + p_vst_specific) / p_sct +
  plot_annotation(
    title = "Mean–Variance Relationship: Three Normalization Strategies",
    caption = "Note: x-axes are not comparable — log1p/VST show transformed means; SCTransform shows log10(geometric mean of raw counts).",
    theme = theme(plot.title   = element_text(size = 13, face = "bold"),
                  plot.caption = element_text(size = 9, color = "grey40"))
  )
`geom_smooth()` using formula = 'y ~ x'
`geom_smooth()` using formula = 'y ~ x'

i) Characterization of mean–variance across panels

In a successfully variance-stabilized dataset, Var(zᵢg) ≈ 1 across all values of the geometric mean ḡg. The three panels show a progression: log1p leaves a pronounced mean-dependent trend; the gene-specific VST largely flattens it; SCTransform achieves the tightest stabilization with the trend closest to the dashed reference line Var = 1, because it uses gene-specific regularized NB regression.

ii) Comparison of pseudo-count distributions

Code
feature_attrs <- pbmc_vst@assays$SCT@SCTModel.list[[1]]@feature.attributes
feature_attrs$y0_g <- feature_attrs$theta / 4
fa_clean  <- feature_attrs[is.finite(feature_attrs$y0_g) & feature_attrs$y0_g > 0, ]

cat("Global MLE y₀ (manual):      ", round(y0_global,               4), "\n")
Global MLE y₀ (manual):       0.4344 
Code
cat("SCTransform y₀_g — median:   ", round(median(fa_clean$y0_g),   4), "\n")
SCTransform y₀_g — median:    0.0846 
Code
cat("SCTransform y₀_g — mean:     ", round(mean(fa_clean$y0_g),     4), "\n")
SCTransform y₀_g — mean:      0.1211 
Code
cat("SCTransform y₀_g — range:    ",
    round(range(fa_clean$y0_g),   4), "\n")
SCTransform y₀_g — range:     0.0013 1.9849 
Code
ggplot(fa_clean, aes(x = y0_g)) +
  geom_histogram(bins = 80, fill = "steelblue", color = "white", alpha = 0.8) +
  geom_vline(xintercept = y0_global, color = "firebrick",
             linewidth = 1.2, linetype = "dashed") +
  annotate("text", x = y0_global, y = Inf, vjust = 2, hjust = -0.15,
           label = paste0("Global ŷ₀ = ", round(y0_global, 3)),
           color = "firebrick", size = 4) +
  scale_x_log10() +
  labs(title    = "Distribution of Gene-Specific Pseudo-counts (SCTransform)",
       subtitle = "y₀_g = θ̂_g / 4; dashed line = global MLE estimate",
       x = "Pseudo-count y₀_g (log10 scale)", y = "Number of genes") +
  theme_bw(base_size = 13)

SCTransform applies kernel smoothing to regularize per-gene θ̂g estimates as a function of mean expression. This means that lowly expressed genes (noisy individual estimates) are pooled towards the global trend, preventing extreme pseudo-counts driven by estimation noise. The SCTransform estimates differ most from the unregularized per-gene estimates for lowly expressed genes, precisely where individual MLE fits are most unreliable due to the small number of non-zero counts.


f) UMAP comparison across normalization strategies

Run the full Seurat pipeline (PCA → FindNeighbors → FindClusters (resolution 0.5) → RunUMAP) for all three approaches and compare the UMAPs side by side.

Code
# Gemeinsame Parameter definieren
UMAP_NEIGHBORS <- 30
UMAP_MIN_DIST  <- 0.1   
UMAP_DIMS      <- 1:20

# --- log1p ---
pbmc_log <- FindVariableFeatures(pbmc_log, nfeatures = 2000)
pbmc_log <- ScaleData(pbmc_log, features = rownames(pbmc_log))
pbmc_log <- RunPCA(pbmc_log, npcs = 30, verbose = FALSE)
pbmc_log <- FindNeighbors(pbmc_log, dims = UMAP_DIMS)
pbmc_log <- FindClusters(pbmc_log, resolution = 0.5)
pbmc_log <- RunUMAP(pbmc_log, dims = UMAP_DIMS,
                    n.neighbors = UMAP_NEIGHBORS,
                    min.dist    = UMAP_MIN_DIST)
Warning: The default method for RunUMAP has changed from calling Python UMAP via reticulate to the R-native UWOT using the cosine metric
To use Python UMAP via reticulate, set umap.method to 'umap-learn' and metric to 'correlation'
This message will be shown once per session
Code
# --- Manual VST --- (unverändert bis RunUMAP)
# --- Manual VST ---
Y_all  <- t(as.matrix(LayerData(pbmc3k, assay = "RNA", layer = "counts")))
sf_all <- rowSums(Y_all) / 1e4
Y_for_manual <- Y_all[, fitted_genes]
X_man <- log(
  sweep(Y_for_manual, 1, sf_all, "/") +
  matrix(y0_per_gene, nrow = nrow(Y_for_manual),
         ncol = length(y0_per_gene), byrow = TRUE)
)

# Diese Schritte fehlten:
pbmc_manual <- CreateSeuratObject(counts = t(X_man))
Warning: Data is of class matrix. Coercing to dgCMatrix.
Code
pbmc_manual <- SetAssayData(pbmc_manual, assay = "RNA", layer = "data",
                             new.data = t(X_man))
pbmc_manual <- FindVariableFeatures(pbmc_manual, nfeatures = 2000)
Warning in eval(predvars, data, env): NaNs produced
Warning in hvf.info$variance.expected[not.const] <- 10^fit$fitted: number of
items to replace is not a multiple of replacement length
Code
pbmc_manual <- ScaleData(pbmc_manual, features = rownames(pbmc_manual))
pbmc_manual <- RunPCA(pbmc_manual, npcs = 30, verbose = FALSE)
pbmc_manual <- FindNeighbors(pbmc_manual, dims = UMAP_DIMS)
pbmc_manual <- FindClusters(pbmc_manual, resolution = 0.5)

# Jetzt erst RunUMAP
pbmc_manual <- RunUMAP(pbmc_manual, dims = UMAP_DIMS,
                       n.neighbors = UMAP_NEIGHBORS,
                       min.dist    = UMAP_MIN_DIST)

# --- SCTransform ---
pbmc_vst <- RunPCA(pbmc_vst, npcs = 30, verbose = FALSE)
pbmc_vst <- FindNeighbors(pbmc_vst, dims = UMAP_DIMS)
pbmc_vst <- FindClusters(pbmc_vst, resolution = 0.5)
pbmc_vst <- RunUMAP(pbmc_vst, dims = UMAP_DIMS,
                    n.neighbors = UMAP_NEIGHBORS,
                    min.dist    = UMAP_MIN_DIST)
Code
p_umap_log <- DimPlot(pbmc_log, reduction = "umap", label = TRUE,
                       label.size = 4, repel = TRUE) +
  labs(title = "log1p normalization") +
  theme_bw(base_size = 11) + theme(legend.position = "none") 

p_umap_man <- DimPlot(pbmc_manual, reduction = "umap", label = TRUE,
                       label.size = 4, repel = TRUE) +
  labs(title = "Manual gene-specific VST (ŷ0 per gene)") +
  theme_bw(base_size = 11) + theme(legend.position = "none") 

p_umap_vst <- DimPlot(pbmc_vst, reduction = "umap", label = TRUE,
                       label.size = 4, repel = TRUE) +
  labs(title = "SCTransform (VST)") +
  theme_bw(base_size = 11) + theme(legend.position = "none") 

p_umap_log + p_umap_man + p_umap_vst +
  plot_annotation(
    title = "UMAP: log1p vs. Manual Gene-Specific VST vs. SCTransform",
    theme = theme(plot.title = element_text(size = 14, face = "bold"))
  )

Observations:

log1p normalization (left) produces a compressed embedding with poorly separated clusters — several cell types overlap in the center, and only clusters 2 and 7 are clearly isolated. Manual gene-specific VST (middle), applying a per-gene variance-stabilizing transformation with estimated θ₀, substantially improves cluster separation: all 9 clusters are well-resolved and more evenly distributed across the UMAP space. SCTransform (right) yields a similar improvement, with clearly separated and compact clusters, broadly consistent with the manual VST result. Overall, both VST-based approaches outperform log1p normalization in resolving distinct cell populations, highlighting the importance of appropriate variance stabilization for count data prior to dimensionality reduction.

Code
cat("Clusters (log1p):    ", length(levels(pbmc_log$seurat_clusters)),    "\n")
Clusters (log1p):     8 
Code
cat("Clusters (Manual):   ", length(levels(pbmc_manual$seurat_clusters)), "\n")
Clusters (Manual):    9 
Code
cat("Clusters (SCTransform):", length(levels(pbmc_vst$seurat_clusters)),  "\n")
Clusters (SCTransform): 10 

Bonus: Kabsch Alignment

Kabsch alignment finds the optimal rotation matrix that minimizes the Root Mean Square Deviation (RMSD) between two sets of paired points. Applied to UMAP embeddings, it rotates one embedding to best match the other in a shared coordinate space — allowing direct visual comparison of cluster structures across normalization methods without artifacts from arbitrary UMAP orientation.

Code
kabsch <- function(P, Q) {

  centroid_P <- colMeans(P)
  centroid_Q <- colMeans(Q)
  Pc <- sweep(P, 2, centroid_P)
  Qc <- sweep(Q, 2, centroid_Q)
  
  # SVD
  H <- t(Pc) %*% Qc
  svd_res <- svd(H)
  
  # Rotationmatrix
  R <- svd_res$v %*% t(svd_res$u)
  
  if (det(R) < 0) {
    svd_res$v[, 2] <- -svd_res$v[, 2]
    R <- svd_res$v %*% t(svd_res$u)
  }
  
  # Translation
  t_vec <- centroid_Q - R %*% centroid_P
  
  # Transformation
  P_aligned <- t(R %*% t(P)) + matrix(t_vec, nrow(P), 2, byrow = TRUE)
  
  list(R = R, t = t_vec, P_aligned = P_aligned)
}

# UMAP
P <- pbmc_log[["umap"]]@cell.embeddings
Q <- pbmc_vst[["umap"]]@cell.embeddings

result <- kabsch(P, Q)
P_aligned <- as.data.frame(result$P_aligned)
Q_df      <- as.data.frame(Q)

# Plot
df_plot <- rbind(
  data.frame(UMAP1 = P_aligned[,1], UMAP2 = P_aligned[,2], method = "log1p (aligned)"),
  data.frame(UMAP1 = Q_df[,1],      UMAP2 = Q_df[,2],      method = "SCTransform")
)

ggplot(df_plot, aes(UMAP1, UMAP2, color = method)) +
  geom_point(size = 0.5, alpha = 0.5) +
  coord_fixed() +
  scale_color_manual(values = c("log1p (aligned)" = "steelblue",
                                "SCTransform"     = "firebrick")) +
  theme_bw(base_size = 12) +
  labs(title = "UMAP Kabsch-Alignment: log1p vs. SCTransform")

Following Kabsch alignment, the log1p (blue) and SCTransform (red) embeddings are superimposed in a common coordinate space, allowing direct visual comparison of their cluster structures. Where the two methods agree on cell population identity, blue and red points co-localize tightly — this is visible for several clusters in the lower-left region and the upper-center area. However, notable mismatches remain: some clusters are spatially offset between the two methods, and several SCTransform clusters appear more compact and better separated than their log1p counterparts. A few isolated red points with no corresponding blue neighbors (e.g., the outlier near UMAP1 = 5, UMAP2 = 8) suggest cells whose neighborhood structure differs substantially between normalizations. Overall, the alignment reveals that while the two methods recover broadly similar global topology, SCTransform produces tighter, more internally consistent clusters, consistent with its superior variance stabilization.