10  Matrix Factorization and Autoencoders

Chapter Aim

This chapter asks whether the high-dimensional measurement matrix can be explained through a small number of latent factors. A factor may represent a biological gradient, a cell state, a pathway activity, a hidden batch effect, a regulatory program, or an axis of multi-omics variation. The key idea is to replace \(p\) observed features by \(k\) latent coordinates, with \(k\ll p\), while keeping enough structure to reconstruct, denoise, or interpret the original measurements.

Matrix factorization and autoencoders are two versions of the same broad idea. Classical matrix factorization writes the data approximately as a product of two low-rank matrices. Autoencoders replace this linear product by two learned functions: an encoder that maps observations into a bottleneck representation and a decoder that reconstructs the data. For biological count data, the reconstruction loss should often be distributional rather than purely squared-error based. This connects the chapter back to Chapter 02 and Chapter 03: transformations, likelihoods, and latent representations are not separate topics.

TipNotation convention

Notation follows the shared convention in the notation chapter. Vectors are columns by default, row vectors are written with transposes, and dimensions should be checked whenever matrices are multiplied, transformed, modeled, or decomposed.

NoteMain external references

For the classical foundation, see MSMB Multivariate Analysis, especially PCA, SVD, biplots, and rank-reduced views of high-dimensional data. For heterogeneous and multi-table data, see MSMB Heterogeneous Data. Nonnegative matrix factorization is a useful parts-based baseline (Lee and Seung 1999). Network component analysis illustrates how prior graph structure can constrain a factorization (Liao et al. 2003). Multi-omics factor analysis is a probabilistic factor model for shared and view-specific latent structure across molecular modalities (Argelaguet et al. 2018; Nguyen and Wang 2020). Count-aware autoencoders for single-cell RNA-seq are exemplified by the deep count autoencoder (Eraslan et al. 2019). The corresponding computations for negative-binomial matrix factorization, glmpca, MSE versus NB autoencoder losses, nonlinear DCA, and PBMC3k are in Exercise Sheet 6, Exercise 03 (Exercise Sheet 06 2026).

Learning Goals

After this chapter, students should be able to write down a low-rank factorization with correct dimensions, connect PCA and SVD to reconstruction error, explain why NMF imposes an additive nonnegative representation, formulate generalized matrix factorization for count data using a negative-binomial likelihood, describe how multi-omics factor models share latent factors across modalities, define an autoencoder through encoder, bottleneck, and decoder maps, and explain why count-aware losses can matter for single-cell denoising.

10.1 The Factorization Object

The basic factorization model is

\[\begin{equation} X \approx H W^\top, \qquad X\in\mathbb{R}^{n\times p}, \qquad H\in\mathbb{R}^{n\times k}, \qquad W\in\mathbb{R}^{p\times k}, \qquad k\ll \min(n,p). \label{eq:ch10-basic-factorization} \end{equation}\]

The matrix \(H\) contains latent coordinates or scores for the observations. Row \(h_i^\top\) describes observation \(i\) in \(k\) latent dimensions. The matrix \(W\) contains feature loadings. Row \(w_j^\top\) describes how feature \(j\) relates to the latent factors. The reconstructed value for entry \((i,j)\) is

\[\begin{equation} \hat{x}_{ij} = h_i^\top w_j. \label{eq:ch10-entry-reconstruction} \end{equation}\]

Thus a factorization replaces \(np\) entries by approximately

\[\begin{equation} nk+pk = k(n+p) \label{eq:ch10-factorization-parameter-count} \end{equation}\]

parameters. This is useful only when the low-rank structure captures real signal. If \(k\) is too small, important biological variation is lost. If \(k\) is too large, the factorization starts to memorize noise.

The reconstruction error for a squared-error factorization is

\[\begin{equation} \mathcal{L}_{\mathrm{sq}}(H,W) = \frac{1}{2} \left\| X-HW^\top \right\|_F^2 = \frac{1}{2} \sum_{i=1}^{n} \sum_{j=1}^{p} \left( x_{ij}-h_i^\top w_j \right)^2. \label{eq:ch10-squared-reconstruction-loss} \end{equation}\]

Many methods in this chapter can be understood by changing one of three ingredients: the loss function, the constraints on \(H\) and \(W\), or the way the latent representation is parameterized.

10.2 PCA And The Singular Value Decomposition

For PCA-like factorizations, the data matrix should first be centered:

\[\begin{equation} X_c = X-\mathbf{1}_n\bar{x}^\top, \qquad \bar{x} = \frac{1}{n} X^\top\mathbf{1}_n. \label{eq:ch10-centered-data} \end{equation}\]

The singular value decomposition writes

\[\begin{equation} X_c = U\Lambda V^\top, \label{eq:ch10-svd} \end{equation}\]

where \(U\in\mathbb{R}^{n\times r}\) and \(V\in\mathbb{R}^{p\times r}\) have orthonormal columns, \(\Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_r)\) contains singular values with \(\lambda_1\geq\cdots\geq\lambda_r>0\), and \(r=\operatorname{rank}(X_c)\). The rank-\(k\) truncated SVD is

\[\begin{equation} X_{c,k} = U_k\Lambda_k V_k^\top. \label{eq:ch10-truncated-svd} \end{equation}\]

It solves the best rank-\(k\) approximation problem under squared error:

\[\begin{equation} X_{c,k} = \arg\min_{\operatorname{rank}(M)\leq k} \left\| X_c-M \right\|_F^2. \label{eq:ch10-best-rank-k} \end{equation}\]

In the notation of this chapter, the PCA factorization can be written as

\[\begin{equation} X_c \approx H W^\top, \qquad H=U_k\Lambda_k, \qquad W=V_k. \label{eq:ch10-pca-factorization} \end{equation}\]

The columns of \(H\) are the principal component scores, and the columns of \(W\) are loading vectors. The amount of centered variation explained by the first \(k\) components is

\[\begin{equation} \operatorname{PVE}(k) = \frac{ \sum_{\ell=1}^{k} \lambda_\ell^2 } { \sum_{\ell=1}^{r} \lambda_\ell^2 }. \label{eq:ch10-pve} \end{equation}\]

PCA is powerful because it gives the optimal linear rank-\(k\) reconstruction under squared error. It is also restrictive. The factors are orthogonal, entries of \(H\) and \(W\) can be positive or negative, and the Gaussian squared-error loss is most natural for continuous, approximately homoscedastic data. For normalized expression values, metabolomics intensities, or transformed measurements, this can be appropriate. For raw sequencing counts \(C\in\mathbb{N}_0^{n\times p}\), it usually is not.

10.3 Nonnegative Matrix Factorization

Nonnegative matrix factorization is used when the data and the intended representation should be additive. Suppose

\[\begin{equation} X\in\mathbb{R}_{\geq 0}^{n\times p}. \label{eq:ch10-nonnegative-data} \end{equation}\]

NMF estimates

\[\begin{equation} X \approx H W^\top, \qquad H\in\mathbb{R}_{\geq 0}^{n\times k}, \qquad W\in\mathbb{R}_{\geq 0}^{p\times k}. \label{eq:ch10-nmf-factorization} \end{equation}\]

A common squared-error NMF objective is

\[\begin{equation} (\hat{H},\hat{W}) = \arg\min_{H\geq 0,\ W\geq 0} \left\| X-HW^\top \right\|_F^2. \label{eq:ch10-nmf-objective} \end{equation}\]

The nonnegativity constraint changes the interpretation. PCA can reconstruct an observation through cancellations between positive and negative loadings. NMF allows only additive combinations of nonnegative parts (Lee and Seung 1999). If features are gene expression levels, taxa abundances, or peak intensities, each factor can be interpreted as a nonnegative pattern of features, and each observation receives a nonnegative exposure to that pattern.

The factorization is not unique. For any positive diagonal matrix \(D\in\mathbb{R}^{k\times k}\),

\[\begin{equation} H W^\top = (H D)(W D^{-1})^\top. \label{eq:ch10-factorization-scaling-nonidentifiability} \end{equation}\]

Therefore factor scales must be normalized or interpreted with care. NMF is also nonconvex in \((H,W)\) jointly. Different initializations can lead to different local optima, so stability of factors should be checked before assigning biological meaning.

10.4 Prior Structure And Network Component Analysis

Sometimes prior biological knowledge constrains which features can load on which factors. Network component analysis is an example in which the factorization represents hidden regulatory activities and their effects on measured outputs (Liao et al. 2003). A simplified model is

\[\begin{equation} X \approx H W^\top, \label{eq:ch10-nca-factorization} \end{equation}\]

where \(H\in\mathbb{R}^{n\times k}\) contains latent regulator activities across observations and \(W\in\mathbb{R}^{p\times k}\) contains regulatory effects on features. If feature \(j\) is not believed to be regulated by factor \(\ell\), the loading is constrained to zero:

\[\begin{equation} W_{j\ell}=0 \qquad \text{whenever prior knowledge excludes edge } \ell\to j. \label{eq:ch10-nca-zero-constraint} \end{equation}\]

Let \(B\in\{0,1\}^{p\times k}\) encode this prior regulatory graph. The constraint can be written as

\[\begin{equation} W = B\odot W, \label{eq:ch10-nca-mask} \end{equation}\]

where \(\odot\) denotes entrywise multiplication. This form makes clear how network information enters the factorization. It does not estimate an arbitrary low-rank representation. It estimates a low-rank representation compatible with a hypothesized feature-factor graph.

The corresponding constrained least-squares fitting problem is

\[\begin{equation} (\hat{H},\hat{W}) = \arg\min_{H\in\mathbb{R}^{n\times k},\ W\in\mathbb{R}^{p\times k}} \left\| X-HW^\top \right\|_F^2 \quad \text{subject to} \quad W=B\odot W. \label{eq:ch10-nca-constrained-optimization} \end{equation}\]

Equivalently, one can parameterize only the allowed entries of \(W\). If \(\mathcal{E}_B=\{(j,\ell):B_{j\ell}=1\}\) is the set of allowed feature-factor edges, then

\[\begin{equation} (\hat{H},\hat{W}_{\mathcal{E}_B}) = \arg\min_{H,\ W_{\mathcal{E}_B}} \sum_{i=1}^{n} \sum_{j=1}^{p} \left[ x_{ij} - \sum_{\ell:(j,\ell)\in\mathcal{E}_B} h_{i\ell}w_{j\ell} \right]^2. \label{eq:ch10-nca-edge-parameterization} \end{equation}\]

The optimization is still nonconvex in \(H\) and \(W\) jointly, and the scale of a regulator activity can be traded against the scale of its outgoing effects. NCA therefore relies not only on minimizing reconstruction error but also on identifiability conditions for the prior network and on a normalization convention for the recovered activities and loadings (Liao et al. 2003).

10.5 Multi-Omics Factor Models

In multi-omics studies, the same \(n\) observations may be measured in several modalities. Let

\[\begin{equation} X^{(m)} \in \mathbb{R}^{n\times p_m}, \qquad m=1,\ldots,M, \label{eq:ch10-multiomics-data} \end{equation}\]

where \(m\) indexes modalities such as transcriptomics, methylation, chromatin accessibility, proteomics, metabolomics, drug response, or mutation data. A shared factor model writes

\[\begin{equation} X^{(m)} \approx H \left(W^{(m)}\right)^\top, \qquad H\in\mathbb{R}^{n\times k}, \qquad W^{(m)}\in\mathbb{R}^{p_m\times k}. \label{eq:ch10-shared-multiomics-factorization} \end{equation}\]

The shared score matrix \(H\) places all observations in one latent space. The loading matrix \(W^{(m)}\) describes how modality \(m\) relates to those latent factors. A factor with nonzero loadings in several modalities captures shared variation; a factor with loadings concentrated in one modality captures view-specific variation.

A probabilistic multi-omics factor model can be written as

\[\begin{equation} x_{ij}^{(m)} \sim p_m \left( x \mid \eta_{ij}^{(m)}, \psi_m \right), \qquad \eta_{ij}^{(m)} = h_i^\top w_j^{(m)}. \label{eq:ch10-multiomics-likelihood} \end{equation}\]

Here \(p_m\) is the observation model for modality \(m\), and \(\psi_m\) contains modality-specific noise or dispersion parameters. This is the core idea behind methods such as MOFA: infer hidden factors that explain sources of variation across heterogeneous molecular views (Argelaguet et al. 2018). The biological payoff is that one factor can be linked simultaneously to several molecular layers. The statistical risk is that factors can also capture batch effects, library size, sample quality, or other technical axes. Factor interpretation must therefore use loadings, covariates, variance explained by view, and external biological knowledge.

10.6 Generalized Matrix Factorization For Counts

For sequencing data, the raw matrix is

\[\begin{equation} C\in\mathbb{N}_0^{n\times p}. \label{eq:ch10-count-matrix} \end{equation}\]

A squared-error loss on raw counts is usually poorly matched to the data. Counts are nonnegative, heteroscedastic, and often overdispersed. A generalized matrix factorization keeps the low-rank structure but places it inside a distributional model. For a negative-binomial count model,

\[\begin{equation} c_{ij} \sim \operatorname{NB}(\mu_{ij},\phi_j), \qquad \operatorname{Var}(c_{ij}) = \mu_{ij} + \phi_j\mu_{ij}^2. \label{eq:ch10-nb-count-factorization-model} \end{equation}\]

The mean is positive, so the low-rank predictor is passed through a link function. With size factors \(s_i>0\),

\[\begin{equation} \log \mu_{ij} = \log s_i + h_i^\top w_j, \label{eq:ch10-nb-factorization-link} \end{equation}\]

or equivalently

\[\begin{equation} \mu_{ij} = s_i \exp \left( h_i^\top w_j \right). \label{eq:ch10-nb-factorization-mean} \end{equation}\]

The negative log-likelihood objective is

\[\begin{equation} \mathcal{L}_{\mathrm{NB}}(H,W,\phi) = - \sum_{i=1}^{n} \sum_{j=1}^{p} \log \Pr \left( c_{ij} \mid \mu_{ij}, \phi_j \right). \label{eq:ch10-nb-factorization-loss} \end{equation}\]

This is the same factorization idea as Equation \(\ref{eq:ch10-basic-factorization}\), but the reconstruction target is now the mean of a count distribution rather than a real-valued matrix under Gaussian noise. The practical consequence is important: the loss function encodes the mean-variance relationship. A negative-binomial factorization can use the bottleneck to model biological structure without treating every large count deviation as equally surprising. This is the logic behind methods such as GLM-PCA for count data and the negative-binomial factorization computations in Exercise Sheet 6, Exercise 03 (Exercise Sheet 06 2026).

10.7 Autoencoders As Nonlinear Factorizations

An autoencoder replaces the linear map \(x_i\mapsto h_i\) and the linear reconstruction \(h_i\mapsto \hat{x}_i\) by learned functions. Let \(x_i\in\mathbb{R}^{p}\) denote the column-vector version of row \(i\) of \(X\). The encoder is

\[\begin{equation} h_i = f_{\theta}(x_i), \qquad h_i\in\mathbb{R}^{k}, \label{eq:ch10-autoencoder-encoder} \end{equation}\]

and the decoder is

\[\begin{equation} \hat{x}_i = g_{\omega}(h_i), \qquad \hat{x}_i\in\mathbb{R}^{p}. \label{eq:ch10-autoencoder-decoder} \end{equation}\]

The bottleneck matrix is

\[\begin{equation} H = \begin{bmatrix} h_1^\top\\ \vdots\\ h_n^\top \end{bmatrix} \in \mathbb{R}^{n\times k}. \label{eq:ch10-autoencoder-bottleneck-matrix} \end{equation}\]

With squared reconstruction loss, the autoencoder objective is

\[\begin{equation} \mathcal{L}_{\mathrm{AE}}(\theta,\omega) = \frac{1}{2} \sum_{i=1}^{n} \left\| x_i - g_{\omega} \left( f_{\theta}(x_i) \right) \right\|_2^2. \label{eq:ch10-autoencoder-mse-loss} \end{equation}\]

If both \(f_\theta\) and \(g_\omega\) are linear maps and the bottleneck has dimension \(k\), the autoencoder is closely related to PCA: it learns a rank-\(k\) linear reconstruction. If nonlinear activation functions are included, the model becomes a nonlinear factorization. The representation \(h_i\) can describe curved manifolds, nonlinear gene-gene relationships, or complex denoising rules that a linear factorization cannot capture.

This flexibility is also the main danger. A sufficiently flexible autoencoder can reconstruct technical noise or cell-specific artifacts. The bottleneck dimension, regularization, training time, and validation metric become part of the statistical model.

10.8 Count-Aware Autoencoders And DCA

For single-cell RNA-seq counts, the input is again a count matrix \(C\in\mathbb{N}_0^{n\times p}\). A count-aware autoencoder decodes each bottleneck representation into distributional parameters rather than directly into a real-valued reconstruction. In a negative-binomial autoencoder,

\[\begin{equation} h_i = f_{\theta}(c_i), \qquad \mu_i = g_{\omega}^{(\mu)} \left( h_i \right), \label{eq:ch10-count-autoencoder-mean} \end{equation}\]

with \(\mu_i\in\mathbb{R}_{>0}^{p}\). Entrywise,

\[\begin{equation} c_{ij} \sim \operatorname{NB}(\mu_{ij},\phi_j). \label{eq:ch10-count-autoencoder-nb} \end{equation}\]

The negative-binomial autoencoder loss is

\[\begin{equation} \mathcal{L}_{\mathrm{NB-AE}}(\theta,\omega,\phi) = - \sum_{i=1}^{n} \sum_{j=1}^{p} \log \Pr \left( c_{ij} \mid \mu_{ij}, \phi_j \right). \label{eq:ch10-nb-autoencoder-loss} \end{equation}\]

The deep count autoencoder extends this idea to account for overdispersion and sparsity in single-cell RNA-seq, with negative-binomial or zero-inflated negative-binomial observation models (Eraslan et al. 2019). In a zero-inflated version, an additional dropout probability \(\pi_{ij}\in[0,1]\) is decoded:

\[\begin{equation} \pi_i = g_{\omega}^{(\pi)} \left( h_i \right), \qquad \pi_i\in[0,1]^p. \label{eq:ch10-dca-dropout-decoder} \end{equation}\]

The zero-inflated negative-binomial probability mass function is

\[\begin{equation} \Pr(c_{ij}=y) = \pi_{ij}\mathbf{1}\{y=0\} + (1-\pi_{ij}) \Pr_{\operatorname{NB}} \left( Y=y \mid \mu_{ij}, \phi_j \right). \label{eq:ch10-zinb-pmf} \end{equation}\]

The denoised expression value is usually taken to be the fitted mean or a transformation of it:

\[\begin{equation} \hat{x}_{ij}^{\mathrm{denoise}} = \hat{\mu}_{ij}. \label{eq:ch10-denoised-mean} \end{equation}\]

The bottleneck \(H\) can also be used as a low-dimensional representation for clustering, visualization, or trajectory analysis. The key point is that the autoencoder is not just a neural-network trick. It is a generalized factorization in which the decoder defines a mean-variance-aware reconstruction model.

10.9 Validation And Interpretation

Latent factors are tempting because they give names to hidden axes of variation. In biological data analysis, that temptation should be handled carefully. A factor can correspond to cell type, disease state, differentiation, pathway activity, sequencing depth, batch, donor, or a mixture of these. The interpretation should therefore use several checks.

First, factor scores should be related to known covariates. If \(z_i\) is a known covariate and \(h_{i\ell}\) is factor \(\ell\), one can examine

\[\begin{equation} h_{i\ell} = \alpha_0 + \alpha_1 z_i + \varepsilon_i. \label{eq:ch10-factor-covariate-regression} \end{equation}\]

Second, loadings should be inspected at the feature level. Large absolute loadings indicate features that define a factor:

\[\begin{equation} \mathcal{J}_{\ell}(m) = \left\{ j: |w_{j\ell}| \text{ is among the }m\text{ largest values} \right\}. \label{eq:ch10-top-loading-set} \end{equation}\]

Third, downstream performance should be evaluated on the representation that is actually used. If \(H\) is used for clustering, clustering quality should be measured on \(H\), not only on a two-dimensional visualization. The Exercise Sheet 6 autoencoder analysis makes this point directly: the native bottleneck representation and a two-dimensional t-SNE view can give different impressions of separability (Exercise Sheet 06 2026).

Finally, reconstruction quality alone is not enough. A model can reconstruct noise. A useful representation should reconstruct relevant signal, improve or preserve downstream structure, and remain stable under reasonable changes in initialization, regularization, and bottleneck dimension.

Summary

Matrix factorization represents a high-dimensional biological matrix through latent scores and feature loadings. PCA is the canonical linear squared-error factorization. NMF adds nonnegativity and yields additive parts-based representations. Prior-structured factorizations, such as network component analysis, constrain loadings using biological knowledge. Multi-omics factor models share latent scores across several molecular views and use modality-specific loadings to identify shared and view-specific axes of variation.

Generalized matrix factorization changes the loss from squared error to a distributional likelihood. For count data, a negative-binomial factorization respects nonnegativity, overdispersion, and size factors. Autoencoders extend the same idea by replacing linear scores and loadings with learned encoder and decoder functions. Count-aware autoencoders such as DCA combine nonlinear representation learning with a negative-binomial or zero-inflated negative-binomial observation model. The central message is that the latent representation is only meaningful relative to the data scale, likelihood, constraints, and validation strategy used to learn it.

References

Argelaguet, Ricard, Britta Velten, Damien Arnol, et al. 2018. “Multi‐omics Factor Analysis—a Framework for Unsupervised Integration of Multi‐omics Data Sets.” Molecular Systems Biology 14 (6). https://doi.org/10.15252/msb.20178124.
Eraslan, Gökcen, Lukas M. Simon, Maria Mircea, Nikola S. Mueller, and Fabian J. Theis. 2019. “Single-Cell RNA-Seq Denoising Using a Deep Count Autoencoder.” Nature Communications 10 (1). https://doi.org/10.1038/s41467-018-07931-2.
Exercise Sheet 06. 2026. Exercise Sheet 06 Solutions: Task 1 Sparse Microbial Network Inference with SPIEC-EASI, Task 2 Network Analysis with NetCoMi, and Deep Count Autoencoders. https://bio-datascience.github.io/ahbd/assets/exercises/AHDBD_SS26_Exercise06_Solution.html.
Lee, Daniel D., and H. Sebastian Seung. 1999. “Learning the Parts of Objects by Non-Negative Matrix Factorization.” Nature 401 (6755): 788–91. https://doi.org/10.1038/44565.
Liao, James C., Riccardo Boscolo, Young-Lyeol Yang, Linh My Tran, Chiara Sabatti, and Vwani P. Roychowdhury. 2003. “Network Component Analysis: Reconstruction of Regulatory Signals in Biological Systems.” Proceedings of the National Academy of Sciences 100 (26): 15522–27. https://doi.org/10.1073/pnas.2136632100.
Nguyen, Nam D., and Daifeng Wang. 2020. “Multiview Learning for Understanding Functional Multiomics.” PLOS Computational Biology 16 (4): e1007677. https://doi.org/10.1371/journal.pcbi.1007677.