11  Deep Learning and Foundation Models

Chapter Aim

This final chapter is an outlook. The previous chapters built a statistical language for high-dimensional biological data: distributions, normalization, embeddings, clustering, testing, regression, graphical models, networks, and latent factors. Deep learning and foundation models do not remove that language. They change where some of the modeling work happens. Instead of constructing a small number of features by hand and then fitting a simple model, we often learn a representation directly from raw or lightly processed biological data. In the most ambitious cases, the representation is pretrained once on a very large biological corpus and then adapted to many downstream tasks.

The central question is therefore not whether deep learning is a replacement for high-dimensional statistics. It is how high-dimensional statistics reappears inside larger learned systems. The final prediction layer may be a linear or logistic regression. The validation problem is still a sampling problem. Batch effects, leakage, confounding, calibration, and distribution shift remain statistical problems. What changes is that the covariates may now be learned embeddings rather than measured genes, peaks, taxa, or variants.

TipNotation convention

Notation follows the shared convention in the notation chapter. Vectors are columns by default, row vectors are written with transposes, and dimensions should be checked whenever matrices are multiplied, transformed, modeled, or decomposed.

NoteMain external references

For the statistical background, see MSMB Supervised Learning, especially cross-validation, bias-variance trade-offs, penalization, and method hacking. See also MSMB Experimental Design for reproducibility, data quality, and leaky workflows, and MSMB Image Data for a concrete setting where feature extraction and representation learning meet. Deep learning in biology is reviewed in broad terms (Ching et al. 2018). Sequence-based regulatory prediction is exemplified by DeepSEA (Zhou and Troyanskaya 2015). Count-aware single-cell autoencoding connects directly to Chapter 10 (Eraslan et al. 2019). Protein structure prediction and protein foundation models provide the clearest archetypical case for this chapter (Senior et al. 2020; Jumper et al. 2021; Rives et al. 2021; Hayes et al. 2025). Low-rank adaptation is one modern way to fine-tune large pretrained models with relatively few trainable parameters (Hu et al. 2021).

Learning Goals

After this chapter, students should be able to explain the difference between hand-crafted features and end-to-end representation learning, describe the pretrain/fine-tune paradigm, recognize why fine-tuning often reduces to classical high-dimensional prediction on top of learned embeddings, use ESM as an archetypical biological foundation model, and evaluate deep biological models with the same statistical caution used throughout the course.

11.1 From Hand-Crafted Features To End-To-End Learning

In a classical high-dimensional workflow, raw measurements are converted into an analysis representation and then into features used by a statistical model. Written schematically,

\[\begin{equation} R \longrightarrow \widetilde{X} \longrightarrow h(\widetilde{X}) \longrightarrow \hat{y}, \label{eq:ch11-classical-pipeline} \end{equation}\]

where \(R\) denotes the raw measurement object, \(\widetilde{X}\) denotes the chosen analysis representation, and \(h(\widetilde{X})\) denotes hand-designed or separately estimated features such as principal components, clusters, pathway scores, graph summaries, or selected genes. The statistical model acts only after this feature construction has happened.

End-to-end learning changes the pipeline by making the feature map part of the fitted model. A neural network is a parameterized function \(f_\theta\) with many layers and many parameters. It may take normalized continuous features, raw count-like inputs, one-hot encoded sequences, images, protein sequences, molecular graphs, or multimodal data as input. The predictor is fitted by minimizing an empirical loss,

\[\begin{equation} \hat{\theta} = \underset{\theta}{\operatorname{argmin}}\, \frac{1}{n} \sum_{i=1}^{n} \ell\left\{ y_i, f_\theta(x_i) \right\}, \label{eq:ch11-end-to-end-risk} \end{equation}\]

where \(x_i\) denotes the input object for observation \(i\), \(y_i\) is the target, and \(\ell\) is a task-specific loss. For a continuous response, \(\ell\) may be squared error. For a binary response, it is often cross-entropy. For count data, it may be the negative log-likelihood of a Poisson or negative-binomial distribution. This is the same modeling principle as before: one specifies a data object, a prediction target, a loss, and a validation strategy. The difference is that \(f_\theta\) may learn both the representation and the prediction rule.

For binary classification with predicted probability \(\hat{\pi}_i=f_\theta(x_i)\in(0,1)\), the empirical cross-entropy loss is

\[\begin{equation} \mathcal{L}_{\mathrm{bin}}(\theta) = - \frac{1}{n} \sum_{i=1}^{n} \left\{ y_i\log \hat{\pi}_i + (1-y_i)\log(1-\hat{\pi}_i) \right\}. \label{eq:ch11-binary-cross-entropy} \end{equation}\]

For a multiclass problem with \(K\) classes, one typically uses a softmax output. If \(o_i\in\mathbb{R}^K\) are the output scores and \(\hat{\pi}_{ik}\) is the predicted probability of class \(k\), then

\[\begin{equation} \hat{\pi}_{ik} = \frac{\exp(o_{ik})} {\sum_{\ell=1}^{K}\exp(o_{i\ell})}. \label{eq:ch11-softmax-probability} \end{equation}\]

Thus deep learning does not evade likelihood thinking. It often hides a familiar generalized linear model at the output layer, while replacing the input covariates by learned representations.

11.2 Representations And Multimodal Biology

The representation view is the cleanest way to connect this chapter to matrix factorization. In Chapter 10, an autoencoder learned a low-dimensional representation

\[\begin{equation} H = f_\theta(X), \qquad \hat{X} = g_\phi(H), \label{eq:ch11-autoencoder-representation} \end{equation}\]

where \(H\in\mathbb{R}^{n\times k}\) contains one learned vector \(h_i^\top\) per observation. Deep supervised models use the same idea, but the representation is optimized for a prediction task rather than only for reconstruction.

Modern biological datasets often contain several modalities. For observation \(i\), let

\[\begin{equation} x_i^{(1)},\ldots,x_i^{(M)} \label{eq:ch11-multimodal-observation} \end{equation}\]

denote measurements from \(M\) modalities, for example RNA counts, chromatin accessibility, protein abundance, histology images, perturbation labels, or clinical covariates. Modality-specific encoders map these objects into latent vectors,

\[\begin{equation} h_i^{(m)} = f_{\theta_m}^{(m)} \left( x_i^{(m)} \right), \qquad m=1,\ldots,M. \label{eq:ch11-modality-specific-encoder} \end{equation}\]

A fusion map then constructs a shared representation,

\[\begin{equation} h_i = a_\psi \left( h_i^{(1)},\ldots,h_i^{(M)} \right), \label{eq:ch11-multimodal-fusion} \end{equation}\]

which may be used for classification, regression, imputation, matching, or generation. The statistical challenge is that modalities have different noise models and missingness patterns. RNA-seq and ATAC-seq are count-like and sparse. Protein abundance may be continuous after transformation. Mutation status may be binary. Images are high-dimensional continuous arrays. A good model must respect these differences, even if the final representation is a shared vector.

This is why the modeling lessons from Chapters 02 and 03 remain relevant. A model that treats raw sparse counts as ordinary Gaussian measurements may learn technical variation. A multimodal model that ignores missingness mechanisms may learn which assay was performed rather than the biology of interest.

11.3 Sequence-To-Function Models

Genomic sequence models provide a clean example of end-to-end learning because the input is not an expression matrix but a string. For a DNA window of length \(L\), define a one-hot encoded sequence

\[\begin{equation} S_i \in \{0,1\}^{L\times 4}, \label{eq:ch11-one-hot-dna} \end{equation}\]

where the four columns correspond to the nucleotides A, C, G, and T. A sequence model maps \(S_i\) to one or many functional readouts, such as transcription-factor binding, DNase hypersensitivity, histone marks, or variant effects. In a multitask setting with \(K\) binary chromatin features, the model outputs

\[\begin{equation} \hat{\pi}_{ik} = f_{\theta,k}(S_i), \qquad k=1,\ldots,K. \label{eq:ch11-sequence-multitask-output} \end{equation}\]

The loss sums over both genomic windows and tasks:

\[\begin{equation} \mathcal{L}_{\mathrm{seq}}(\theta) = - \frac{1}{nK} \sum_{i=1}^{n} \sum_{k=1}^{K} \left\{ y_{ik}\log \hat{\pi}_{ik} + (1-y_{ik})\log(1-\hat{\pi}_{ik}) \right\}. \label{eq:ch11-sequence-multitask-loss} \end{equation}\]

The important point is conceptual. The model is not first given motif counts, manually selected windows, or a small list of engineered sequence features. Instead, filters and higher-level sequence patterns are learned from the prediction task. DeepSEA is an early influential example of this idea for noncoding variant effect prediction (Zhou and Troyanskaya 2015).

11.4 Count-Aware Deep Models For Single Cells

Single-cell RNA-seq is a second example, but here the input is not a sequence. It is a sparse count vector. For cell \(i\), let

\[\begin{equation} c_i = \left(c_{i1},\ldots,c_{ip}\right)^\top \in \mathbb{N}_0^p \label{eq:ch11-single-cell-count-vector} \end{equation}\]

denote raw gene counts. A count-aware autoencoder maps this high-dimensional vector into a latent representation and decodes it into distributional parameters, for example a negative-binomial mean and dispersion:

\[\begin{equation} h_i = f_\theta(c_i), \qquad \mu_i = g_{\phi,\mu}(h_i), \qquad \alpha_i = g_{\phi,\alpha}(h_i). \label{eq:ch11-count-autoencoder-parameters} \end{equation}\]

The fitting objective is then a negative log-likelihood rather than an ordinary squared reconstruction error:

\[\begin{equation} \mathcal{L}_{\mathrm{NB-AE}}(\theta,\phi) = - \sum_{i=1}^{n} \sum_{j=1}^{p} \log p_{\mathrm{NB}} \left( c_{ij}; \mu_{ij}, \alpha_{ij} \right). \label{eq:ch11-negative-binomial-autoencoder-loss} \end{equation}\]

This example is a useful warning against the phrase “deep learning” as if it were a single model. A good deep model for count data should still encode an appropriate observation model. DCA is a concrete example of such a count-aware approach for single-cell denoising (Eraslan et al. 2019), and it is the natural continuation of the count-factorization ideas in Chapter 10.

11.5 The Pretrain And Fine-Tune Paradigm

The conceptual shift from deep learning to foundation models is the move from task-specific supervised training to large-scale pretraining. In supervised training, one fits \(\theta\) directly for the task of interest. In pretraining, one first learns parameters from a broad data source, often without manual labels:

\[\begin{equation} \hat{\theta}_{\mathrm{pre}} = \underset{\theta}{\operatorname{argmin}}\, \frac{1}{N} \sum_{i=1}^{N} \mathcal{L}_{\mathrm{pre}} \left( u_i;\theta \right), \label{eq:ch11-pretraining-objective} \end{equation}\]

where \(u_i\) are unlabeled or weakly labeled biological objects and \(N\) is typically very large. The pretrained encoder \(f_{\hat{\theta}_{\mathrm{pre}}}\) maps a new object into an embedding. A downstream task can then be fitted on top of these embeddings:

\[\begin{equation} z_i = f_{\hat{\theta}_{\mathrm{pre}}}(x_i) \in \mathbb{R}^{d}. \label{eq:ch11-pretrained-embedding} \end{equation}\]

At this point, much of the downstream analysis becomes classical high-dimensional statistics again. If the pretrained representation is frozen, a regression or classification head estimates only a small set of downstream parameters. For a binary phenotype, a logistic fine-tuning head is

\[\begin{equation} \Pr(Y_i=1\mid z_i) = \sigma \left( \beta_0+z_i^\top\beta \right), \qquad \sigma(t) = \frac{1}{1+\exp(-t)}. \label{eq:ch11-logistic-finetuning-head} \end{equation}\]

Here \(\beta_0\) is the intercept and \(\beta\in\mathbb{R}^d\) contains regression coefficients. If \(d\) is large relative to the number of labeled examples, regularization is again necessary:

\[\begin{equation} \hat{\beta} = \underset{\beta}{\operatorname{argmin}}\, \left[ - \sum_{i=1}^{n} \left\{ y_i\log \hat{\pi}_i + (1-y_i)\log(1-\hat{\pi}_i) \right\} + \lambda\|\beta\|_1 \right]. \label{eq:ch11-sparse-logistic-head} \end{equation}\]

This equation is deliberately familiar. It is the sparse logistic regression problem from Chapter 07, now applied to pretrained embeddings instead of hand-defined molecular features. The foundation model changes the covariates; it does not remove the need for regularization, validation, or interpretation.

Fine-tuning can also update part of the pretrained model. Low-rank adaptation is one parameter-efficient strategy. For a pretrained weight matrix \(W_0\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}\), one fits a low-rank update

\[\begin{equation} W = W_0+\Delta W, \qquad \Delta W = BA, \label{eq:ch11-lora-update} \end{equation}\]

where \(A\in\mathbb{R}^{r\times d_{\mathrm{in}}}\), \(B\in\mathbb{R}^{d_{\mathrm{out}}\times r}\), and \(r\ll\min(d_{\mathrm{in}},d_{\mathrm{out}})\). The number of trainable parameters is therefore

\[\begin{equation} r d_{\mathrm{in}}+d_{\mathrm{out}}r = r \left( d_{\mathrm{in}}+d_{\mathrm{out}} \right), \label{eq:ch11-lora-parameter-count} \end{equation}\]

which can be much smaller than training all entries of \(W_0\). This is a modern version of a familiar idea: restrict the effective dimension of the fitted model so that limited labeled data are not asked to estimate too many free parameters.

11.6 ESM As An Archetypical Foundation Model

Protein language models give perhaps the cleanest biological example of the foundation-model paradigm. A protein sequence is a string

\[\begin{equation} a_{1:L} = \left(a_1,\ldots,a_L\right), \qquad a_t\in\mathcal{A}, \label{eq:ch11-protein-sequence} \end{equation}\]

where \(\mathcal{A}\) is the amino-acid alphabet. In ESM-style masked language modeling, a subset \(M\subset\{1,\ldots,L\}\) of positions is masked, and the model is trained to predict the missing amino acids from the unmasked context:

\[\begin{equation} \mathcal{L}_{\mathrm{MLM}}(\theta) = - \mathbb{E}_{a_{1:L}} \mathbb{E}_{M} \sum_{t\in M} \log p_\theta \left( a_t \mid a_{\{1,\ldots,L\}\setminus M} \right). \label{eq:ch11-masked-language-loss} \end{equation}\]

This is not a supervised phenotype-prediction problem. The training signal comes from the sequence itself. Yet the resulting representation can encode biological structure and function because the distribution of natural protein sequences reflects evolutionary constraints (Rives et al. 2021). After pretraining, each residue and each full sequence has an embedding:

\[\begin{equation} E = f_{\hat{\theta}_{\mathrm{pre}}} \left( a_{1:L} \right) \in \mathbb{R}^{L\times d}, \label{eq:ch11-esm-residue-embedding} \end{equation}\]

where row \(e_t^\top\) represents residue position \(t\). A sequence-level embedding can be obtained by pooling,

\[\begin{equation} \bar{e} = \frac{1}{L} \sum_{t=1}^{L} e_t \in \mathbb{R}^{d}. \label{eq:ch11-esm-pooled-embedding} \end{equation}\]

Downstream tasks can then use \(E\) or \(\bar{e}\) for contact prediction, structure prediction, mutational effect prediction, annotation, or design. In its simplest form, contact prediction is again a fitted prediction problem on representations:

\[\begin{equation} \Pr(A_{st}=1\mid e_s,e_t) = \sigma \left( \gamma_0+ \eta^\top \psi(e_s,e_t) \right), \label{eq:ch11-contact-head} \end{equation}\]

where \(A_{st}\) indicates whether residues \(s\) and \(t\) are in contact, \(\psi(e_s,e_t)\) is a pair representation constructed from the two residue embeddings, \(\gamma_0\) is an intercept, and \(\eta\) contains regression coefficients. This is the point of the archetype: the impressive biological behavior comes from large-scale sequence pretraining, but many downstream uses still involve familiar supervised heads, calibration, test splits, and regularization.

ESM also illustrates the path from predictive to generative biological modeling. Later ESM models treat sequence, structure, and function as coupled token tracks and can be prompted or sampled to generate proteins (Hayes et al. 2025). Conceptually, this extends the language-model idea from “predict masked residues” to “model a joint biological object” whose components may include amino-acid identity, structural tokens, functional annotations, and geometric information.

11.7 Protein Structure And Learned Geometry

Protein structure prediction is another example of end-to-end representation learning, but with a geometric output. Given a sequence \(a_{1:L}\), the model predicts three-dimensional coordinates

\[\begin{equation} R = \left( r_1,\ldots,r_L \right)^\top \in \mathbb{R}^{L\times 3}, \label{eq:ch11-protein-coordinates} \end{equation}\]

where \(r_t\) is a coordinate vector for residue \(t\) or for a representative atom. Earlier systems combined learned pairwise potentials with geometric optimization (Senior et al. 2020). AlphaFold-style models make a larger part of the pipeline differentiable and learned, while still using strong geometric inductive structure (Jumper et al. 2021). From the perspective of this course, the key lesson is that high-dimensional biological prediction often benefits from combining large learned representations with domain constraints: sequences are discrete strings, structures live in three-dimensional geometry, and biological outputs have symmetries and physical restrictions.

11.8 Evaluation Still Decides The Claim

Deep learning makes leakage easier because the preprocessing and representation learning can involve enormous datasets. A model may perform well because homologous sequences, related donors, repeated images, shared batches, or near-duplicate perturbations appear across train and test splits. Therefore the statistical object of interest is not the training loss but the expected prediction risk

\[\begin{equation} R(f) = \mathbb{E}_{(X,Y)\sim P_{\mathrm{test}}} \left[ \ell\{Y,f(X)\} \right]. \label{eq:ch11-test-risk} \end{equation}\]

The empirical test risk estimates this quantity on held-out data:

\[\begin{equation} \hat{R}_{\mathrm{test}}(f) = \frac{1}{n_{\mathrm{test}}} \sum_{i\in\mathcal{I}_{\mathrm{test}}} \ell \left\{ y_i, f(x_i) \right\}. \label{eq:ch11-empirical-test-risk} \end{equation}\]

For biological applications, the split must match the intended generalization claim. If the claim is generalization to new patients, split by patient. If the claim is generalization to new studies, split by study. If the claim is generalization to remote protein families, split by sequence similarity or family. If the claim is perturbation response prediction, hold out perturbations, cell types, or combinations according to the desired use case.

Calibration is equally important. For a binary predictor that outputs \(\hat{\pi}(X)\), ideal calibration means

\[\begin{equation} \Pr(Y=1\mid \hat{\pi}(X)=s) = s \qquad \text{for all } s\in(0,1). \label{eq:ch11-calibration} \end{equation}\]

A model can have high accuracy and poor calibration. In biology and medicine, this distinction matters because predicted probabilities are often used for prioritization, screening, or decision support. The final warning is distribution shift:

\[\begin{equation} P_{\mathrm{train}}(X,Y) \neq P_{\mathrm{test}}(X,Y). \label{eq:ch11-distribution-shift} \end{equation}\]

This inequality is common rather than exceptional. Assay protocols change, batches differ, populations differ, sequence databases are unevenly sampled, and images or omics profiles may carry site-specific signatures. Foundation models can sometimes improve robustness because pretraining exposes the model to broader variation. They can also amplify hidden biases from the pretraining corpus.

11.9 What Changes, And What Does Not

Deep learning changes high-dimensional biological statistics in three ways. First, representation learning becomes central: the data matrix \(X\) may be replaced by an embedding matrix \(H\) learned from images, sequences, counts, graphs, or multiple modalities. Second, prediction and preprocessing become more tightly coupled: normalization, denoising, imputation, and classification may be trained inside a single model. Third, foundation models introduce a separation between large-scale unsupervised or self-supervised pretraining and smaller supervised downstream analyses.

What does not change is just as important. Biological data still have dimensions, distributions, nuisance variables, and sampling designs. Count data remain count data. Compositional data remain compositional. Binary labels remain binary. A pretrained embedding is not automatically causal, interpretable, calibrated, or free of batch effects. The last layer of a foundation-model workflow is often a classical regression or classification problem, and the credibility of the result still depends on careful validation.

The course therefore ends where it began: with the relationship between biological measurement, statistical modeling, and scientific claims. Deep models provide powerful new function classes and pretrained representations. High-dimensional statistics provides the language for asking what was measured, what was learned, what was validated, and what can reasonably be concluded.

References

Ching, Travers, Daniel S. Himmelstein, Brett K. Beaulieu-Jones, et al. 2018. “Opportunities and Obstacles for Deep Learning in Biology and Medicine.” Journal of The Royal Society Interface 15 (141). https://doi.org/10.1098/rsif.2017.0387.
Eraslan, Gökcen, Lukas M. Simon, Maria Mircea, Nikola S. Mueller, and Fabian J. Theis. 2019. “Single-Cell RNA-Seq Denoising Using a Deep Count Autoencoder.” Nature Communications 10 (1). https://doi.org/10.1038/s41467-018-07931-2.
Hayes, Thomas, Roshan Rao, Halil Akin, et al. 2025. “Simulating 500 Million Years of Evolution with a Language Model.” Science 387 (6736): 850–58. https://doi.org/10.1126/science.ads0018.
Hu, Edward J., Yelong Shen, Phillip Wallis, et al. 2021. LoRA: Low-Rank Adaptation of Large Language Models. https://arxiv.org/abs/2106.09685.
Jumper, John, Richard Evans, Alexander Pritzel, et al. 2021. “Highly Accurate Protein Structure Prediction with AlphaFold.” Nature 596 (7873): 583–89. https://doi.org/10.1038/s41586-021-03819-2.
Rives, Alexander, Joshua Meier, Tom Sercu, et al. 2021. “Biological Structure and Function Emerge from Scaling Unsupervised Learning to 250 Million Protein Sequences.” Proceedings of the National Academy of Sciences 118 (15): e2016239118. https://doi.org/10.1073/pnas.2016239118.
Senior, Andrew W., Richard Evans, John Jumper, et al. 2020. “Improved Protein Structure Prediction Using Potentials from Deep Learning.” Nature 577 (7792): 706–10. https://doi.org/10.1038/s41586-019-1923-7.
Zhou, Jian, and Olga G Troyanskaya. 2015. “Predicting Effects of Noncoding Variants with Deep Learning–Based Sequence Model.” Nature Methods 12 (10): 931–34. https://doi.org/10.1038/nmeth.3547.