1 Course Overview, Biology Recap, and Statistical Workflows
Chapter Aim
This chapter sets up the course. We first establish the biological motivation: modern biology can now measure genomes, transcriptomes, proteomes, metabolomes, microbiomes, images, and other molecular or cellular signals at large scale. We then introduce the statistical object that will appear throughout the lecture:
\[\begin{equation} X \in \mathbb{R}^{n \times p}. \label{eq:ch01-analysis-matrix-opening} \end{equation}\]
Here rows are biological observations and columns are molecular features. The central difficulty is that \(p\), the number of measured features, is often much larger than \(n\), the number of observations. This high-dimensional structure makes visualization, normalization, modeling, testing, prediction, and interpretation tightly connected.
Notation follows the shared convention in the notation chapter. Vectors are columns by default, row vectors are written with transposes, and dimensions should be checked whenever matrices are multiplied, transformed, modeled, or decomposed.
For the statistical philosophy behind this chapter, read the MSMB Introduction. For the role of plots in statistical workflows, see MSMB Data Visualization. For experimental design, confounding, quality control, and reproducibility, see MSMB Experimental Design.
Learning Goals
After this chapter, students should be able to:
- Explain why high-throughput biological measurements naturally lead to high-dimensional data.
- Interpret \(X \in \mathbb{R}^{n \times p}\) as a samples-by-features or cells-by-features data matrix.
- Distinguish raw count matrices \(C\) from processed analysis matrices \(X\).
- Give examples of biological observations and molecular features across omics data types.
- Describe common statistical tasks: visualization, prediction, association testing, clustering, network construction, and regression.
- Explain why high dimensionality, compositionality, sparsity, noise, batch effects, and confounding make biological data analysis difficult.
- Describe the course workflow: biological question, measurement technology, data matrix, preprocessing, exploratory analysis, modeling, validation, and interpretation.
1.1 Why This Course Exists
Modern biology is no longer limited to measuring one gene, one protein, or one phenotype at a time. A single experiment can now produce thousands to millions of measurements. Sequencing technologies can read DNA fragments, quantify RNA molecules, profile microbial communities, or measure genomic variation across many individuals. Other technologies measure proteins, metabolites, images, spatial locations, or combinations of these modalities.
This creates a new kind of statistical problem. The data are rich enough to reveal biological structure, but they are also noisy, heterogeneous, and high-dimensional. More measurements do not automatically mean more understanding. Without careful statistical workflows, high-throughput data can produce convincing but misleading patterns.
The course therefore has two linked goals. First, we need enough biological intuition to understand what was measured and what the entries of the data matrix mean. Second, we need enough statistical intuition to analyze those entries without losing track of dimensions, assumptions, and biological meaning. The main object of the course is not a single method, but the disciplined translation from a biological question to a mathematical object and then to a statistical operation.
The lecture script is a mathematical companion to the highly visual lecture slides and the course book. The slides provide intuition and examples, while the script keeps notation, assumptions, dimensions, and model statements explicit. A deeper external reference is Modern Statistics for Modern Biology, which provides worked examples, R and Bioconductor code, exercises, and more detail.
1.2 From Biology To Data Objects
The biological recap is not meant to replace a biology textbook. Its role in this course is to explain what the rows, columns, and entries of high-dimensional biological data matrices can represent.
At the level of notation, the main point is:
\[\begin{equation} \text{biological system} \longrightarrow \text{measurement technology} \longrightarrow \text{data object} \longrightarrow \text{statistical model}. \label{eq:ch01-biology-to-model} \end{equation}\]
Most chapters begin somewhere along this chain. For example, a sequencing experiment starts with DNA, RNA, or amplicons, but the statistical analysis usually begins once the measurement has been summarized as a table of counts or abundances.
The biological recap starts with the cell as the fundamental biological unit. Cells contain DNA, RNA, proteins, metabolites, organelles, membranes, and molecular machines. Many high-throughput measurements are attempts to observe one layer of this system at scale.
DNA, genes, and genomes. DNA stores hereditary information as a sequence over a four-letter alphabet. A genome is the full DNA content of an organism. Genes are DNA segments that can contribute to functional products, such as proteins or functional RNAs. The molecular basis of DNA structure is historically anchored by the double-helix model (WATSON and CRICK 1953).
Sequencing technologies read pieces of DNA or RNA. Historically, sequencing was expensive and slow. The Human Genome Project made whole-genome sequencing a landmark scientific project (Venter et al. 2001). Since then, sequencing costs have fallen dramatically, and next-generation sequencing has made large-scale genomic measurement routine.
For statistics, the important point is not only that we can read sequences. It is that sequencing creates structured data. Reads may be aligned to genomic locations, summarized as counts assigned to genes, transcripts, peaks, taxa, or variants, or converted into categorical variables such as mutation status. In microbiome experiments, sequencing reads often become relative abundances or discrete compositions. In addition, every molecular measurement is accompanied by metadata about samples, patients, tissues, batches, treatments, or experimental conditions.
These measurements become the raw material for \(C\) and \(X\). In this script, the biological story is always translated into the dimension and type of a data object. Whenever genes, cells, tissues, or sequencing reads appear in an example, the first questions are: what is the row, what is the column, and what kind of entry is being measured?
This translation from biological sequence to statistical object is developed computationally in Exercise Sheet 01, where DNA sequences are summarized by nucleotide frequencies and GC content before being connected to simple generative models and likelihood calculations (Exercise Sheet 01 2026).
Transcription and translation. The central molecular biology flow is often summarized as DNA to RNA to protein, following the central dogma of molecular biology (Crick 1970). Through transcription, a gene is copied into RNA. Through translation, ribosomes read messenger RNA and synthesize proteins.
This flow motivates several omics layers. Genomics studies DNA sequence and genomic variation. Transcriptomics studies RNA abundance and gene expression. Proteomics studies proteins and protein abundance. Metabolomics studies small molecules and metabolic states. Each layer has its own measurement technology and its own statistical complications. This is why the same broad matrix notation \(X \in \mathbb{R}^{n \times p}\) is useful, but not sufficient by itself. We must also know what the entries mean.
Transcriptomics, proteomics, and metabolomics. Transcriptomic data measure RNA abundance. In modern workflows this often means RNA sequencing, so the raw entries are usually nonnegative counts. After filtering, normalization, and variance-stabilizing or logarithmic transformations, the entries of the analysis matrix \(X\) are often treated as approximately continuous values. Early genome-wide expression studies already used matrix displays and clustering to reveal structure across genes and conditions (Eisen et al. 1998).
Proteomic data measure proteins or peptides. Their entries are usually continuous or semi-continuous abundance estimates rather than sequencing counts. They can have missing values, dynamic range issues, platform-specific noise, and batch effects. Metabolomic data are similar in that they often produce continuous intensity or abundance measurements, but they are additionally affected by compound annotation, sample preparation, and instrument sensitivity.
The key lesson is that every omics layer can be put into a matrix, but the entries do not all mean the same thing. A genomic mutation matrix may be binary or categorical. A copy-number matrix may be integer-valued or continuous after preprocessing. A transcriptomic RNA-seq matrix begins as counts. A proteomic or metabolomic matrix is often continuous. A microbiome matrix usually begins as counts but is often interpreted compositionally. These distinctions determine which probability models, transformations, distances, and tests are appropriate.
1.3 The Central Data Object
Throughout the course, the central data object is:
\[\begin{equation} X \in \mathbb{R}^{n \times p}. \label{eq:ch01-analysis-matrix} \end{equation}\]
The rows of \(X\) are observations. Depending on the experiment, an observation may be a patient, tissue sample, cell, spatial location, environmental sample, or experimental unit. The columns of \(X\) are molecular features. A feature may be a gene, transcript, protein, metabolite, taxon, genomic locus, chromatin peak, or image-derived measurement.
The entry \(x_{ij}\) is the value of feature \(j\) in observation \(i\). The row vector \(x_i^\top \in \mathbb{R}^{1 \times p}\) describes one observation across all features. The column vector \(x_{\cdot j} \in \mathbb{R}^{n}\) describes one feature across all observations.
For raw sequencing counts, we use:
\[\begin{equation} C \in \mathbb{N}_0^{n \times p}. \label{eq:ch01-count-matrix} \end{equation}\]
The count \(c_{ij}\) is the raw count for feature \(j\) in observation \(i\). After filtering, normalization, transformation, centering, or scaling, the analysis matrix is written as \(X\). This distinction matters. A method that is reasonable for transformed values \(x_{ij}\) may be inappropriate for raw counts \(c_{ij}\).
The following objects are used repeatedly:
| Object | Dimension | Interpretation in Chapter 01 |
|---|---|---|
| \(C\) | \(n \times p\) | raw nonnegative count matrix |
| \(X\) | \(n \times p\) | processed analysis matrix |
| \(x_i^\top\) | \(1 \times p\) | feature profile of observation \(i\) |
| \(x_{\cdot j}\) | \(n\) | feature \(j\) measured across observations |
| \(m_i\) | context-dependent | metadata for observation \(i\) |
| \(Z\) | \(n \times r\) | design or covariate matrix |
| \(y\) | \(n\) | response vector |
| \(Y\) | \(n \times q\) | response matrix with \(q\) outcomes |
The row dimension must match whenever objects describe the same observations. For example, if \(X\) contains molecular measurements and \(Z\) contains metadata, both have \(n\) rows.
1.4 What Can Serve As \(X\)?
The same statistical layout can represent many biological data types.
Genomic data. Features may be genomic locations, variants, copy number regions, or mutation indicators. The statistical type depends on preprocessing: a single-nucleotide variant can be binary, categorical, or encoded as an allele count, while copy-number data may be integer-valued or continuous after log-ratio transformation and segmentation.
Transcriptomic data. Features are often genes or transcripts. For RNA-seq, the raw data matrix \(C\) contains nonnegative counts: \(c_{ij}\) is the number of reads or fragments assigned to feature \(j\) in observation \(i\). After normalization and transformation, the analysis matrix \(X\) is typically treated as continuous. Unlike germline genomic variation, gene expression is dynamic across tissues, time, disease states, perturbations, and experimental conditions.
Proteomic and metabolomic data. Proteomic features are proteins or peptides; metabolomic features are small molecules or molecular peaks. Entries are usually continuous abundance or intensity estimates, although detection indicators, censoring, and missingness can introduce binary or semi-continuous components. These modalities are often affected by dynamic range, annotation uncertainty, instrument variation, and batch effects.
Microbiome data. Features may be taxa, operational taxonomic units, amplicon sequence variants, genes, or pathways. The raw entries are usually sequencing counts, so \(C\) is a nonnegative integer matrix. However, the observed row total depends on sequencing depth, and the data are often interpreted through relative abundances. This makes microbiome data both count data and discrete compositional data. They are also typically sparse and zero-inflated. Large-scale microbiome studies such as global ocean microbiome analyses illustrate how biological context, sequencing technology, and statistical structure meet (Sunagawa et al. 2015).
For example, if \(C\) is a microbiome count table, the row sum
\[\begin{equation} N_i = \sum_{j=1}^p c_{ij}. \label{eq:ch01-library-size} \end{equation}\]
is the sequencing depth for observation \(i\). The vector \(c_i\) can be viewed as a discrete composition constrained by \(N_i\). This is one reason microbiome analysis requires care: changing one component’s relative abundance changes the apparent proportions of others.
1.5 What Are The \(Y\) Variables?
Many statistical tasks relate \(X\) to an outcome or response \(Y\). The type of \(Y\) again matters. Disease status is binary, disease subtype or tissue type is categorical, disease stage may be ordinal, body mass index and many molecular phenotypes are continuous, and survival outcomes combine an event indicator with a time variable. Treatment response may be binary, categorical, continuous, or time-to-event depending on the study design. Environmental conditions and experimental perturbations are often encoded as covariates in a design matrix rather than as a single response.
When there is a single response for each observation, we often write:
\[\begin{equation} y \in \mathbb{R}^{n}. \label{eq:ch01-response-vector} \end{equation}\]
For multiple responses, we may use:
\[\begin{equation} Y \in \mathbb{R}^{n \times q}. \label{eq:ch01-response-matrix} \end{equation}\]
The row dimension must match \(X\): the same \(n\) observations appear in both the molecular data and the response or metadata.
1.6 What Are The Statistical Tasks?
Once \(X\), \(C\), \(Z\), and \(Y\) have been defined, the statistical task becomes a statement about one or more of these objects. The same biological data matrix can be used for prediction, association testing, clustering, dimension reduction, regression, or network construction. The task determines which mathematical object must be specified next: a prediction function, a hypothesis, a distance, a loss function, a graph, or a probability model.
Prediction. We may want to predict \(Y\) from \(X\). Examples include predicting disease status from gene expression, treatment response from molecular profiles, or obesity status from microbiome composition.
The high-dimensional challenge is that \(p \gg n\). There may be many more features than observations, so flexible models can overfit. Later chapters discuss regression, penalization, cross-validation, and prediction.
In the simplest supervised setting, the target is a vector \(y \in \mathbb{R}^n\), and the goal is to learn a function
\[\begin{equation} \hat{y} = f(X). \label{eq:ch01-prediction-function} \end{equation}\]
For a new observation \(x_{\mathrm{new}}^\top \in \mathbb{R}^{1 \times p}\), the model produces a prediction \(\hat{y}_{\mathrm{new}}\). The central statistical question is whether the learned function generalizes beyond the observed training data.
Association and biomarker discovery. We may want to test whether features are associated with a phenotype, condition, treatment, or outcome. In feature-wise analysis, we often examine one column \(x_{\cdot j}\) or \(c_{\cdot j}\) at a time.
This creates a multiple testing problem. If \(p = 20{,}000\), then testing each feature separately means testing thousands of hypotheses. Chapter 06 returns to p-values, false discoveries, and multiple testing correction.
Written formally, feature-wise testing considers hypotheses
\[\begin{equation} H_{0j}: \text{feature } j \text{ is not associated with the condition}, \qquad j=1,\ldots,p. \label{eq:ch01-feature-wise-null} \end{equation}\]
The result is a vector of p-values
\[\begin{equation} (p_1,\ldots,p_p), \label{eq:ch01-pvalue-vector} \end{equation}\]
which must be adjusted or interpreted with multiplicity in mind.
Structure discovery. We may want to discover structure within \(X\) without a response variable. This includes visualizing observations with PCA or nonlinear embeddings, clustering cells or samples, grouping features into modules, finding latent factors, and constructing networks between features. These tasks are often exploratory. They can suggest biological hypotheses, but they do not automatically prove them.
For example, dimension reduction constructs a lower-dimensional representation
\[\begin{equation} U = f(X), \qquad U \in \mathbb{R}^{n \times d}, \label{eq:ch01-low-dimensional-representation} \end{equation}\]
where \(d \ll p\). Clustering constructs labels
\[\begin{equation} z_i \in \{1,\ldots,K\}, \qquad i=1,\ldots,n. \label{eq:ch01-cluster-labels} \end{equation}\]
Both \(U\) and \(z\) depend on preprocessing choices, distance choices, and modeling assumptions.
Network construction. We may want to examine associations between features, for example gene-gene relationships, microbial co-abundance patterns, or protein interaction patterns. Network construction asks which features appear connected, conditionally dependent, or functionally related.
If a network connects features, its adjacency matrix has dimension \(p \times p\). If a graph connects observations, its adjacency matrix has dimension \(n \times n\). This distinction will matter in the network chapters.
Formally, a network is written as
\[\begin{equation} G = (V,E), \label{eq:ch01-graph} \end{equation}\]
with adjacency matrix \(A\). The dimension of \(A\) is determined by the node set \(V\), not by the word “network.” Always define the nodes.
1.7 Distributional Starting Points
Many high-throughput sequencing data sets begin as counts. A minimal entry-wise count model is:
\[\begin{equation} c_{ij} \sim \text{Poisson}(\lambda_{ij}). \label{eq:ch01-poisson-count-model} \end{equation}\]
In practice, biological count data are often more variable than a Poisson model allows. A common feature-wise alternative is:
\[\begin{equation} c_{ij} \sim \text{Negative Binomial}(\mu_{ij}, \phi_j), \label{eq:ch01-negative-binomial-count-model} \end{equation}\]
where \(\mu_{ij}\) is the expected count and \(\phi_j\) is a feature-specific dispersion parameter.
Sequencing data also have a compositional aspect. For observation \(i\), define the library size
\[\begin{equation} N_i = \sum_{j=1}^p c_{ij}. \label{eq:ch01-library-size-composition} \end{equation}\]
A row-wise model can treat the count vector \(c_i\) as a discrete composition:
\[\begin{equation} c_i \sim \text{Multinomial}(N_i, \pi_i), \qquad \pi_i \in \Delta^{p-1}. \label{eq:ch01-multinomial-composition} \end{equation}\]
This row-wise view is especially important for microbiome data, where the observed counts are constrained by sequencing depth and often interpreted through relative abundance.
1.8 Why High-Dimensional Biology Is Hard
High-dimensional biological data are difficult for several reasons.
Large \(p\), small \(n\). The number of features often greatly exceeds the number of observations. A typical biological study may have tens or hundreds of samples but thousands or tens of thousands of features. This makes ordinary low-dimensional intuition unreliable.
Measurement noise. High-throughput technologies are powerful but imperfect. Sequencing depth, amplification bias, alignment, batch effects, instrument variation, sample quality, and preprocessing choices can all affect the data.
Heterogeneity. Biological systems are heterogeneous. Samples may differ by cell type composition, disease stage, genetic background, tissue origin, treatment history, and environment. Some heterogeneity is the biological signal; some is unwanted variation.
Compositionality. Sequencing often produces counts under a finite total. In microbiome data, amplicon sequencing measures relative abundance. In RNA-seq, library size differences also matter. This means raw counts cannot be compared naively across observations.
Sparsity and zeros. Many biological count matrices contain many zeros. A zero may mean absence, low abundance below detection, insufficient sequencing depth, dropout, or technical failure. The statistical interpretation depends on the technology.
Multiple testing. Testing thousands of features creates many opportunities for false positives. A method that seems reasonable for one feature can become misleading when repeated across all features.
Interpretation. Even when a model predicts well or a pattern looks clear, the biological interpretation may be uncertain. A predictive feature is not necessarily causal. A cluster is not automatically a cell type. A network edge is not automatically a direct interaction.
1.9 A Statistical Workflow As Objects
A useful analysis workflow starts with a biological question and turns it into a target quantity. If the technology produces counts, the first mathematical object is often the raw count matrix \(C \in \mathbb{N}_0^{n \times p}\). The next objects are the metadata and covariates, usually represented by a design matrix \(Z \in \mathbb{R}^{n \times r}\), and the analysis matrix \(X \in \mathbb{R}^{n \times p}\) produced by filtering, normalization, transformation, centering, scaling, or embedding. At every stage, the analyst should be able to say what the rows are, what the columns are, and what statistical type the entries have.
After these objects are defined, the analysis can move to the statistical task. Prediction requires a function from \(X\) to \(y\) or \(Y\). Testing requires null hypotheses and a multiplicity strategy. Clustering and dimension reduction require a representation and usually a distance or loss. Network analysis requires a node set and an edge definition. Regression requires a response, a design matrix, and assumptions about the error or count distribution. Validation then asks whether the result is stable under reasonable changes, whether it generalizes to new data, and whether it is biologically plausible.
This workflow is iterative. A first plot may reveal a batch effect. A clustering result may reveal mislabeled samples. A regression model may expose a confounder. Statistical analysis often loops back rather than moving in a straight line.
This interplay between exploratory data analysis and confirmatory analysis is a recurring theme in the main reference. The relevant distinction is whether a step is being used to generate hypotheses, validate a model, or support a confirmatory claim.
The script will use this principle throughout: first define the object and its dimensions, then define the statistical operation on that object.
Suggested Reading
Read the MSMB Introduction after this chapter. Pay particular attention to the discussion of heterogeneous biological data, the large \(p\), small \(n\) problem, exploratory data analysis, and computational reproducibility.
For students who want to connect this chapter immediately to practice, MSMB Data Visualization shows how graphics support statistical thinking, MSMB Experimental Design explains why design and metadata matter before modeling, and MSMB High-Throughput Count Data gives an early view of RNA-seq count data.
The corresponding Exercise Sheet 01 moves from sequence data to count summaries, maximum likelihood, and mixture models, providing a practical bridge from the biological recap to the probability models of Chapter 02 (Exercise Sheet 01 2026).