Data Diversity and Sequence Length: Key Levers for Powerful Biological AI

## The two levers

A biological foundation model is a deep network pretrained by self-supervision on unlabeled protein or nucleotide sequence, then fine-tuned or prompted for a downstream task. Its value is reuse: one pretrained model predicts protein function, mutational effect, subcellular localization, or chromatin profile with far less labeled data than a task-specific model, and inference is often orders of magnitude faster than the alignment-based methods it displaces. Two properties of the pretraining regime set the ceiling on that reuse. The first is the biodiversity of the corpus — how many species, environments, and functional classes the sequences span. The second is the sequence input length — the maximum context the model can ingest in one pass. Model architecture and parameter count enter as a third, coupled variable: architecture decides whether long context is affordable and whether diverse data is learnable without underfitting.

## Biodiversity of the pretraining corpus

Corpus biodiversity spans orders of magnitude. The curated Swiss-Prot subset of UniProt holds on the order of 10⁵ sequences; UniProt as a whole holds 10⁸–10⁹; metagenomic assembly efforts contribute billions more, though only 1–2% of those proteins align to a known functional annotation. Genomic models sample the tree of life at varying breadth: the Nucleotide Transformer integrates more than 3,000 human genomes plus additional species, Evo trains on more than 80,000 bacterial and archaeal genomes, and LookingGlass reaches strong transfer from just 330 genomes when they are sampled uniformly across that tree. Broader corpora encode richer biochemical priors and generalize to unfamiliar taxa. The gains do not compound indefinitely: single-cell transcriptomic pretraining plateaus once training data reaches tens of millions of cells, and indiscriminate training on divergent or erroneous sequence injects noise that can hinder performance. Diversity is a lever with a stop; past it, quality filtering and balanced sampling of under-represented taxa matter more than raw volume.

## Sequence length and the architectures that permit it

The dependencies that make length matter are structural: regulatory elements act on genes thousands of base pairs away, and residues distant in sequence contact one another in a folded protein. Early foundation models could not see them. Protein language models trained on windows of a few hundred to roughly 1,000 amino acids; early genomic models on 512–4,096 tokens, under 0.001% of a human chromosome. The limit is architectural. Self-attention cost scales quadratically in sequence length L, so extending context under a standard transformer is prohibitive.

State-space and long-convolution models break the scaling. HyenaDNA uses implicit long convolutions to reach a context window of one million base pairs at single-nucleotide resolution; Mamba's selective state dynamics achieve linear-time inference and hold up on sequences of millions of tokens where transformers fail. These architectures make megabase context tractable. They do not make it free: focusing capacity on long-range structure can degrade tasks that need only local features, and very long inputs introduce new failure modes such as overfitting to spurious long-range correlations.

## Where the two levers collide

The two levers are neither independent nor aligned. Biodiversity is concentrated in short sequences: metagenomic reads, the most diverse data available, run 100–300 base pairs and cannot carry a long-range dependency at all. Long context, conversely, is exploited on assembled genomes, which are fewer and taxonomically narrower — Evo pairs its megabase-scale capability with prokaryotic genomes only. A single corpus cannot maximize both at once. The synthesized verdict is blunt: adding raw data or raw context without a matching architecture yields diminishing returns, and the largest untapped gains lie in architectures that fuse short diverse reads with long assembled context rather than in scaling either lever alone.

                    +-----------------------+
                    |  Dataset biodiversity |
                    +-----------+-----------+
                                | generality of features
                                v
  +---------------+     +---------------+     +---------------------+
  |   Sequence    |---->|   Transfer    |<----| Model architecture  |
  | input length  |     |   learning    |     | & size              |
  +---------------+     |  performance  |     +---------------------+
   contextual depth     +---------------+       learning capacity

## The dual-use consequence

The generative capacity that improves biodefense is the capacity that lowers the barrier to harm. A model that designs functional enzymes from diverse data can propose a protein with a trait combination absent from nature; a model that reads regulatory context can reason over pathways rather than single genes. The chapter treats the biosecurity exposure as intrinsic to capability, not incidental to it, and names concrete controls: excluding sequences from known high-risk pathogens during data curation, constrained decoding and output filtering at generation, and watermarking to trace AI-designed sequences to their source. It closes on the venue's concern — for NATO and its member nations, the levers that set model capability also set the terms of governance, and neither a model's diversity nor its context length can be procured without asking what it was trained on and what it can be made to generate.