Filtering viral sequences blunts zero-shot protein models, but $79 of fine-tuning restores capability
Pretraining 23 ESM-2 models without viral sequences cut zero-shot viral mutation-prediction scaling gains to 36%, yet fine-tuning restored full viral sequence recovery in 12 to 20 H100-hours.
Excluding dangerous pathogens from pretraining corpora is a widely proposed safeguard for open-weight biological foundation models.
In an October 19 arXiv preprint (arXiv:2610.26651), researchers from SecureBio and MIT tested both sides of that safeguard by pretraining 23 ESM-2 protein language models from scratch across four parameter scales (8M, 35M, 650M and 3B) and four UniRef50 viral-exclusion tiers. [1]
Before post-deployment modification, pretraining exclusion worked as intended without harming general biology. Removing all viral sequences (Tier 1) preserved 99% of the 8M-to-3B zero-shot scaling improvement on non-viral ProteinGym deep mutational scanning benchmarks while cutting scaling gains on viral benchmarks to 36% (and to 42% on human-infecting viruses under Tier 2). [1][2]
However, when model weights are released openly, the barrier is brittle: fine-tuning the 3B Tier-1 filtered model on excluded viral sequences erased the masked-language-modeling loss gap within 1.4 billion tokens—requiring 12 to 20 H100 GPU hours, or roughly $47 to $79 at commercial cloud rates. Pretraining curation can prevent accidental zero-shot viral design in hosted APIs, but open-weight biosecurity cannot rely on pretraining data filtering alone. [1]
Take it further
Read the 23-model viral data filtering and fine-tuning study For biosecurity policy teams and biological foundation-model developers evaluating pretraining curation versus weight-access controls.
Sources & details 2 sources
Discuss in your voice app
Take the reporting, visual explanations and original sources into a conversation. Copy the brief, paste it into your preferred assistant, then start its voice mode.
Paste the brief into your voice app to discuss this reporting.Reporting sources
- arXiv · Effectiveness and Durability of Viral Data Filtering in Protein Language Models
Original October 19, 2026 preprint (arXiv:2610.26651v1) by SecureBio and MIT researchers across 23 pretrained ESM-2 models.
- GitHub · OATML-Markslab/ProteinGym benchmark suite
Reference deep mutational scanning (DMS) benchmark suite used to evaluate viral versus non-viral zero-shot fitness scaling.
Images & figures
Tier-1 exclusion blunts zero-shot viral scaling to 36%, but open weights can be fine-tuned back in ≤20 H100-hours.
Euangelion · original data figure · Source · CC BY 4.0 · Download figure