Biology benchmarks expose gaps between plausible answers and completed work
Four recent benchmark releases or updates report substantial failures on regulatory, antibody-development, rare-disease and surveillance tasks, but their different task sets cannot be read as one clinical performance ranking.
Recent biology evaluations ask agents to produce defensible decisions and artifacts rather than pass a general knowledge quiz.
In ARCA’s November 4 LifeSci Diamond update, the best reported configuration passed 16 of 55 regulatory-affairs tasks, or 29.1%, under an all-criteria rule. The benchmark combines expert-reviewed and less-reviewed answer keys and uses a model judge, so the score inherits those design choices. [1][2]
LatchBio’s November 4 TxBench–Antibody Development release tests 60 retrospective decisions across seven development stages, with three attempts per task. Its best configuration passed 39.4% of attempts, with a reported 95% interval of 28.6–50.3% calculated across task-level pass rates. Answers are checked against fixed keys; the tasks do not demonstrate that an agent can safely develop a drug or choose a patient’s dose. Only three example tasks are in the public repository, limiting outside reconstruction of the complete evaluation. [3][4][7]
LatchBio’s November 6 RareDiseaseBench reports a best attempt pass rate of 43.8%, with three attempts on each of 35 diagnostic cases requiring agents to obtain and interpret records. Its November 5 BioSecBench reports 25.5% of attempts passing, with three attempts on each of 110 surveillance tasks. Unlike LifeSci’s 16-of-55 task count, these are repeated-attempt rates, not the share of tasks solved at least once within three tries. Different inputs, grading and systems also prevent a common clinical ranking. [5][6][8][9]
Sources & details 9 sources
Discuss in your voice app
Take the reporting, visual explanations and original sources into a conversation. Copy the brief, paste it into your preferred assistant, then start its voice mode.
Paste the brief into your voice app to discuss this reporting.Reporting sources
- LifeSci Diamond November 4 update
Original September 15 benchmark updated November 4; strict pass criteria, 55 tasks, answer-key review limits and model-based grading.
- LifeSci Diamond public repository
Public task examples and benchmark documentation from the same developer; not a complete independent replication.
- TxBench report
Four-page technical report; 60 tasks, configurations, scores and confidence intervals. Rates concern attempts, three per task; confidence intervals use per-evaluation pass rates.
- TxBench public examples
Three public examples rather than the full 60-task evaluation.
- RareDiseaseBench release
November 6 developer evaluation on 35 retrospective diagnostic cases; not a prospective clinical trial. Best reported score is share of runs, with three attempts per case.
- BioSecBench release
November 5 developer evaluation on 110 surveillance tasks; developer-run results and failure analysis. Pass rate counts attempts, with three attempts per task.
- TxBench November 4 announcement
Dated first-party release and explanation; same study as the technical report.
- RareDiseaseBench technical report
Technical report linked by the announcement; corroborating documentation of the same evaluation.
- BioSecBench technical report
Four-page working manuscript; 110 tasks and best preliminary pass rate of 25.5%; same developer evaluation, not independent replication.
Images & figures
LifeSci counts passing tasks; the three Latch studies count passing attempts, with three attempts per task. These are separate evaluation designs.
Euangelion · Source · Usage terms · Download figure