Behind the Paper: Can General-Purpose AI Models Understand Protein Functions?
Published in Protocols & Methods, Computational Sciences, and Mathematics
Behind the Paper: Can General-Purpose AI Models Understand Protein Functions?
The question that started our research
Artificial intelligence is rapidly changing scientific research. Large language models (LLMs) can summarize scientific publications, answer complex questions, and assist researchers with many knowledge-intensive tasks. As these models become increasingly capable, an important question emerges: can they also support reliable biological knowledge annotation?
One particularly important challenge in computational biology is protein function annotation. Proteins perform essential roles in living organisms, from regulating cellular processes to supporting metabolism and immune responses. Understanding what a protein does helps researchers investigate diseases, identify potential therapeutic targets, and understand how biological systems work.
Although resources such as UniProt Swiss-Prot and the Gene Ontology (GO) Consortium have supported decades of biological annotation, experimentally validated functional annotations remain limited compared with the enormous number of known protein sequences.
We became interested in whether general-purpose LLMs could help address part of this challenge. These models have learned from enormous collections of scientific and general text. But does that knowledge translate into the ability to identify precise biological functions using standardized scientific terminology?
This question motivated us to develop PROBE.
What exactly were we trying to evaluate?
Before explaining PROBE, it is important to clarify the research question.
Our goal was not to ask an AI model to discover a protein's function directly from its amino-acid sequence. Instead, we investigated whether a model could take an existing natural-language description of an experimentally characterized protein function and map that description to the appropriate Gene Ontology terms.
Gene Ontology provides a standardized vocabulary for describing biological knowledge through three major categories:
-
Molecular Function: What does a protein do at the molecular level?
-
Biological Process: Which broader biological processes does it participate in?
-
Cellular Component: Where does it operate inside a cell?
For example, a protein description might explain that a protein binds DNA and regulates gene expression. The challenge is to identify the appropriate standardized GO identifiers that represent these functions.
This seemingly straightforward task requires more than generating biologically plausible text. It requires precise identification of the correct ontology terms.
Introducing PROBE
We developed PROBE, short for PRotein functiOn Benchmark Evaluation, as a systematic framework for evaluating general-purpose LLMs on this task.
We constructed a benchmark of 1,000 proteins from UniProt Swiss-Prot, covering five organisms: humans, mice, zebrafish, yeast, and Escherichia coli.
A central design decision was to use experimentally supported Gene Ontology annotations as the reference standard. We applied evidence-based filtering to exclude computationally inferred annotations and selected proteins with sufficient experimental annotations. We then sampled 200 proteins from each organism to create a balanced multi-organism benchmark.
We evaluated ten open-source language models using five different prompting strategies. These strategies included direct zero-shot prompting, constrained instructions, few-shot examples, step-by-step reasoning instructions, and candidate-selection prompting.
The experiments were conducted without task-specific fine-tuning or retrieval augmentation. This allowed us to examine the models' existing capabilities rather than the benefits of additional biological training.
Across the benchmark, the evaluation generated 150,000 model predictions.
What did we discover?
The results revealed an important gap between the ability of language models to generate plausible biological information and their ability to produce precise, experimentally supported annotations.
Mistral Large 123B achieved the highest overall macro-F1 score of 9.3%, closely followed by Llama 3.3 70B at 9.2%. Interestingly, a simple baseline that always selected the five most frequently occurring GO terms achieved an F1 score of 6.2%, outperforming four of the ten evaluated language models.
This was one of the most striking findings of our study. Even relatively sophisticated language models can struggle when a task requires precise identification rather than general biological reasoning.
We also investigated the impact of prompting strategies. Candidate-selection prompting, in which models selected from a predefined list of 15 candidate GO terms, achieved the highest average F1 score across the five strategies. It also reduced the benchmark-unmatched prediction rate by 33.5 percentage points compared with direct zero-shot prompting.
However, this improvement came with an important distinction: candidate selection is a constrained task, not unrestricted discovery of new annotations.
Another important finding involved predictions that did not match the benchmark's experimentally supported annotations. Overall, 41.3% of predicted GO terms were unmatched to the benchmark reference set.
We investigated these predictions further rather than assuming that every unmatched term was necessarily incorrect. A temporal validation against updated UniProt records examined 240 sampled unmatched predictions. Only three (1.25%) had gained support in the updated records. This suggested that incomplete reference annotations were unlikely to explain most of the observed unmatched predictions, while recognizing that database-based validation cannot establish that every unmatched prediction is biologically incorrect.
We also observed differences across organisms. Yeast achieved the highest average F1 score, while zebrafish showed the lowest. However, the reasons behind these differences remain unresolved. Our analysis did not establish a statistically significant explanation based on literature density or annotation-pool size.
Finally, semantic similarity analysis provided another interesting perspective. Although exact GO identifier matching was often unsuccessful, the models' predictions showed stronger biological relationships to the reference annotations than random predictions. This suggests that models can capture some meaningful biological relationships even when they fail to identify the precise annotation.
Why does this research matter?
Our findings highlight an important challenge for applying generative AI to scientific workflows.
A model may produce an answer that sounds scientifically reasonable, but scientific applications often require much greater precision. In biological annotation, assigning a valid GO identifier to the wrong protein can still lead to an incorrect scientific conclusion.
This is why evaluating AI systems requires more than checking whether their responses look plausible. We need experimentally supported reference datasets, reproducible benchmarks, precise evaluation metrics, and careful investigation of errors.
PROBE provides a framework for examining these issues across multiple organisms, language models, and prompting strategies.
We also made the benchmark, evaluation pipeline, and experimental model-response records publicly available to support reproducibility and further investigation.
Looking ahead
We see PROBE as a foundation for further research rather than a final solution to protein function annotation.
Future work can investigate how retrieval-augmented approaches, specialized biological language models, structured knowledge resources, and improved grounding mechanisms might improve annotation reliability.
We also believe that expanding evaluations to more organisms and larger protein collections will be important for understanding the generalizability of these systems.
Ultimately, our goal is to contribute to a more rigorous understanding of where general-purpose AI models can help scientific research, where they still fall short, and what is needed to make their outputs more reliable.
Acknowledgements
This research was conducted at the University of Idaho by Kallol Naha and Hasan Jamil, who contributed equally to the work.
We acknowledge the University of Idaho Research Computing team for providing access to the MindRouter API gateway used for the model inference experiments.
This research was supported in part by the National Institutes of Health IDeA grant P20GM103408, the National Science Foundation CSSI grant OAC 2410668, and the USDA-NIFA Sustainable Agriculture Systems grant 2020-69012-31871.
We hope that PROBE will encourage further discussion and research on reliable AI-assisted scientific annotation.
Read the published research:
Naha, K. and Jamil, H. (2026). PROBE: a multi-organism benchmark for evaluating general-purpose large language models on Gene Ontology protein function annotation. BMC Bioinformatics.