Recently, I've been testing AI's performance for reviewing manuscripts. In addition to the general purpose AI (e.g. ChatGPT), I also work together with developers of AI reviewers. The review-specialized AI performed amazingly! I was very surprised that they could caught  inconsistences or flaws in the studies and find the right data to compare. Some of them can even evaluate the conceptual differences between studies. 

When mentioning these testing in social media, some commentors expressed their concerns, including data privacy (very important issue for all AI tool developers), their capacity to interpret the results, etc. On top of them, there was worrying voice that AI will not and should not replace human reviewers; AI could not understand novelty; the scientific publishers will use AI to abuse the current publication system. 

These are all fair and important comments. However, I think that AI will not be incorporated into the current journal-centered scientific publication system. Instead, it will create a new mode of publication. Imagine this: when you look for papers of your interest, the search results list not only individual papers, but also a summary of AI review comments and even scores in quality, relevance, reproducibility, and degree of uniqueness. You choose the paper to download because the review result suggests it's the most reliable one and can best guide your study design, not because it is published in the so-called high-impact journal. In other words, you don't need journals to select papers for you. The impact factor of the journal and citation number of a paper will become meaningless.   

When the research institutes and funding agencies implement AI reviewers into their system, they no longer need impact factors of the journals to evaluate the performance of the researcher. The author can use AI to get her/his preprint reviewed, and post the review report together with the preprint, and compare her/his study with other preprints and/or published papers.

I don't mean that AI reviewer can fully evaluate conceptual novelty or breakthrough and determine the scientific value of a study. That's impossible at this moment. Here is my point: impact factor does not have such function, either. However, the academics allow the impact factor to dictate performance evaluation, funding criteria, or decision making in someone's career. The time and cost to chase publication in the so-called high-impact journals have become every researcher's nightmare. The current publication mechanism contributes to the creation of caste system in academic community (https://communities.springernature.com/posts/serfs-predators-and-robots-the-past-and-future-of-scientific-publishing).

The machine is not solution for all the problems in academic research. However, it is a tool to give some control power of publication process back to researchers, with reasonable cost. This is why I am so optimistic about the development of AI reviewers.