Bibliometric Exercise or Scientific Excellence?

Research rankings can inspire scientific ambition, especially in resource-scarce regions. But scientific excellence is much bigger than a number.

Published in Research Data

Bibliometric Exercise or Scientific Excellence?

Share this post

Choose a social network to share with, or copy the URL to share elsewhere

This is a representation of how your post may appear on social media. The actual post will vary between social networks

Science has developed remarkable ways of measuring the world, from the age of stars to changes in the climate. Yet, when measuring scientific excellence itself, we often rely on publications, citations, h-index scores, journal impact factors, and international rankings.

These indicators are useful. They help us understand research productivity and academic influence. The problem begins when we mistake what they measure for the overall quality and value of science.

The Stanford–Elsevier database of the world's top 2% scientists offers a useful starting point. It identifies researchers using standardised citation indicators. However, the significance of such recognition differs between regions.

For a resource-scarce region like Kashmir, where research infrastructure, funding, and institutional support may be limited, inclusion in an internationally recognised list can inspire young researchers and encourage universities to invest in science. It demonstrates that researchers from less-resourced regions can gain international recognition. In this context, the list has genuine motivational value.

In established research systems across North America and Europe, and in countries such as China, Japan, and Russia, international rankings may serve a different purpose. This does not mean researchers in these countries disregard such lists. Rather, the motivational importance of a ranking depends partly on the research environment in which it operates.

For Kashmir, therefore, the list deserves appreciation. But encouragement and scientific excellence are not the same thing.

A citation records a connection between scholarly publications. It may indicate that research has influenced subsequent work, but it does not automatically establish that a finding is correct, original, reproducible, or socially beneficial.

Hirsch (2005), who introduced the h-index, combined publication productivity and citation impact into a single indicator. Its simplicity helped make it popular. However, no single number can capture the full range of a researcher's contributions.

Seglen (1997) explained why journal impact factors should not be used to evaluate individual research contributions. A journal's average citation rate cannot establish the quality of every article it publishes.

A measurement can be useful without being comprehensive. This distinction is central to understanding research rankings.

The problem arises when universities treat rankings as symbols of overall excellence. The number of listed researchers becomes an institutional achievement, while citation scores enter performance assessments and promotional material.

Researchers may then feel pressure to publish more frequently, select fashionable subjects, or prioritise projects likely to attract citations. Publication and citation are essential to scientific communication, but they become problematic when measurable recognition takes priority over meaningful discovery.

Hicks et al. (2015), in Nature, argued that quantitative indicators should support expert judgement rather than replace it. The Leiden Manifesto also emphasises disciplinary differences, institutional missions, and the importance of locally relevant research.

Similarly, the San Francisco Declaration on Research Assessment cautions against using journal-based metrics as substitutes for evaluating the actual scientific contribution (Cagan, 2013).

The message is simple: measure research, but do not allow the measurement to define research.

Academic influence, scientific quality, and societal value are related, but they are not identical.

Academic influence concerns how research informs subsequent scholarship. Scientific quality concerns originality, methodological rigour, and the reliability of evidence. Societal value concerns whether research contributes to improvements in health, technology, the environment, public policy, or human well-being.

A theoretical discovery may transform scientific understanding without immediate practical applications. An environmental study may help a village manage wastewater. A long-term ecological investigation may reveal changes that short-term observations cannot detect.

Such contributions cannot always be judged by the same indicators.

This does not mean that locally relevant research is automatically excellent or that highly cited research lacks social value. Every scientific claim must withstand scrutiny. The point is that different achievements require different forms of assessment.

Moher et al. (2020), through the Hong Kong Principles for assessing researchers, emphasised the importance of research integrity, transparency, open science, and recognising a broader range of contributions.

Research opportunities are not equally distributed. Some institutions have advanced laboratories, substantial funding, and established international networks. Others struggle with limited infrastructure and fewer opportunities for sustained investigation.

Citation indicators cannot, by themselves, distinguish differences in research ability from differences in research opportunity.

Citation practices also vary across disciplines. Some fields generate citations rapidly, while others require years of observation before meaningful conclusions emerge. Comparing researchers without considering these differences can produce misleading assessments.

There is another danger. Researchers may become reluctant to pursue difficult questions whose answers are uncertain or whose results will take years to emerge. Foundational science, long-term environmental monitoring, and practical technology development often require patience rather than a steady stream of publications.

Wilsdon et al. (2015), in The Metric Tide, examined the benefits and limitations of quantitative research assessment. Their work highlighted the need for context and warned, more broadly, that indicators can influence the behaviour they are intended to measure.

If institutions reward only what is easy to count, research priorities may gradually shift towards what is easiest to measure.

A better approach combines quantitative indicators with informed judgement.

Researchers should be assessed on the originality of their work, the reliability of their methods, the strength of their evidence, and the significance of their findings. Relevant contributions may include publications, datasets, software, patents, validated technologies, and evidence of influence on policy or practice.

Assessment should also consider disciplinary differences, career stages, available resources, and institutional missions. The criteria for evaluating fundamental physics cannot simply be applied unchanged to environmental research, public health, or the humanities.

Transparency is equally important. Institutions should explain which indicators they use, why they use them, and what conclusions those indicators can reasonably support.

The objective is not to replace numbers with subjective opinions. It is to combine reliable measurements with informed judgement.

For Kashmir, the challenge is not to stop celebrating international recognition. It is to turn recognition into stronger scientific capacity.

Universities should ask not only how many researchers appear on international lists, but also what they have discovered, how reliably their work advances knowledge, and whether it addresses important scientific questions.

The region needs both recognition and the conditions for sustained research: reliable funding, stronger laboratories, research freedom, interdisciplinary collaboration, and opportunities to investigate local problems.

The same principle applies globally. Universities should create research cultures that reward originality, reliability, intellectual independence, and meaningful contributions to knowledge and society.

Research rankings can encourage scholars, reveal patterns of influence, and bring visibility to institutions. Researchers who appear on these lists deserve recognition. But absence from a ranking does not establish a lack of merit, just as inclusion does not automatically prove exceptional originality.

Scientific excellence is not a single quantity waiting to be calculated. It is a judgement informed by evidence, originality, rigour, significance, and context.

A ranking can tell us how researchers perform according to specified indicators. It cannot, by itself, tell us whether the right questions are being asked, whether discoveries will withstand the test of time, or whether scientific knowledge is improving people's lives.

We should celebrate rankings when they inspire better science, but never mistake the ranking for the science itself.

References

1. Cagan, R. (2013). The San Francisco Declaration on Research Assessment. Disease Models & Mechanisms, 6(4), 869–870. https://doi.org/10.1242/dmm.012955

2. Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520, 429–431. https://doi.org/10.1038/520429a

3. Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences, 102(46), 16569–16572. https://doi.org/10.1073/pnas.0507655102

4. Moher, D., Bouter, L., Kleinert, S., et al. (2020). The Hong Kong Principles for assessing researchers: Fostering research integrity. PLOS Biology, 18(7), e3000737. https://doi.org/10.1371/journal.pbio.3000737

5. Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ, 314, 497–502. https://doi.org/10.1136/bmj.314.7079.497

6. Wilsdon, J., Allen, L., Belfiore, E., et al. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. Higher Education Funding Council for England.