• L. Barza

What makes a top-tier journal top-tier?

Updated: Sep 25, 2018

Qualities, biases and use of journal rankings to measure research quality

Black Cat Blog: Issues in academia, research, & publishing

Lydia Barza, PhD

Journal rankings are highly contested in academia. Many departments use them to determine faculty promotion and tenure, under the assumption that publishing in “top-tier”, “A-grade” journals is a measure of research quality. The logic follows that if you are successful in publishing in these journals, then your work is of superior quality. A journal’s reputation is based on a number of factors and a high ranking comes with the assumption that the journal only accepts research of the highest merit.

A fair system for assessing journal quality in itself would be useful to academics and consumers alike. With the rise of predatory journals and conferences, some kind of quality check for journals is certainly warranted.

Let’s briefly look at what the research says about how journal rankings are determined and what major biases are documented regarding this system. Others have looked at this issue from a purely mathematical perspective, either criticizing or affirming the ranking algorithms. My perspective is centered on a review of relevant research on the logic behind methods used and its impact on scholars and their work.

Spoiler alert…Articles that contribute to knowledge can be found in BOTH high- and low-prestige journals.

Qualities of “A” Journals

Almost all journal ranking systems are citation based. Citations or references to an author’s work is considered to be a major factor in determining the impact of the work. When a researcher’s work is “trending”, it is cited frequently in other academics’ work. It’s like being retweeted, liked, and shared - for nerds.

Journal prestige is the greatest factor influencing citation rates (Singh, Haddad & Chow, 2007). Top-tier journals are more frequently cited, rendering it a self-fulfilling prophecy. In fact, the more a paper is cited, the more likely it is to be cited again (Macdonald & Kam, 2007). This type of “persuasive” citation may account for up to 40% of citations and is primarily based on the reputation of the journal or scholar, rather than its merit (Bornmann & Daniel, 2008). It has also been suggested that some editors may favor articles that they predict will more likely to be cited (Adler & Harzing, 2009).

A few highly cited papers can significantly affect a journal’s ranking. For example, about 75% of papers in the journals Nature and Science were cited below the journals’ impact factors. The impact factor reflects the average number of citations to articles published. Therefore, the journal ranking may be an indication of the overall quality of the journal (in a broad sense) but does not speak to the quality of individual papers.

Despite the fact that articles in top-tier journals are cited more frequently, several studies have shown that many high quality papers are published in medium and even low quality journals (Oswald, 2007; Singh, Haddad & Chow, 2007; Starbuck, 2005). In fact, one review concluded that “experiments reported in high-ranking journals are no more methodologically sound than those published in other journals” (Brembs, 2018). In another study of over 1,000 manuscripts submitted to 3 elite medical journals, 14 of the most highly cited articles were rejected by those journals (Siler, Lee & Bero, 2015). Further, 12 of the 14 were rejected by editors and never sent for peer review. This shows that editors and reviewers of top journals are not always adept at finding the diamonds in the rough and that there are more factors at play.

“because even articles in the lowest quintile of journals may actually belong among the best 20% written, it makes no sense to dismiss these articles as valueless merely based on where they appeared” (Starbuck, 2005, p.195)

A journal is considered top-tier if it has a high rejection rate. This rate is like a badge of honor for some journals that like to tout the fact that only, say 10% of the total number of submissions are ever successful through their peer review process. Indeed, any reputable journal would not accept every paper that is submitted. However, “the more authors are encouraged to submit their papers to quality journals, the higher will be the rejection rates of these journals” (Macdonald & Kam, 2007). So, the push for academics to publish in highly ranked journals inflates the number of submissions to those journals versus ones that rank lower on the totem pole. A large volume of submissions are problematic for publishers. For example, “the higher the rejection rate of a journal, the less likely that submissions will be refereed at all”, most being either fast-tracked by editors who invite prestigious authors to submit their work or simply reviewed by the editors and not sent on for peer review (Macdonald & Kam, 2007).

Many low-tier journals publish articles that appeal to a more narrow audience. In addition, some top-tier journals reject articles that go against the grain in the field or reflect something outside the current rhetoric. Some prestigious journals also require that papers are fundamentally interesting. Some good research is simply not that interesting to most people and may, again, appeal only to a specialized group. This, however, does not make it any less valuable or speak to the quality of the work. Some of my own work is specific to a population in the Middle East. As a result, I have gotten push-back from editors and reviewers about its generalizability and applicability to a wider international audience. My intention is to have a “high impact” on the local community whether or not it appears in any high-impact journal.

Sources for Journal Rankings

Journal Citation Reports (JCR)

Journal Metrics

SCImago Journal (SJR) - SJR Educational Journals rankings spreadsheet

Google Scholar Top Publications

In short, it is suggested that limiting submissions to a narrow list of journals perverts the research process. It gives too much power to the few journals who have their own limitations in aim and scope. It limits one’s research audience and binds creativity and diversity of the work.

“Not everything that can be counted counts, and not everything that counts can be counted.”

—Albert Einstein

What determines whether a journal is top-tier adheres to some circuitous logic. For instance, top-tier journals publish authors with high reputation, but then authors acquire a high reputation by publishing in top ranked journals (Dewett and DeNisi, 2004). It is also the case that authors from top-tier schools tend to publish in top-tier journals. It’s possible that we have a chicken and egg scenario.

Not sure who started this whole thing, but when colleges review their own publication goals, they usually start by looking at journal lists by top-tier or comparable institutions. Compiled lists of quality journals, usually published by departments, are then copied by other departments (Macdonald & Kam, 2007). This, of course, does not reflect a careful and systematic review of the quality of these dissemination sources. Rather, it is a generally outdated rehash.

At a Disadvantage: Interdisciplinary Scholars, Qualitative Researchers, & Non-English Speakers

Research also reveals particular biases of top-tier journals against interdisciplinary work (Pfirman & Martin, 2010) and qualitative research. “A” journals often do not include multidisciplinary work (Adler & Harzing, 2009). In addition, they do not represent all disciplines and tend to be more generalist,