So these days I try not to simply make passing reference to the fact that a particular practice does or doesn't have an evidence-base that can be located in the peer-reviewed literature when I talk to such audiences. If time permits, I take a moment to try to unpack what this means, and also outline its importance, as well as some of the traps for young players concerning peer-reviewed publications.
I have mentioned the peer-review process briefly in a previous blogpost, but will cover it in a little more detail here.
What is "peer review" and who are these "peers"?
One of the most important considerations when assessing new research findings is their source. If I were to tell you about a new cancer treatment that seemed to be achieving unusually positive results, but the only place it was described was in a popular women's magazine, you'd probably be a bit skeptical, and would want to know that it had been published in a medical journal. And rightly so.
Peer review refers to the process of subjecting one's academic work to the scrutiny of one's peers in the scientific community - other academics who have both content knowledge (e.g., of a particular disorder) and methodology knowledge (i.e., of how to go about rigorously studying a disorder and/or its management). It also helps when the reviewers have some knowledge of the existing literature, so they can say "The answer to this question is already known", or "There's important studies missing from the literature review" (hopefully not just their own.....that's an aspect of peer review for another day).
By the time researchers have spent many, many hours working on a study manuscript, they typically have a bit of a love-hate relationship with it. They are tired of having to edit it to meet journal length limits, frustrated by challenges of resolving statistical and other analytic challenges, and driven to distraction trying to locate references that support passing assertions that cannot be stated without some kind of back-up (I often remind students that we can't even say "the sky is blue" in a research paper without providing supporting citations).
Eventually, the research team decides that the manuscript is ready to be tested in the crocodile-infested waters of peer review.
To do this, the researchers who have conducted the study select an appropriate academic journal that they think might be interested in publishing on their chosen topic. Historically, academic journals were published by professional bodies, and in many instances that is still the case. Some of these links are quite obvious - the British Medical Journal, for example, is the official research publication of the British Medical Association. Academic publishing, however, is a big, competitive business, so international publishing houses create new titles where they think there will be an adequate market. Consequently knowledge generation and dissemination is a rapidly proliferating enterprise, and there is a quality-hierarchy of journals, based to a large extent on the laws of supply and demand.
In disciplines such as health, education, and psychology, there are hundreds, if not thousands of journal alternatives from which to chose. So you might ask, "If they're all peer-reviewed journals, what does it matter?"
It matters because the bar is set much higher in some journals than others. A highly-ranked journal in medicine, psychology, social sciences, or education might receive hundreds of article submissions per week, and be in a position to publish only 5-10% of these. That means they can be very fussy about the originality, rigour, and scope of papers they anoint (whether they are original research or a review of existing studies). Meanwhile the Journal of Acme Research down the road might be scraping to find sufficient material to fill a forthcoming issue, and so will gladly receive submissions and be less discerning about their quality.
Does this mean "peer-reviewed" isn't an important standard?
No, it doesn't mean that at all, but it does mean that some knowledge of the rankings and status of a journal is helpful when gauging the quality and importance of papers published within its covers. Knowledge of the ideological stance of the Editor and members of the Editorial Board can also add to these insights. There are many instances of the peer-review process not working as well as it should (e.g., the famous and catastrophic case of The Lancet publishing a fraudulent study purportedly showing links between MMR vaccines and autism).Sir Winston Churchill famously said of democracy that "It is the worst form of government, except for all the others".
The same might be said of peer review as a research quality-control mechanism.
Tweet
