Thursday, 25 June 2015

There's Peer-Review and then there's ....peer-review

The term "peer-review" is an inherent aspect of academic life, but like so many terms that have highly specific meanings to certain groups, it is not necessarily widely understood outside the hallowed halls of universities and research centres. I try to be cognisant of this when I give presentations to audiences of practitioners such as teachers and welfare staff, as there is really no reason why we should assume shared knowledge or understanding of research-related terms. Such audiences have their own specialised lexicons which can be equally opaque to those of us working in academic contexts.

So these days I try not to simply make passing reference to the fact that a particular practice does or doesn't have an evidence-base that can be located in the peer-reviewed literature when I talk to such audiences. If time permits, I take a moment to try to unpack what this means, and also outline its importance, as well as some of the traps for young players concerning peer-reviewed publications.

I have mentioned the peer-review process briefly in a previous blogpost, but will cover it in a little more detail here.

What is "peer review" and who are these "peers"?

One of the most important considerations when assessing new research findings is their source. If I were to tell you about a new cancer treatment that seemed to be achieving unusually positive results, but the only place it was described was in a popular women's magazine, you'd probably be a bit skeptical, and would want to know that it had been published in a medical journal. And rightly so.

Peer review refers to the process of subjecting one's academic work to the scrutiny of one's peers in the scientific community - other academics who have both content knowledge (e.g., of a particular disorder) and methodology knowledge (i.e., of how to go about rigorously studying a disorder and/or its management). It also helps when the reviewers have some knowledge of the existing literature, so they can say "The answer to this question is already known", or "There's important studies missing from the literature review" (hopefully not just their own.....that's an aspect of peer review for another day).

By the time researchers have spent many, many hours working on a study manuscript, they typically have a bit of a love-hate relationship with it. They are tired of having to edit it to meet journal length limits, frustrated by challenges of resolving statistical and other analytic challenges, and driven to distraction trying to locate references that support passing assertions that cannot be stated without some kind of back-up (I often remind students that we can't even say "the sky is blue" in a research paper without providing supporting citations).

Eventually, the research team decides that the manuscript is ready to be tested in the crocodile-infested waters of peer review.

To do this, the researchers who have conducted the study select an appropriate academic journal that they think might be interested in publishing on their chosen topic. Historically, academic journals were published by professional bodies, and in many instances that is still the case. Some of these links are quite obvious - the British Medical Journal, for example, is the official research publication of the British Medical Association. Academic publishing, however, is a big, competitive business, so international publishing houses create new titles where they think there will be an adequate market. Consequently knowledge generation and dissemination is a rapidly proliferating enterprise, and there is a quality-hierarchy of journals, based to a large extent on the laws of supply and demand.

In disciplines such as health, education, and psychology, there are hundreds, if not thousands of journal alternatives from which to chose. So you might ask, "If they're all peer-reviewed journals, what does it matter?"

It matters because the bar is set much higher in some journals than others. A highly-ranked journal in medicine, psychology, social sciences, or education might receive hundreds of article submissions per week, and be in a position to publish only 5-10% of these. That means they can be very fussy about the originality, rigour, and scope of papers they anoint (whether they are original research or a review of existing studies).  Meanwhile the Journal of Acme Research down the road might be scraping to find sufficient material to fill a forthcoming issue, and so will gladly receive submissions and be less discerning about their quality.

Does this mean "peer-reviewed"  isn't an important standard?

No, it doesn't mean that at all, but it does mean that some knowledge of the rankings and status of a journal is helpful when gauging the quality and importance of papers published within its covers. Knowledge of the ideological stance of the Editor and members of the Editorial Board can also add to these insights. There are many instances of the peer-review process not working as well as it should (e.g., the famous and catastrophic case of The Lancet publishing a fraudulent study purportedly showing links between MMR vaccines and autism).

Sir Winston Churchill famously said of democracy that "It is the worst form of government, except for all the others".

The same might be said of peer review as a research quality-control mechanism.

(C) Pamela Snow 2015

Monday, 8 June 2015

Renovating the Language House

This is just a very brief blogpost to say that I have again updated my schematic "Language House" that I use when talking to primary school teachers about the role of early oral language competence in both the transtion to literacy and the development of prosocial interpersonal skills.

I've blogged previously about how this schematic representation came about, and have it updated here in response to my reflections on discussions with colleagues at the recent Speech Pathology Australia Conference in Canberra.

Most importantly, I thought it was important to show that language, literacy and social skills development is dynamic and ongoing throughout childhood and adolescence (and of course across the whole life-span, but this diagram focuses on the developmental period). Hence the brick pattern on the pillars in this version, as a nod to the vertical progress that needs to occur in both the literacy and interpersonal skill domains, as manifestations of continued language development. Note too, that the walls in this model are more visibly sunken into their granite-like foundations - not just sitting on top of them.

I'm particularly indebted to Dr Ros Neilson for inspiring me to come back and do some renovations.

Please feel free to use this for professional development and related discussions, but I would appreciate being acknowledged as its source. I'd be very happy to hear your thoughts and to take further suggestions/comments.

(c) Pamela Snow 2015