A few teachers
have commented to me this year that they would have no idea how to determine
whether a new piece of research is “any good” or represents a change of
practice that they should adopt. While it’s probable that many
health science practitioners would also say they struggle with the task of
critically appraising new research, I think this is a particular challenge for
teachers, who historically, have not been taught about research methods and
data analysis in their pre-service training. This leaves teachers vulnerable to
“the next new thing” that policy-makers decide to introduce, and makes it hard
for them to argue their corner with any confidence.
No blog post can
adequately stand in for two, three, four or more years of research methods
training, but I thought it might be helpful here to sign-post a few key points
to de-mystify some of the research landscape for teachers.
Here’s my
Top-10 questions to keep in mind when reading about new research:
1. Where is the
study published?
a. The optimum answer
to this question is “In a peer reviewed journal”. By “peer review” we mean that the
researchers sent their manuscript to an academic journal editor, the editor
considered its general suitability for the journal, and then nominated a couple
of academic “peers” to conduct a detailed review. This process is often
conducted on a double-blind basis – i.e. the reviewer does not know the author’s
identity and vice versa. However some journals use an open-review process. The
distinction is not important for our purposes here. What matters is the level
of scrutiny the paper receives, in terms of the theoretical logic behind its
rationale, its method, data collection, analysis and interpretation.
As any academic will attest, this can be a bruising process and we often need to don a metaphorical rhino hide before opening the email with a subject line “MS 2014XYZ Decision” or similar. Reviewers rarely spend a lot of time on the study’s strengths, highlighting instead its flaws and limitations. This is not a game for the faint-hearted.
The upside though is that most published papers have undergone considerable revision by the time they go to print, and the researchers may have had to patiently and painstakingly address a myriad of queries and challenges to their argument.
As any academic will attest, this can be a bruising process and we often need to don a metaphorical rhino hide before opening the email with a subject line “MS 2014XYZ Decision” or similar. Reviewers rarely spend a lot of time on the study’s strengths, highlighting instead its flaws and limitations. This is not a game for the faint-hearted.
The upside though is that most published papers have undergone considerable revision by the time they go to print, and the researchers may have had to patiently and painstakingly address a myriad of queries and challenges to their argument.
b. But not all journals
are created equal. Academics are in the know about esteem hierarchies and metrics
such as impact factors. Universities are in the know about these as well, and bring
considerable pressure to bear on academics to publish in high-impact journals. One problem with this is that such journals may only be read by other academics and never by practitioners “on the ground” – so while it’s gratifying
to have your research cited by other researchers, it may not be translating
into meaningful, real-life change.
c. Media reports, blogs
and websites often provide accessible, easy to read summaries of research, but
should not be the primary source of research. If they are the primary source,
you should remember that the rigorous peer-review process outlined above has
almost certainly not taken place.
2. Who are the
authors?
a. Is there a well-qualified
academic on the research team? Particular knowledge of research methodology,
data collection, analysis and interpretation is needed in order to conduct rigorous
research. Look for evidence that this exists in the research team (e.g. a
university-affiliated team leader).
b. Are there potential or
actual conflicts of interest for any of the team? Examples of this might be
someone employed by a particular publishing house being part of a team that is
evaluating an intervention in which that publishing house has a commercial stake.
You should also look at the funding source(s) and ask yourself whether there
might be vested interests in the data telling a particular story.
c. What else has this
team published? What do we know about their ideological stance / bias?
(everyone has one!)
3. What was the
context of the study?
a. What country was the
study conducted in? You might read a fabulous report of a rigorous piece of
research that was conducted in Uzbekistan, and be quite confident that it is
tight and well-controlled. But if there
are significant differences between the Uzbeki educational context and your
own, you might want to think carefully before adopting any recommended changes.
b. What is the policy framework
in which the study was conducted? Are there particular teaching approaches that
are explicitly or implicitly associated with this setting?
c. What are the demographic
characteristics of the sample? Here we
need to think about socio-economic status (SES) factors, ethnicity, culture,
religious influences, age and gender characteristics and any other wider
influences on the context that might be relevant. The authors might tell you
that the study was conducted in “10 schools with similar socio-economic characteristics”,
but this doesn’t help you very much if you don’t know what those characteristics
were – i.e. were the schools in a disadvantaged area, or were they middle
or high-SES? This has important implications for the extent to which findings
can be generalised beyond the study – no
matter how rigorous the study itself may have been.
4. How clearly was
the research question stated?
a. Some studies are
highly specific with respect to their purpose and this can be easy to see even
from the title. Unfortunately, though, some research studies are a bit like
fishing expeditions – the researchers pack their gear and head out into the
wild to see what they can find. While it is absolutely appropriate for
qualitative studies to take a broader sweep around “exploring and understanding”
a phenomenon, you should always have a clear sense of what the researchers are
examining and why.
5. How adequate was
the sample and the description of the intervention?
a. Here we’re
interested in issues like sample size (e.g. the number of teachers, students,
schools etc) included. However there is no simple absolute answer to the
question “How many is enough?” Sample size should, however be based on some
kind of “power analysis” – a statistical consideration of the nature of the
questions asked and the number of participants needed to test an hypothesis.
If for example, you wanted to know about differences in vocabulary size
between four year olds and eight year olds, we would expect that age would have
a “big effect” and we would need a relatively smaller sample than if we were
studying the differences in vocabulary between four year olds and
four-and-a-half year olds – here there will be more developmental blurring
between the two groups, and so to find an age effect (assuming one actually
exists), a larger sample would be needed.
b. Is there any potential
bias/distortion due to sampling processes? An obvious issue in schools-based
research is the (usual) requirement for parent/guardian consent. However it may
be that parents from non-English speaking backgrounds cannot adequately
understand the Information Sheet and Consent Form, and so decide (quite reasonably!) to not complete them and return them to the school. This will
then introduce a systematic bias into the sample, and means findings can
really only be generalised to other groups of similar composition.
c. In studies that
involve any kind of pre-post comparison (e.g. collection of baseline data at
Time 1, an intervention phase, and collection of follow-up data at Time 2), it’s
important to think about retention of participants over time, and most
importantly to look at the characteristics of participants who were lost to
follow-up. Often, these are from minority groups or have some other defining
characteristic (e.g., frequent suspensions due to behaviour problems) that might in itself influence the Time 2 scores.
d. If it is an
intervention study, how were participants allocated to study arms (research Vs
control?) Ideally this should occur via a process of randomisation, so that
potentially confounding variables (e.g. ethnicity, IQ) are equally distributed
across study arms and so are “cancelled out” in the analysis. In some medical
research, it is possible to conduct “double blind” trials, in which neither the
participants nor the researcher interacting with them is aware of who is in
which group. This is harder to do in schools, for obvious reasons, but in
general, you should look for evidence that the researchers did not influence
the allocation of individuals or schools to one study arm or the other.
e. Also with intervention studies, ask yourself about the basis of the intervention. Does it have a theoretical rationale that draws on previous research, or is it just someone's idea about what might work? There's unfortunately been way too much of the latter in education. You should also ask whether the intervention was delivered as intended (so-called fidelity) and whether anything else might have happened during the intervention that could independently account for an apparent improvement in student performance.
e. Also with intervention studies, ask yourself about the basis of the intervention. Does it have a theoretical rationale that draws on previous research, or is it just someone's idea about what might work? There's unfortunately been way too much of the latter in education. You should also ask whether the intervention was delivered as intended (so-called fidelity) and whether anything else might have happened during the intervention that could independently account for an apparent improvement in student performance.
6. How suitable are
the measures for the questions asked?
a. If I told you that I
was going to measure children’s IQs, and then proceeded to take out a tape
measure and record their head circumferences, I think you would rightly howl me
down for using an inappropriate measure of IQ. Fortunately, extremely poor
choices such as this are not common, though it is common for researchers to select
assessment tools that others consider to lack validity (accuracy) or reliability
(consistency and trustworthiness). We also need to consider how current the
measures are and whether they are widely known and well-regarded.
b. Who conducted the
assessments / measurements? Just as we
don’t want doctors interviewing their own patients about the acceptability of a
new treatment, we don’t want teachers assessing their own students. Humans are
prone to all sorts of conscious and unconscious bias, whether as the observer
(see Rosenthal Effect) or as the observed (see Hawthorne Effect).
7. How clearly are
the results presented?
a. Are all of the
results presented, or just some of them?
b. Often it’s necessary
to have a good grasp of statistics to wade through this section of a paper, so
don’t be put off if you don’t feel you bring the necessary background knowledge
to the table. If necessary, consult with someone who is more confident with
this territory, but persevere with other sections of the paper. Many academics
would probably privately admit that they don't give this part of the paper the
focus they should – which is a shame as it’s often the most difficult to write!
8. When results are
discussed, are a range of possibilities canvassed to account for the findings,
or do the authors just stick with their original hypothesis?
a. Unfortunately there
is a well-known bias in what gets published, and findings that don’t sit well
with researcher bias and/or the prevailing zeitgeist often just don’t see the
light of day. Happily though, that is beginning to change, and academics and
journal editors alike are a little more open to publishing findings that might
be unexpected. Here we want to see a range of possibilities being canvassed,
and the importance of future replication studies being noted. The language used should be appropriately cautious and circumspect, e.g., "These findings suggest....", or "Our results are consistent with the notion that .....".
9. Are limitations
acknowledged and addressed?
a. All research has limitations,
and most researchers are acutely aware of this when they submit a paper to a
journal (if they weren't beforehand, the review process normally fixes that!). So
you should expect that some limitations and their potential importance are
considered (e.g. small or biased sample,
limited follow-up time).
10. Are implications
for theory, practice, policy and/or further research stated?
a. The purpose of
research is to effect change - in
at least one of theory, practice, and policy. So the authors should present
some ideas about the implications of their work (without over-reaching of
course) and should make constructive suggestions as to how other researchers
can advance the field even further.
This is by no means an exhaustive guide to critical appraisal and nor is it intended to be such. It is however, intended to guide the novice and instil some confidence that even without detailed statistical knowledge, you can still be an astute consumer of new research.
Remember too,
that we rarely change tack on the basis of one study. Instead we rely on
consistent trends in well-conducted research, to guide policy and practice – so
look out for systematic reviews or meta-analyses, both of which pool findings on a particular question and synthesise the
current state of the evidence.
I would
recommend that all teachers bookmark the Macquarie University MUSEC Briefings page, as this open-access site provides reliable, independent assessments of a range of approaches that
may or may not be well-supported by research evidence.
As in all
things though, it is wise to remember that when something seems too good to be true ..... it probably is.
"Trust me, I'm a researcher" is never enough.
"Trust me, I'm a researcher" is never enough.
(c) Pamela Snow 2014
