Saturday, 30 March 2019

Let’s all throw out the scientific method. It’s not perfect and it's too hard. AKA The soft bigotry of gendering research methodologies.

Last week, a piece entitled The problem with using scientific evidence in education (why teachers should stop trying to be more like doctors) appeared on the AARE Blog. This was co-authored by an education academic, Dr Lucinda McKnight (Deakin University) and a medical education academic Dr Andy Morgan (Monash University). The authors purport to mount an argument as to why the notion of evidence-based practice should be resisted in education. 
I believe the article is deeply flawed at a number of levels, and have provided a detailed response to it here.
By way of background, I worked in medical education for ten years, and have also taught teachers at postgraduate level, as well as having taught across some ten allied health professions.


 McKnight & Morgan  
For teachers to be like doctors, and base practice on more “scientific” research, might seem like a good idea. But medical doctors are already questioning the narrow reliance in medicine on randomised controlled trials that Australia seems intent on implementing in education.
 My response
Where is the evidence that all that is being recommended is randomised controlled trials (RCTs)?

It is right that evidence derived from RCTs be questioned, because it is right that evidence from all research be questioned.

In medicine, there is a sound understanding of an efficacy trial Vs an effectiveness trial. This difference should be considered and discussed in education as well.
In education, though, students are very different from each other.
They are no more different, nor similar to each other than are patients. Doctors, like teachers, rely on pattern recognition to form and test hypotheses. They could not do their jobs if this wasn’t the case.
Unlike those administering placebos and real drugs in a medical trial, teachers know if they are delivering an intervention. Students know they are getting one thing or another. The person assessing the situation knows an intervention has taken place.
Yes, but students do not necessarily know which teachers have been exposed to an intervention, e.g. a series of professional learning seminars. And researchers with overall responsibility for a trial can easily be blinded to allocation group - I speak from personal experience on this.

This statement betrays an unfortunate lack of understanding of the nature of RCTs in education.
Constructing a reliable educational randomised controlled trial is highly problematic and open to bias.
Yes, doing rigorous research is challenging.

Yes, all research is open to bias.

Skilled researchers make it their business to recognise, and minimise sources of bias, and to report their findings with caution.
Before Australia decides teachers need to be like doctors, we want to tell you what is happening and give you some reasons why evidence based medicine itself is said to be in crisis
The fact that researchers are questioning an approach does not mean it is being thrown out in its entirety. That’s exactly the kind of thinking that has plagued education for decades, as shown by the tendency to adopt fads and fashions, with zero research behind them, let alone any that has any supporting evidence. I have written about this here with my colleague, Dr Caroline Bowen.
Randomised controlled trials are just one kind of evidence
And this is news because??

Of course RCTs are only one kind of evidence. That’s like saying Toyota is only one make of car.
Medicine now recognises a much broader evidence base than just randomised controlled trials.
This is also not news. Medicine has always recognised a range of study designs. What seems to be overlooked in educational discourse, however is the notion of levels of evidence.

In health, it is recognised that different study designs have different degrees of strength in establishing the efficacy or effectiveness of an approach.
Other kinds of medical evidence include: practical “on-the-job” expertise; professional knowledge; insights provided by other research such as case studies; intuition; wisdom gained from listening to patient histories and discussions with patients that allow for shared decision-making or negotiation.
These are obviously important in all fields. They just don’t sit at the top of the hierarchy as to what can be established, replicated, and/or refuted, using the scientific method.
Privileging randomised controlled trials allows them to become sticks that beat practitioners into uniformity of practice, no matter what their patients want or need. Such practitioners become “cookbook” doctors or, in education, potentially, “cookbook” teachers. The best and most recent forms of evidence based medicine value a broad range of evidence and do not create hierarchies of evidence. Education policy needs to consider this carefully and treat all forms of evidence equally.
All forms of evidence are not “equal”. This does not mean that they should not all be considered, including expert opinion, but human beings are prone to all kinds of cognitive bias

Sometimes our intuitions tell us that something “should” work, or even that it “seems” to work, but the scientific evidence counters our intuitions. This is the subject of Andrew Leigh’s book Randomistas.
Teaching is a feminised profession, with a much lower status than medicine. It is easy for science to exert a masculinist authority over teachers, who are required to be ever more scientific to seem professional.  They are called on to be phallic teachers, using data, tools, tests, rubrics, standards, benchmarks, probes and scientific trials, rather than “soft” skills of listening, empathising, reflecting and sharing.  
Where to start with this one? Let me point out a few facts:
Medicine is rapidly becoming feminised, with more females enrolled to study medicine in many Australian universities than males. Does this mean that it will now abandon centuries of commitment to the scientific method? I certainly hope not.
Dark Ages, here we come, if it does.
      There’s an assumption here that the scientific method would be an imposition on those poor feeble women in teaching, who would not be able to cope with the rigours of its analytic tools.
How insulting.
     There is no connection between genitalia and the tools of scientific inquiry. This is just silly. What about all those women who conduct (and use) quantitative education research?
Where does this leave them?
      So-called “soft skills” are just as important in medicine as knowledge of human biosciences, pharmacology, and so on. No-one is suggesting otherwise.

Why can’t doctors and teachers be content experts AND competent consumers of new research? When did it become either-or?

A Western scientific evidence-base for practice similarly does not value Indigenous knowledges or philosophies of learning. Externally mandated guidelines also negate the concepts of student voice and negotiated curriculum.
Education and medicine both need to show a deep respect for and understanding of indigenous knowledge and practices. That does not mean that the Aboriginal man presenting to the Emergency Department with chest pain automatically wants to receive a different type of care from his non-indigenous counterpart. If the latter receives an immediate ECG and blood tests, then so should the Aboriginal patient. There is a place for the student/patient voice and there is a place for professionals to do what professionals are trained and paid to do.
While confident doctors know the randomised controlled trial-based statistics and effect sizes need to be read with scepticism, this is not so easy for many teachers. If randomised controlled trial-based guidelines are to rule teaching, teachers will also potentially be monitored for compliance with guidelines they may not fully understand or accept, and which may potentially harm their students.
If teachers are not confident in interpreting research studies (and I agree they are not), then education faculties need to step up and teach them how to be critical consumers of research – quantitative, qualitative, and mixed methods.

All professionals are monitored for compliance – that’s part of what being a professional means. It is a highly constrained form of public accountability. I have blogged about this previously.
Evidence based medicine is about populations, not people
The fallacy here of course, is that populations are not made up of people. Evidence-based medicine is about using robust study designs to control a range of sources of error so that appropriate conclusions are drawn. It is then up to the individual practitioner to consider the findings in the course of their clinical decision making on a case-by-case basis. As noted below, this entails consideration of evidence, patient values, and clinical resources. But the evidence part is non-negotiable.
While medical randomised controlled trials save lives by demonstrating the broad effects of interventions, they make individuals and their needs harder to perceive and respect.  Randomised controlled trial-based guidelines can mean that diverse people are forced to conform to simplistic ideals. Rather than starting with the patient, the doctor starts with the rule. Is this what we want for teaching?.
Well at least we have some acknowledgement here that RCTs can help to save lives!

It is not the role of an RCT to bring individuals into sharp focus. We have many other study designs that do that much better, and they are considered alongside the findings of RCTs in the development of treatment guidelines.

When medical guidelines are applied in rigid ways, patients can be harmed
When any guidelines are applied in rigid ways people can be harmed. There is nothing illuminating about this statement.

Anyone who is familiar with the pioneering evidence-based medicine work of Dr David Sackett and his colleagues will know that this model emphasises empirical research + patient values + clinical resources. It is not, and never has been, about research evidence alone. Right from the start in medicine, it was emphasised that evidence-based medicine is not a cook-book approach. This is just a straw man.

Interestingly, evidence-based practice was initially seen as an unnecessary imposition on medicine in its early days. Now it underpins the way we educate all health professionals, and the community is the beneficiary – both as patients and as tax-payers.
Trials cannot be done on every single kind of person and so inevitably, many individuals are forced to have treatments that will not benefit them at all, or that are at odds with their wishes and beliefs.
This is another nonsensical truism. No, we cannot include all kinds of people on planet earth in trials (clinical or educational). The whole purpose of research is that we sample from populations, in an effort to generalise back to the population as closely as possible.

Welcome to Research Methods 101.

Just because rigorous methodologies cannot answer every question, for every patient, every time, does not mean they are not the best horse in the race to back.
Educators need to ensure that teachers, not bureaucrats or researchers, remain the authority in their classrooms.
Well a good way to make a start on this would be for education faculties to equip pre-service teachers with scientifically-derived knowledge and skills on (for example) the teaching of literacy and numeracy, as well as the ability to read and critique new research and make decisions about how this should inform practice.

Teachers cannot speak with authority if they do not know the research behind an approach and the extent to which this is contested. This is why, for example, medical students are taught that prescribing antibiotics for children with middle ear infections is controversial. They know the ground will shift under them over time, as the science changes, and are primed to watch for new evidence as it arises, and adjust their practice accordingly.

This is called accountability.
Scientific evidence gives rise to gurus
A more critical and discerning teaching workforce would counter this in a flash – in the same way that it does in medicine. Gurus flourish where audiences can be easily wooed and charmed by pretty graphs and impressive looking numbers.
While medical-style guidelines may seem to have come from God, such guidelines, even in medicine are often multiple and contradictory. The “cookbook” teacher will always be chasing the latest guideline, disempowered by top-down interference in the classroom.
Yes, this is the nature of scientific evidence. It changes, and is sometimes contradictory. Rather than “chasing” the latest guideline, professionals need to avail themselves of new evidence and work out how it should influence their practice. 

This is called accountability.
In medicine, over five years, fifty percent of guideline recommendations are overturned by new evidence. A comparable situation in education would create unimaginable turmoil for teachers.
The paper linked to here states “This investigation sheds light on low-value practices and patterns of medical research”. Wouldn’t this be a good thing in education too? That way, we might never go down the Brain Gym, coloured lenses, learning styles, multiple intelligences, left brain-right brain, brain-based learning (etc) time wasting and expensive rabbit holes that education is so fond of.

One of the challenges of living in a knowledge economy is that information changes. We all have an obligation to keep up as much as possible. Choosing your own adventure, whether as a doctor or a teacher, is not acceptable to the community.
Evidence-based practice risks conflicts of interest
Then let’s be careful not to throw the baby out with the bath water. The more discerning and informed teachers (and doctors) are, the less prone they will be to commercial interests. This is part of the imperfect world in which we live and is not a reason to abandon the scientific method.

There are plenty of commercial interests at work in classrooms around the world today, regardless of the level of evidence underpinning the teaching that is occurring.
Randomised controlled trials in medicine routinely produce outcomes that are to the benefit of industry. Only certain trials get funded. Much unfavourable research is never published. Drug and medical companies set agendas rather than responding to patient needs, in what has been described as a guideline “factory”.
These are all legitimate concerns about health research that need to be managed. They are not reasons to abandon the scientific method. See babies and bath water, above.
Do we want what happens in classrooms to be dictated by profit driven companies, or student-centred teachers?
As noted above, there are plenty of profit-making companies doing their thing in classrooms around the world right now, cashing in on the fact that teachers are a soft target for approaches with a slick marketing spin, and a few researchy-sounding words in the glossy brochure and on the equally glossy box.

The purpose of having a more research-informed teaching workforce is being able to head off snake-oil sales people at the school gate.
We call for an urgent halt to the imposition of ‘evidence-based’ education on Australian teachers, until there a fuller understanding of the benefits and costs of narrow, statistical evidence-based practice. In particular, education needs protection from the likely exploitation of evidence-based guidelines by industries with vested interests.
Ironically, such a halt wouldn’t cause a great deal of disruption, given the limited extent to which evidence-based practice has genuinely found its way into education discourse.
Rather than removing teacher agency and enforcing subordination to gurus and data-based cults, education needs to embrace a wide range of evidence and reinstate the teacher as the expert who decides whether or not a guideline applies to each student.
Perhaps we need to consider the possibility of enhanced teacher agency, in a world where teachers are knowledgeable and confident consumers of new research, by virtue of their grasp of research methodologies and critical appraisal skills?

We can’t consider “a wide range of evidence” while disregarding evidence from RCTs, and not understanding the notion of levels of evidence means that equal weight is inappropriately assigned to a single case study and a meta-analysis of several RCTs. They all contribute to the understanding of an issue, but not necessarily equally, on a study-by-study basis.

One of the key differences between medicine and education, is that doctors frequently need to gain informed consent for their actions. In education, however, consent is implied. Students cannot give or withhold their consent for a particular instructional approach. It just comes their way, like it or not. This only serves to increase, not decrease the ethical burden on teachers to teach in ways that are supported by strong empirical evidence.
Pretending teachers are doctors, without acknowledging the risks and costs of this, leaves students consigned to boring, standardised and ineffective cookbook teaching. Do we want teachers to start with a recipe, or the person in front of them?
No-one is pretending teachers are doctors. But if they want to be afforded at least some professional autonomy, then they have to accept professional accountability, just like other professions do. We all need to acknowledge, however, that no profession is completely autonomous, least of all medicine. We all need to be accountable to our “consumers”, our employers, our professional bodies, and the community.

No, we don’t want recipes, but nor do we want random chaos, and the wild west of everyone choosing their own adventure, either. The Age of Enlightenment created some enduring legacies that we would all do well to hang on to.

As I have stated previously:

Education and medicine, for example, have a great deal in common; they both concern people, interactions between people, complex co-occurrences, and hard-to-control (actually impossible to control) variables, such as race, gender, ethnicity, religion, intelligence, empathy, sometimes unpredictable and seemingly inexplicable behaviour, resource limitations, and the need to establish trust and rapport.

Most importantly, both have to deal with uncertainty, coupled with a weight of responsibility and accountability to communities, peers, and policy-makers for outcomes

Perhaps some in education think that they can dispense with research rigour because children can't be harmed by sub-optimal or mis-directed classroom practices. 

They are wrong.

(C) Pamela Snow (2019)


  1. kerry hempenstall30 March 2019 at 13:09

    Yes, Pam, you may need some time in the garden after wading through, and responding to, this blog. Ironically, it highlights why a commitment to evidence-based practice is so beneficial in education - even at the tertiary level.

  2. Oh dear... to think that so called academics could publish stuff like this!! Poor women teachers can't deal with evidence-based methods of teaching... ha! Great responce to that nonsense.

  3. The AARE blog post seems to me to be an example of a growing, worldwide divide -- between people who look to science for guidance and people who look to some (other) belief system. This is even true in the "hard" sciences, such as physics. For example, see this article about how ideas can be tested in physics: "Questioning Truth, Reality and the Role of Science" [].

    It seems like science skeptics balk at very first element in the scientific method: A clearly described model of how things work, including assumptions (hypotheses) of how elements of the model relate to each other and external variables.

    For example, McKnight and Morgan stress the central nature of "the 'soft' skills of listening, empathising, reflecting and sharing" in education outcomes, but without any clearly defined model that could actually be tested using a fair or arms-length procedure.

  4. Thanks Pam, for defending the importance of evidence-based context, and that teachers need to read and apply research to their context - couldn't agree more! For a while now, many teachers have been overloaded with their professional work - and this isn't changing... I think I'd support providing teachers with opportunities to read research, and also apply that - just not sure where that 25th hour of the day comes from? Great summary and comments, nevertheless! thanks, Gail Brown, Education Consultant and school-based, action researcher

  5. Brilliant rebuttal, @PamelaSnow2

  6. Your responses are on-point. So grateful for your knowledge base and ability to articulate your response. As a parent I need material like yours, because as we know these points of view filter down to classroom teachers where these thoughts get expressed in parent teacher meetings. Love to borrow you to counter other issues we with literacy in Colorado !

  7. I too despair over the publication of such misleading information by these educational academics and applaud Pam for her erudite rebuttal. Well done! As stated, change and the development of critical thinking skills has to begin with teacher training. As part of my own quest to challenge my thinking, I am reading: This book makes similar comments to many of Pam's valuable points above.

  8. Excellent summary and rebuttal Pamela. I just can't believe a piece like the AARE blog post would be published in this day and age, I'm trying to scrape my jaw off the floor!

  9. I've just listened to your excellent interview with Ollie Lovell but there are some things about this post that have bothered me and I don't agree with all that McKnight says either but here's some defence of some of her arguments-

    You rebut-

    Where is the evidence that all that is being recommended is randomised controlled trials (RCTs)?

    But, McKnight does not say all that is being recommended is RCT. She says,

    “We call for an urgent halt to the imposition of ‘evidence-based’ education on Australian teachers, until there a fuller understanding of the benefits and costs of narrow, statistical evidence-based practice.”

    Her article builds on her previous article “Seven reasons to question the hegemony of Visible Learning” where she outlines the domination of Hattie in Australian Education and Victoria in particular.

    There is much evidence for that as outlined by other academics – Eacott (2018) & Ladwig (2019).

    The Victorian Education system is dominated by Hattie, he is written into significant policy documents – “Frameworks to Improves Student Outcomes”. Also, the implementation of the “10 High Impact Teaching strategies” based on Hattie’s work has been forced on over 70,000 teachers.

    The latest Literacy/Numeracy Strategy, costing over $140 million will be based on Hattie.

    You cannot question this research, it usurps teacher professional experience. The Principal’s performance review is based on whether the Hattie strategies are implemented. Also, there is virtually a leading teacher in every school whose job it is, is to see that these are incorporated into lesson plans, lesson observation, interviews for promotion, PD and School plans.

    The AEU union detailed “the restoration of the value of teacher professional experience” as 1 of its 2 MAJOR aims this year. Indicating the dominance of this so-called evidence base over teacher professional experience.

    McKnight’s summary is certainly consistent with all this-

    Your colleague Greg Ashman also comments about the next generation of “Hattie” the EEF-

    "The EEF is an independent charity that operates in England and that was founded with the help of £125 million of UK taxpayer money. Its mission is to help improve educational outcomes, particularly for the children of disadvantaged and low income households. To this end, it publishes a Toolkit that aims to summarise the evidence for different types of educational intervention, as well as conducting its own randomised controlled trials (RCTs) in order to generate new evidence. Latterly, the evidence from its RCTs has become incorporated into the toolkit.
    There are some fairly major implications for Australia. Evidence for Learning (E4L) is a social venture that has licensed the EEF Toolkit for use in Australia and I strongly suspect that this will be a central plank of the ‘Gonski 2.0’ proposals for Australian education when they are finally published. So there is a lot at stake."

    The Australian Productivity Commission recommended in their 2017 report for Education to move toward RCT and has recommended E4L.

  10. Your rebut, It is right that evidence derived from RCTs be questioned, because it is right that evidence from all research be questioned.

    Yes, so you agree with McKnight, the problem is it is not discussed as teachers do not have a forum let alone any power to challenge evidences brought in from administrators or people in power.

    Eacott (2018) details the bigger problem here, that evidence is used to support administrative practice. The administrators have the power to enforce this and teachers don’t have that power.

    Whilst you suggest teachers be better trained in evidence based methods, this still would not solve the power/hierarchy problem.

  11. Thanks for your contribution to the discussion George. I am trying to re-construct a lengthy reply I wrote and then inadvertently deleted - so this one may not be quite as polished as my first attempt.

    I am aware of the debate around Hattie's work, both in relation to his calculation of effect sizes and the sense in some quarters that it is being imposed on teachers. To some extent, I think this is a pendulum problem. For way too long, "choose your own adventure", at either an individual teacher or school level reigned supreme. Parents should not have to enter a lottery when they send their children to school and hope they buy the golden ticket that equates to (for example) effective reading instruction. But that's how things roll in Australia, and parents are left to resource specialist intervention to deal with what are effectively instructional casualties.

    I have blogged previously about the issue of professionalism and autonomy Vs accountability, and you can find that post here:

    I'm curious though about the fact that you haven't commented about (a) the incredibly sexist assumptions underlying the McKnight and Morgan piece (regarding who can "cope" with what types of research methodologies), and (b) the lack of a critical stance on the amount of pseudoscience that regularly and easily finds its way into schools.

  12. Thanks for your reply Pamela, I am a teacher at a local high School, so the only spare time I have is on weekends and school hols. My focus is on Hattie as I believe his work is pseudo-science and I'm doing as much as I can to address that via my blog as I am not allowed to actually present peer reviews of Hattie to staff at my school or any other school!

    The first thing I've tried to do is summarise peer reviews and present that for teachers to read; as you know peer reviews are often behind pay walls (which encourages pseudo-science).

    The 2nd thing is to present the reviews in an understandable fashion, e.g., Bergeron, Pierre-Jérôme (2017). How to engage in pseudoscience with real data: A Criticism of John Hattie's Visible Learning from the perspective of a statistician -

    I've only found 2 peer reviews from Australian Academics, McKnight is one of them. Yet, i've found over 50 reviews from other countries. Given Hattie's dominance and commercial conflicts of interest - why so few Australian Academics? This is really disappointing and has allowed this pseudo-science to go unchallenged.

    Hattie in his book, has "Repeated Reading Programs" ranked #16 and "Phonics" #22. They don't make the ED Dept High Impact Strategies. Why don't you publish some analysis of this?

    I've addressed your first 2 points and with time was going to go through the others. But, I wanted to read The Special Edition of Educational Research and Evaluation first-

    As, a number of academics in this special edition support McKnight's call to action - Wiliam, Cohen, Wrigley & McCusker.

    Regarding Sexism, I don't know enough about gender politics to comment and this is not a major part of McKnight's argument.

    Regarding pseudo-science I thought I did address that (so does McKnight's previous paper), in that the problem currently in Victoria is that Hattie's pseudo-science is promoted by the Ed Dept and all its hierarchical structures and teachers no matter how knowledgeable or competent are able to challenge that.

  13. Hi George
    I think there's an important distinction between flawed science and pseudo-science. I know Hattie's work has been the subject of criticism (and no-one's work is above this), but that does not make it *pseudo* science. Pseudoscience refers to ideas that have no scientific basis at all (learning styles, Brain Gym, coloured lenses etc). I think those who criticise Hattie's work need to be careful not to throw the proverbial baby out with the bath-water, as there may well be substance in some of his claims - that's where the skills of critical appraisal come in. Why don't I blog about Hattie? There's only one of me, and I pick my battles! Cheers, Pam.

  14. Of course everyone has the right to choose their battles. But, given this is the unquestionable evidence base in EVERY Govt school (70% of schools, not just some quarters) I'm very disappointed few Australian academics have chosen not to fight this or even look into it in any detail.

    There is enough independent review to indicated that Hattie's work is pseudo-science. Bergeron & Lysanne (2017)-"Pseudo-science"; Eacott (2018) "cult"; Wrigley (2015)-"bullying by Numbers";Berk (2011)-"statistical malpractice"; Slavin (2018) -"Hattie is wrong"; Larsen (2019) - "blindness"; Nilholm (2017)-"fantasy"; O'Neill, (2012)-"Material Fallacies"; Wiliam (2019)-"has absolutely no role in educational policy making."

    Anyway, the debate wont change here until Australian Academics look into this work in detail.

  15. Hi Pamela,

    Love your work. Great review of "The problem with using scientific evidence in education" and enjoyed your rebuttal. Seems to me that the two authors have ignored some of the related issues which are perhaps more important. Namely, the lack of repeatable research in teaching and learning in education, the lack of "teaching" experience in pre-service teacher training courses and the poor state of the historiography of teaching and learning pedagogy.

    These three areas continue to bedevil attempts to provide coherent advice and support to teachers, including at the tertiary level. The current debate about the number of overseas students in our universities has declined into a discussion about the abuse of group learning. Overseas research absolutely laudes group learning as being critical, in the sciences especially. In fact Abercrombie in her 1957 report on "Group Work" (Talking to learn: improving teaching and learning in small groups) identified it as a key element in doctor training. Ironically Hattie's section on "co-operative learning" as a low cost, immediate means of turning around student performance is let down by the lack of content in his recommendations to teachers. That is another story though about the down-side of meta-analysis.

    Having just had my first human ethics research project application approved, for an English as an Additional Language for tertiary students I must say I am and was impressed by the rigor it applied to my project. Unfortunately I have had to discard most of my literature review (based around my initial enthusiasm for my topic) and start again with one focused on the actual research around my topic and not just the major scholarship as these have turned out to be quite different!! Now I appreciate why so few Arts postgrads have projects which involve "humans."

    Again, thanks for taking the trouble to go through the article. Arpad

  16. The subsistence of open access to scientific journal articles can now go far beyond any blockade providing the scientific society with the opportunity to get better citation and readability. At a number of freely available journals, only a few chosen quality papers are published every year. This is to guarantee the greatest visibility with a high point of indexing. learning development, expression, artifact, with social interaction.