Let’s all throw out the scientific method. It’s not perfect and it's too hard. AKA The soft bigotry of gendering research methodologies.

Last week, a piece entitled The problem with using scientific evidence in education (why teachers should stop trying to be more like doctors) appeared on the AARE Blog. This was co-authored by an education academic, Dr Lucinda McKnight (Deakin University) and a medical education academic Dr Andy Morgan (Monash University). The authors purport to mount an argument as to why the notion of evidence-based practice should be resisted in education. 

I believe the article is deeply flawed at a number of levels, and have provided a detailed response to it here.

By way of background, I worked in medical education for ten years, and have also taught teachers at postgraduate level, as well as having taught across some ten allied health professions.

                                                      

McKnight & Morgan   **************************** For teachers to be like doctors, and base practice on more “scientific” research, might seem like a good idea. But medical doctors are already questioning the narrow reliance in medicine on randomised controlled trials that Australia seems intent on implementing in education.	My response ********************************************** Where is the evidence that all that is being recommended is randomised controlled trials (RCTs)? It is right that evidence derived from RCTs be questioned, because it is right that evidence from all research be questioned. In medicine, there is a sound understanding of an efficacy trial Vs an effectiveness trial. This difference should be considered and discussed in education as well.
In education, though, students are very different from each other.	They are no more different, nor similar to each other than are patients. Doctors, like teachers, rely on pattern recognition to form and test hypotheses. They could not do their jobs if this wasn’t the case.
Unlike those administering placebos and real drugs in a medical trial, teachers know if they are delivering an intervention. Students know they are getting one thing or another. The person assessing the situation knows an intervention has taken place.	Yes, but students do not necessarily know which teachers have been exposed to an intervention, e.g. a series of professional learning seminars. And researchers with overall responsibility for a trial can easily be blinded to allocation group - I speak from personal experience on this. This statement betrays an unfortunate lack of understanding of the nature of RCTs in education.
Constructing a reliable educational randomised controlled trial is highly problematic and open to bias.	Yes, doing rigorous research is challenging. Yes, all research is open to bias. Skilled researchers make it their business to recognise, and minimise sources of bias, and to report their findings with caution.
Before Australia decides teachers need to be like doctors, we want to tell you what is happening and give you some reasons why evidence based medicine itself is said to be in crisis.	The fact that researchers are questioning an approach does not mean it is being thrown out in its entirety. That’s exactly the kind of thinking that has plagued education for decades, as shown by the tendency to adopt fads and fashions, with zero research behind them, let alone any that has any supporting evidence. I have written about this here with my colleague, Dr Caroline Bowen.
Randomised controlled trials are just one kind of evidence	And this is news because?? Of course RCTs are only one kind of evidence. That’s like saying Toyota is only one make of car.
Medicine now recognises a much broader evidence base than just randomised controlled trials.	This is also not news. Medicine has always recognised a range of study designs. What seems to be overlooked in educational discourse, however is the notion of levels of evidence. In health, it is recognised that different study designs have different degrees of strength in establishing the efficacy or effectiveness of an approach.
Other kinds of medical evidence include: practical “on-the-job” expertise; professional knowledge; insights provided by other research such as case studies; intuition; wisdom gained from listening to patient histories and discussions with patients that allow for shared decision-making or negotiation.	These are obviously important in all fields. They just don’t sit at the top of the hierarchy as to what can be established, replicated, and/or refuted, using the scientific method.
Privileging randomised controlled trials allows them to become sticks that beat practitioners into uniformity of practice, no matter what their patients want or need. Such practitioners become “cookbook” doctors or, in education, potentially, “cookbook” teachers. The best and most recent forms of evidence based medicine value a broad range of evidence and do not create hierarchies of evidence. Education policy needs to consider this carefully and treat all forms of evidence equally.	All forms of evidence are not “equal”. This does not mean that they should not all be considered, including expert opinion, but human beings are prone to all kinds of cognitive bias.  Sometimes our intuitions tell us that something “should” work, or even that it “seems” to work, but the scientific evidence counters our intuitions. This is the subject of Andrew Leigh’s book Randomistas.
Teaching is a feminised profession, with a much lower status than medicine. It is easy for science to exert a masculinist authority over teachers, who are required to be ever more scientific to seem professional.  They are called on to be phallic teachers, using data, tools, tests, rubrics, standards, benchmarks, probes and scientific trials, rather than “soft” skills of listening, empathising, reflecting and sharing.	Where to start with this one? Let me point out a few facts:         Medicine is rapidly becoming feminised, with more females enrolled to study medicine in many Australian universities than males. Does this mean that it will now abandon centuries of commitment to the scientific method? I certainly hope not. Dark Ages, here we come, if it does.              There’s an assumption here that the scientific method would be an imposition on those poor feeble women in teaching, who would not be able to cope with the rigours of its analytic tools. How insulting.             There is no connection between genitalia and the tools of scientific inquiry. This is just silly. What about all those women who conduct (and use) quantitative education research? Where does this leave them?              So-called “soft skills” are just as important in medicine as knowledge of human biosciences, pharmacology, and so on. No-one is suggesting otherwise. Why can’t doctors and teachers be content experts AND competent consumers of new research? When did it become either-or?
A Western scientific evidence-base for practice similarly does not value Indigenous knowledges or philosophies of learning. Externally mandated guidelines also negate the concepts of student voice and negotiated curriculum.	Education and medicine both need to show a deep respect for and understanding of indigenous knowledge and practices. That does not mean that the Aboriginal man presenting to the Emergency Department with chest pain automatically wants to receive a different type of care from his non-indigenous counterpart. If the latter receives an immediate ECG and blood tests, then so should the Aboriginal patient. There is a place for the student/patient voice and there is a place for professionals to do what professionals are trained and paid to do.
While confident doctors know the randomised controlled trial-based statistics and effect sizes need to be read with scepticism, this is not so easy for many teachers. If randomised controlled trial-based guidelines are to rule teaching, teachers will also potentially be monitored for compliance with guidelines they may not fully understand or accept, and which may potentially harm their students.	If teachers are not confident in interpreting research studies (and I agree they are not), then education faculties need to step up and teach them how to be critical consumers of research – quantitative, qualitative, and mixed methods. All professionals are monitored for compliance – that’s part of what being a professional means. It is a highly constrained form of public accountability. I have blogged about this previously.
Evidence based medicine is about populations, not people	The fallacy here of course, is that populations are not made up of people. Evidence-based medicine is about using robust study designs to control a range of sources of error so that appropriate conclusions are drawn. It is then up to the individual practitioner to consider the findings in the course of their clinical decision making on a case-by-case basis. As noted below, this entails consideration of evidence, patient values, and clinical resources. But the evidence part is non-negotiable.
While medical randomised controlled trials save lives by demonstrating the broad effects of interventions, they make individuals and their needs harder to perceive and respect.  Randomised controlled trial-based guidelines can mean that diverse people are forced to conform to simplistic ideals. Rather than starting with the patient, the doctor starts with the rule. Is this what we want for teaching?.	Well at least we have some acknowledgement here that RCTs can help to save lives! It is not the role of an RCT to bring individuals into sharp focus. We have many other study designs that do that much better, and they are considered alongside the findings of RCTs in the development of treatment guidelines.
When medical guidelines are applied in rigid ways, patients can be harmed	When any guidelines are applied in rigid ways people can be harmed. There is nothing illuminating about this statement. Anyone who is familiar with the pioneering evidence-based medicine work of Dr David Sackett and his colleagues will know that this model emphasises empirical research + patient values + clinical resources. It is not, and never has been, about research evidence alone. Right from the start in medicine, it was emphasised that evidence-based medicine is not a cook-book approach. This is just a straw man. Interestingly, evidence-based practice was initially seen as an unnecessary imposition on medicine in its early days. Now it underpins the way we educate all health professionals, and the community is the beneficiary – both as patients and as tax-payers.
Trials cannot be done on every single kind of person and so inevitably, many individuals are forced to have treatments that will not benefit them at all, or that are at odds with their wishes and beliefs.	This is another nonsensical truism. No, we cannot include all kinds of people on planet earth in trials (clinical or educational). The whole purpose of research is that we sample from populations, in an effort to generalise back to the population as closely as possible. Welcome to Research Methods 101. Just because rigorous methodologies cannot answer every question, for every patient, every time, does not mean they are not the best horse in the race to back.
Educators need to ensure that teachers, not bureaucrats or researchers, remain the authority in their classrooms.	Well a good way to make a start on this would be for education faculties to equip pre-service teachers with scientifically-derived knowledge and skills on (for example) the teaching of literacy and numeracy, as well as the ability to read and critique new research and make decisions about how this should inform practice. Teachers cannot speak with authority if they do not know the research behind an approach and the extent to which this is contested. This is why, for example, medical students are taught that prescribing antibiotics for children with middle ear infections is controversial. They know the ground will shift under them over time, as the science changes, and are primed to watch for new evidence as it arises, and adjust their practice accordingly. This is called accountability.
Scientific evidence gives rise to gurus	A more critical and discerning teaching workforce would counter this in a flash – in the same way that it does in medicine. Gurus flourish where audiences can be easily wooed and charmed by pretty graphs and impressive looking numbers.
While medical-style guidelines may seem to have come from God, such guidelines, even in medicine are often multiple and contradictory. The “cookbook” teacher will always be chasing the latest guideline, disempowered by top-down interference in the classroom.	Yes, this is the nature of scientific evidence. It changes, and is sometimes contradictory. Rather than “chasing” the latest guideline, professionals need to avail themselves of new evidence and work out how it should influence their practice.  This is called accountability.
In medicine, over five years, fifty percent of guideline recommendations are overturned by new evidence. A comparable situation in education would create unimaginable turmoil for teachers.	The paper linked to here states “This investigation sheds light on low-value practices and patterns of medical research”. Wouldn’t this be a good thing in education too? That way, we might never go down the Brain Gym, coloured lenses, learning styles, multiple intelligences, left brain-right brain, brain-based learning (etc) time wasting and expensive rabbit holes that education is so fond of. One of the challenges of living in a knowledge economy is that information changes. We all have an obligation to keep up as much as possible. Choosing your own adventure, whether as a doctor or a teacher, is not acceptable to the community.
Evidence-based practice risks conflicts of interest	Then let’s be careful not to throw the baby out with the bath water. The more discerning and informed teachers (and doctors) are, the less prone they will be to commercial interests. This is part of the imperfect world in which we live and is not a reason to abandon the scientific method. There are plenty of commercial interests at work in classrooms around the world today, regardless of the level of evidence underpinning the teaching that is occurring.
Randomised controlled trials in medicine routinely produce outcomes that are to the benefit of industry. Only certain trials get funded. Much unfavourable research is never published. Drug and medical companies set agendas rather than responding to patient needs, in what has been described as a guideline “factory”.	These are all legitimate concerns about health research that need to be managed. They are not reasons to abandon the scientific method. See babies and bath water, above.
Do we want what happens in classrooms to be dictated by profit driven companies, or student-centred teachers?	As noted above, there are plenty of profit-making companies doing their thing in classrooms around the world right now, cashing in on the fact that teachers are a soft target for approaches with a slick marketing spin, and a few researchy-sounding words in the glossy brochure and on the equally glossy box. The purpose of having a more research-informed teaching workforce is being able to head off snake-oil sales people at the school gate.
We call for an urgent halt to the imposition of ‘evidence-based’ education on Australian teachers, until there a fuller understanding of the benefits and costs of narrow, statistical evidence-based practice. In particular, education needs protection from the likely exploitation of evidence-based guidelines by industries with vested interests.	Ironically, such a halt wouldn’t cause a great deal of disruption, given the limited extent to which evidence-based practice has genuinely found its way into education discourse.
Rather than removing teacher agency and enforcing subordination to gurus and data-based cults, education needs to embrace a wide range of evidence and reinstate the teacher as the expert who decides whether or not a guideline applies to each student.	Perhaps we need to consider the possibility of enhanced teacher agency, in a world where teachers are knowledgeable and confident consumers of new research, by virtue of their grasp of research methodologies and critical appraisal skills? We can’t consider “a wide range of evidence” while disregarding evidence from RCTs, and not understanding the notion of levels of evidence means that equal weight is inappropriately assigned to a single case study and a meta-analysis of several RCTs. They all contribute to the understanding of an issue, but not necessarily equally, on a study-by-study basis. One of the key differences between medicine and education, is that doctors frequently need to gain informed consent for their actions. In education, however, consent is implied. Students cannot give or withhold their consent for a particular instructional approach. It just comes their way, like it or not. This only serves to increase, not decrease the ethical burden on teachers to teach in ways that are supported by strong empirical evidence.
Pretending teachers are doctors, without acknowledging the risks and costs of this, leaves students consigned to boring, standardised and ineffective cookbook teaching. Do we want teachers to start with a recipe, or the person in front of them?	No-one is pretending teachers are doctors. But if they want to be afforded at least some professional autonomy, then they have to accept professional accountability, just like other professions do. We all need to acknowledge, however, that no profession is completely autonomous, least of all medicine. We all need to be accountable to our “consumers”, our employers, our professional bodies, and the community. No, we don’t want recipes, but nor do we want random chaos, and the wild west of everyone choosing their own adventure, either. The Age of Enlightenment created some enduring legacies that we would all do well to hang on to. As I have stated previously: Education and medicine, for example, have a great deal in common; they both concern people, interactions between people, complex co-occurrences, and hard-to-control (actually impossible to control) variables, such as race, gender, ethnicity, religion, intelligence, empathy, sometimes unpredictable and seemingly inexplicable behaviour, resource limitations, and the need to establish trust and rapport. Most importantly, both have to deal with uncertainty, coupled with a weight of responsibility and accountability to communities, peers, and policy-makers for outcomes.  Perhaps some in education think that they can dispense with research rigour because children can't be harmed by sub-optimal or mis-directed classroom practices.  They are wrong.

(C) Pamela Snow (2019)