Saturday, 2 May 2015

Why not everyone is enthusiastic about the Arrowsmith Program

Recent media reports such as this one in The Age have again drawn attention to the Canadian-based Arrowsmith Program for addressing the needs of struggling learners.

What could possibly be wrong with a program that

" founded on neuroscience research and over 30 years of experience demonstrating that it is possible for students to strengthen the weak cognitive capacities underlying their learning dysfunctions through a program of specific cognitive exercises"?

What parent or teacher would not want children to be able to strengthen "weak cognitive capacities", particularly if this occurs via a program of "specific cognitive exercises"? This sounds a bit like the process undertaken in pathology labs, where microbes in a particular bacterial infection are isolated under a microscope, so that antibiotic sensitivities can be ascertained, and targeted drug therapy can be prescribed.

If only that was how learning worked!

The Arrowsmith program makes liberal use of the kinds of words that are designed to hook parents and teachers and get them believing that this is a rigorous, scientifically-based intervention, offering something unique over and above what can be offered in a well-organised, evidence-based classroom curriculum.

Have a look at the Arrowsmith website, and put a dollar on the table every time one of the following terms appears:

  • cognitive
  • neuro
  • brain-based
  • neuronal
  • synaptic
  • neural sciences
  • cognitive-curricular research
  • brain imaging
  • targeted cognitive exercises
  • neuroplasticity

You'll certainly be a lot poorer at the end of this exercise! But perhaps you're not yet convinced of a need to activate your inner sceptic? Well have a listen to the videoclip interview with Dr. Lara Boyd, discussing neuroimaging studies of children and controls undergoing the Arrowsmith program (see homepage link above). Dr Boyd describes what sounds like a rigorous scientific study into the "changed brains" of children who have undergone Arrowsmith training, compared to those who have not (the control group).

What's the problem here?

The problem is that the good folk behind the Arrowsmith program have reduced an incredibly complex (and to a large extent, poorly understood) phenomenon (human cognition and its representation in the cerebral cortex) to a highly over-simplified narrative that mums and dads can "understand". It's a narrative that has strong face appeal, and encourages those without specialist knowledge or training to make a link between supposedly "known" activity in the brain and certain learning exercises.

If you were a parent of a child with learning difficulties, what would best motivate you to enrol your child in an expensive 3-4 year intervention (yes, that's right, 3-4 years)? Would it be the expectation that s/he would be performing at expected levels across the curriculum, or the reassurance (irrespective of academic outcomes), that s/he now has a thicker cerebral cortex and/or more myelinated neural pathways? I know what I'd be wanting as a parent.

To the best of my knowledge, the "evidence" in support of Arrowsmith comprises in-house research reports, conference poster-presentations, and satisfied client testimonials. Studies "in process" (as at 2014) include Effects of the Arrowsmith Program on Academic Performance: A Pilot Study - University of Calgary. One would have to wonder why a program that has existed for some 30 years, is only now at the point of collecting pilot data - yet it has been charging parents thousands of dollars over three decades, in the absence of robust, independent empirical data.

It is also notable that the first-listed publication at the link above is described as a "case study". Case studies are useful tools in elucidating the impact of a condition and also the response of a small number of individuals  ("cases") to an intervention.  Case studies are often an attractive marketing tool, as they are typically devoid of pesky statistics and other "dense" concepts that reduce their accessibility to lay readers.

In this case, clicking on the case study link reveals a PhD dissertation, which looked at 5 students, and concluded:

"Four of the five students experienced large and significant increases in cognitive, academic, emotional, and/or interpersonal functioning following their participation in the LDAS Arrowsmith program. One of the five students had much smaller gains in cognitive and academic functioning and experienced difficulties with emotional and interpersonal functioning following participation in the program".

So notwithstanding the very small sample size (not in itself inappropriate in case-study methodology), we have a scenario in which 20% of the sample not only failed to derive a benefit, but may have been adversely affected.

It should be stressed that case studies, though accepted in the scientific and academic communities as a form of evidence, are regarded as "weak" alongside other readily available, more robust methodologies that enable to us to accept or reject an intervention with far greater confidence, e.g., cross-sectional studies, randomized controlled trials, systematic reviews of well-controlled studies, etc. (See
this link for a quick summary of the evidence hierarchy). Case studies are a useful starting point, and need to be followed by larger, more rigorous, and independent evaluations. Unfortunately, it does not appear that this has occurred in the case of the Arrowsmith program.

What needs to happen?

In order for academics in education, developmental (neuro)psychology, speech-language pathology etc to be able to give unbiased, accurate advice to parents, schools, and policy-makers, peer-reviewed research is needed that controls for the fact that children in programs such as Arrowsmith receive a great deal of intensive, 1:1 time. That in itself should result in improved knowledge and skills.  Well-conducted studies that control for variables such as socio-economic status, comorbidities, and prior instructional environment are needed. It is important that gains made are tracked over time to see if they are maintained, or are an initial halo-effect. Bear in mind that when a child is significantly behind academically, interventions need to accelerate their progress relative to their typically-developing peers. Otherwise they will never catch up, let alone maintain their gains in the face of an ever-more demanding academic curriculum. We also need studies in which those carrying out outcome assessments are "blind" to which study arm a child was in. Not ensuring this opens the findings up to a range of overt and covert biases.

In the absence of such empirical data that people such as myself can draw on, it is disappointing and worrying that two Victorian education sectors have succumbed to (understandable but regrettable) consumer pressure to countenance an introduction of the Arrowsmith Program in a small number of schools (at this stage in only one of the sectors as far as I am aware).

Consumers should expect providers to go to the market with already-tested, replicated, and high-level evidence before they ask people to sign-up to an expensive intervention. We don't expect cancer patients to organise their own randomised controlled trials of new treatments, so why should it be left to schools (who don't typically have the appropriate expertise on staff) to stumble around and try to work out whether an education intervention is an appropriate investment above and beyond what they are already doing
(or could be doing)?

So - how will it be determined that the adoption of Arrowsmith mentioned above (and others that are popping up here and there) has been successful (or not)? What pre-determined criteria will be applied? Will there be any evaluations by independent and appropriately qualified researchers, or is this yet another education-intervention cul-de-sac?

Postscript, September  4, 2015: Interested readers should also check this blog post about the Arrowsmith Program by Professor Dorothy Bishop (University of Oxford).

(c) Pamela Snow 2015


  1. All of this is fine and well but the fact of the matter is, the program works. I hope the research catches up soon so this program will be widely available.

    1. If something "works", that's a position that should be supported by independent, empirically verifiable evidence. People should not be profiting from parental anxiety and the capacity of some parents to pay enormous amounts of money on the strength of biased testimonials. We would not accept this in medicine and nor should we in education.

    2. Parachutes work. Show me a randomised double-blinded research project that supports this.

    3. That's an anti-intellectual, straw man unfortunately "Unknown". Parachutes "work" until someone proposes something better, which will no doubt be subjected to rigorous scientific scrutiny before it is made widely available and promoted for mass use - as it should be. This is what I advocate in relation to the welfare of struggling students too.

    4. Okay. It works. That's all? What else can you say about it? How come only the maker of the Arrowsmith program talks, and nobody else is?

  2. PS Would you please update the link to Prof. D Bishop's post, I can't find it at the site you have given. Thank you.

    1. So sorry about my very tardy reply - but I have updated the link. You might like to have a look at some of Prof Bishop's other posts while you are on her site.

  3. having attended this school, and I am stretching that word to its limits for the sake of inclusion, I must say that it is a year I will never get back.

    1. Interesting comment - would you care to elaborate? I am sure readers would like to know more.

    2. I would like to know more.

  4. As an adult I experienced no gains after one year of rigorous work at AS. I wonder how many other mature students share my sentiments? I envy those that benefited from the program. I wonder ,though, if they are deluding themselves??

    1. I am an adult who has been contemplating attending the six week summer program that they offer. It will be a financial burden and I really don't have the time to take off work for 6 weeks, however I'm desperately hopeful, but wise enough to know that because I'm an adult I may not benefit as much as younger people with more malleable brains. I'm trying to gather as much into as possible to make an informed choice before I make the leap. Might you be willing to discuss over email? I would so appreciate any insights you may have. Thank you

    2. Yes of course - happy to discuss in a general sense, but of course I can't provide individualized advice. My email is
      Can you p[lease put "Arrowsmith Program" in the subject line so I don't miss it?
      Thanks :-)

    3. Did you see it work for others?

  5. Interesting comment - thank you. It is always helpful and illuminating to hear from people you have had direct experience of programs such as these, but without the gains described in the promotional hype. One of the problems with over-reliance on testimonials is that the voices of dissatisfied consumers are never included.
    Thanks for joining in on the discussion.

  6. I am "anonymous" for the sake of my son who attended an Arrowsmith School and wishes to remain anonymous. However I would like to ask where the randomised, peer reviewed studies are that prove that children should be grouped by age and sit all day long in the name of "education"? And yet it is generally accepted as the way to educate children. My son has benefitted hugely from the program and has gone from needing significant support in terms of extra tutoring and speech and language therapy - none of which made any difference to the underlying learning difficulties, to needing no support at all. He had a huge spread in his psych-ed assessment and every professional that dealt with him said he had a very unusual profile in having very "high highs" and very "low lows". Now in his recent Allwell testing, he achieved high average or above average across every test and in one of the tests, he had previously been in the 5th-7th percentile (on more than one occasion before you suggest it was testing error). A few months in (after leaving the program), he is managing high school with no support or interventions whatsoever and has already closed academic gaps, achieving above average for his age and grade. Sending him was not without its sacrifices and challenges - not least because he missed a big chunk of regular school during that time but I am writing as I feel saddened that more children will not have the opportunities that my son had - especially more so with the naysayers in the professional world saying it can't work. It does. Maybe not for every child - and perhaps that's where the focus on the studies should go instead of wasting time and money investigating whether neuroplasticity works or not - when there is already a lot of evidence to suggest that it does.

  7. Hello and thank you for contributing to the discussion. First of all, I am sorry that you and your family have had a difficult journey and of course I am pleased that you seem to have found something that appears to work - that is after all, every parent's aim. I agree with you that there is a great deal (too much) in fact that goes on in education that is not evidence-based, though it could be argued that there are some broad developmental principles that underpin the way schools are organised, and of course many schools deal flexibly with this by having multi-age/composite classsrooms for at least part of the day.
    My main point in this post is that the Arrowsmith Program has been around for more than THREE DECADES and yet has not played ball with the scientific method, whilst charging parents thousands of dollars. It may well do some (or even a great deal of) good for some children - but wouldn't it be wonderful if we could find out how many, which ones, under what circumstances, at what treatment intensity, and maybe even why? That is what scientific research is all about - serving the greater good, and not just a fortunate few. Those children who are not going to benefit from this approach should be spending their time and family resources elsewhere. As with all testimonial-based approaches, the voices of the unsuccessful are rarely, if ever heard.

  8. The "regular" education system is not designed for the greater good. There are very few studies done supporting it, yet we accept it as better than the Arrowsmith program simply because it's free ?

    Maybe if we supported schools like Arrowsmith there would be subsidy provided for families that feel the need to have their child learn in a tailored environment.

  9. Thanks for your comment "Unknown". I agree that mainstream education needs to be more evidence-informed. If you read other posts on my blog, you will see many examples to that effect. However lack of research evidence in the mainstream does not justify or excuse the entry of a new player that lacks an evidence base AND charges thousands of dollars AND requires parents to sign confidentiality agreements. In my opinion, such a combination is simply unethical.

  10. My 9 yo son just completed his first year of AS. So far, I have seen so many improvements that we to signed him up for a second year. He himself has observed that he is capable of understanding things differently than he used to. I have noticed that he is more confident, more self-motivated, and he is able to perceive things more clearly. And yes, his academics have improved. I appreciate that you would like more peer reviewed scientific evidence - I would too. Might I point out that your article was heavily based on your opinion, and not based on science, which is somewhat ironic to me.
    As someone who works in a scientific field, I understand that it costs a lot of money to thoroughly research a topic. Usually, there has to be a financial interest in funding the research. Perhaps that will come in time, but until then, as far as I can tell you don't have any peer-reviewed science that can disprove the methods of the program. So that begs me to ask: Why such a critical article about something that is helping children, such as my son? Have you considered the harm you might be causing the families who choose not to enroll their children based on this article? Parents who are going through this do not need a bunch of pseudo-facts, such as this article, to come up as the first item in their Google search. This article raises lots of questions, but does not provide any real answers. In my experience there are two types of people in the world: people who see a problem and fix it, and people who see a problem and cause more problems. Which one is is Barbara Arrowsmith Young? Which one are you? Based on the condescending tone of this article, I think we have our answer.
    I for one encourage anyone considering Arrowsmith to give it a shot for a year. In our experience it has been the best thing we could have done for our son.

    1. Could you be more specific in how it worked?

  11. Dear Anonymous
    Thanks for contributing to the discussion. I am pleased that you have found something that seems to be producing gains for your son. I would encourage you though, to read over my responses to some similar posts above. If the AP is beneficial to some children, as evidenced by robust (ideally independent) research, then I will be happy to share that information via my blog. You ask me what evidence I have that it is not effective, but that is the precisely the problem - there is no scientific evidence about this program one way or the other.

    You are fortunate to be in a position to pay the not inconsiderable sums of money to engage with this program, but what of the parents who are not able to bankroll this commitment? They are left feeling guilty and wondering, but in reality the program may not be a panacea at all. I reiterate that this program has been around for over 30 years, but has not played ball with the scientific method.

    I don't think it's ethical to charge money and play on parents' hopes, for something that has plenty of time to be validated or dismissed. Confidentiality agreements prevent us from hearing the "testimonials" from parents whose children have not benefitted from the program.

    I could not agree more that we need better research funding in this space, but I don't see how that legitimises three decades of a successful business model without an empirical basis.

  12. I believe that Arrowsmith has survived over 30+ years with the pioneer's hard work , struggle, courage and believe. I wish Arrowsmith to be successful in business, then more kids can be supported through and beneficial. I decided to put my kid in to count on.

  13. As someone who is considering putting their child in the Arrowsmith Program, I would so like to see studies to either support or not support this program. I have scoured the internet, and find nothing regarding studies or negative reviews. And even most positive reviews are so general in nature. I've seen parents boast about their child's ability to read a clock. That's not enough for me considering they spend hours doing that exact thing. I've read about increased confidence many times. My daughter went from mainstream classroom and then to special day class and her confidence and happiness were drastically improved because she was able to work at her own speed, and was not the lost kid who didn't understand anything anymore. So improved confidence could just mean not being in a mainstream classroom and being lost anymore. And I hear parents say it's helped dramatically, but no detailed information. How has it helped? Will my daughter improve her speech after years of speech pathology? Will she be understood? Will she be able to do math afterwards? I don't care about testing. I want to know how it's changed things in the real world. Do they fit in with the mainstream kids now? Or are they still out of the loop? Sometimes I see two pieces of writing where one is improved, but after two to three years of schooling, wouldn't it be improved anyway? So my dilemma is do I take her out of a school she loves with art and music and some friends, a small classroom setting where she's doing better but far behind her peers or forego all of that and spend the money and see what happens? It's a question that keeps me up nights.