When Teaching Theory Fails: The Effect Size Reality Check
Brief Description
A real-world action research project testing whether graphic organizers with a theoretical effect size of 1.24 actually improve learning outcomes in business administration classes, revealing the gap between educational theory and classroom practice.
Summary
What happens when a teaching method with a massive 1.24 effect size - theoretically capable of boosting student achievement by two full grades - gets tested in a real classroom? The answer might surprise you.
This episode follows teacher Sharon Preston-High as she puts educational theory to the ultimate test. Armed with research from John Hattie showing graphic organizers should dramatically improve learning, she designed an 8-week experiment across four business administration courses with 32 students.
The setup was perfect: two groups used her standard discussion-based methods, while two groups received lessons enhanced with graphic organizers. Same content, same assessments, fair comparison. According to the research, the graphic organizer groups should have significantly outperformed the discussion groups.
But here's the plot twist: they didn't. Despite the impressive theoretical backing, there was virtually no difference in outcomes between the two approaches. In fact, students seemed slightly more engaged during discussions, and some needed extra support with the visual tools.
This isn't a story about failure - it's about the crucial gap between educational theory and classroom reality. Sharon's honest reflection reveals why blended approaches matter more than single "magic bullet" methods, and how action research empowers teachers to build evidence-based practice tailored to their specific students.
Perfect for educators questioning whether research always translates to practice, or anyone curious about how we can better bridge the gap between what works in theory and what works in the real world.
-
[Speaker 2]
Welcome curious minds to another Deep Dive.
[Speaker 1]
Great to be here.
[Speaker 2]
Today we're plunging into the fascinating world of education, specifically how we truly learn and hang on to information.
[Speaker 1]
And what happens when classroom theory gets tested in the real world.
[Speaker 2]
Exactly. Our source for this Deep Dive is an action research project. It's called Effect Sizes Using Graphic Organisers and it's by Sharon Preston-High from 2018.
[Speaker 1]
Right, and this isn't just dry theory, it's a teacher, Sharon, doing a hands-on experiment to see what really makes a difference for her students.
[Speaker 2]
Yeah, that's what's cool about it. So our mission today is to sort of unpack the key insights from her work. We'll look at this idea of action research, what is that, and then effect sizes.
Sounds technical, but what does it mean for learning?
[Speaker 1]
And then we follow Sharon's experiment using graphic organisers in her classroom.
[Speaker 2]
You got it. Get ready to maybe challenge some assumptions and hopefully walk away with some practical thoughts about how you can learn better.
[Speaker 1]
Sounds good.
[Speaker 2]
Okay, let's unpack this. The foundation of Sharon's project, and while our chat today, is action research. Right.
It does sound a bit academic, maybe, but really it seems to be about making deliberate improvements in what you do. So for you listening, whether you're a teacher, a student, or just somebody who likes to get better at things, why should this matter?
[Speaker 1]
Well, at its core, action research is really about practitioners, like Sharon, the teacher in this case, pinpointing areas they want to improve in their own work.
[Speaker 2]
Okay.
[Speaker 1]
And then, you know, systematically experimenting to get better. Think of it like trying a new recipe. You figure out what needs work, plan a change, try it, see how it goes.
[Speaker 2]
Right.
[Speaker 1]
There's a researcher, Wilson, who in 2015 described it as this kind of training cycle. Identify, plan, implement, evaluate. It's all about fine-tuning your practise.
[Speaker 2]
So it's disciplined. Yeah. But it's happening right there in your own environment, your classroom, your office, whatever.
[Speaker 1]
Exactly.
[Speaker 2]
And it's meant to directly inform what you do next. And this isn't like a brand new idea either, is it?
[Speaker 1]
No, not at all. It actually goes way back to Kurt Lewin, a social psychologist in the 1940s.
[Speaker 2]
Wow, the 40s.
[Speaker 1]
Yeah. He coined the term and stressed this cyclical process, planning, acting, observing, reflecting, plan, do, check, think.
[Speaker 2]
Okay. And you mentioned another model.
[Speaker 1]
Yeah. The Kolb cycle, sometimes called the experiential learning cycle. It's maybe a simpler take.
Four stages. Concrete experience, that's the actual doing part.
[Speaker 2]
Right.
[Speaker 1]
Then observation and reflection, thinking about what just happened.
[Speaker 2]
Okay.
[Speaker 1]
Then abstract conceptualisation planning based on that thinking, like, what will I do differently? Got it. And finally, active experimentation, doing it again, but with the changes.
It's this loop.
[Speaker 2]
And the implication is, if you skip the reflection part.
[Speaker 1]
You might just keep making the same mistakes without realising it. You don't learn from the experience.
[Speaker 2]
That makes total sense. We've all done that, right? Just banging your head against the same wall.
[Speaker 1]
Absolutely.
[Speaker 2]
So from a teacher's point of view, like Sharon's, why go to all this trouble? It sounds like extra work.
[Speaker 1]
Well, it lets them really dig into educational theories, but then step back from the day to day rush. They get to reflect, maybe research some new creative ideas. And crucially, they build their own evidence for what actually works.
[Speaker 2]
For their students in their context.
[Speaker 1]
Exactly. Because as Sharon points out in her paper, teachers often work on assumptions. And sometimes, you know, big educational changes nationally might be driven more by politics or ideology than by solid evidence from the classroom floor.
Precisely. Action research is a way for teachers to gather that ground level evidence themselves.
[Speaker 2]
Okay, this is where, for me, it gets really interesting. Sharon's project was specifically sparked by training she got on something called effect sizes.
[Speaker 1]
Ah, yes, effect sizes.
[Speaker 2]
Sounds a bit statsy, maybe, but it seems absolutely key to understanding what actually impacts learning. What are we talking about here?
[Speaker 1]
It's actually a pretty simple concept, fundamentally. An effect size just quantifies the difference between two groups.
[Speaker 2]
Like group A got method X, group B got method Y.
[Speaker 1]
Exactly. And it helps answer that huge question, what has the greatest influence on student learning? Professor John Hattie is probably the most famous name associated with this.
[Speaker 2]
Right, I've heard of him.
[Speaker 1]
He spent his career synthesising thousands upon thousands of studies, meta-analyses, looking at all sorts of teaching methods and factors.
[Speaker 2]
So Hattie basically crunched a mountain of data to find out what really moves the needle in education.
[Speaker 1]
That's the idea.
[Speaker 2]
And what's considered a, well, a significant effect size. What numbers should we look for?
[Speaker 1]
Generally, an effect size of 0.4 is seen as average, sort of the typical impact you'd expect from a year of schooling.
[Speaker 2]
Okay, 0.4 is the benchmark.
[Speaker 1]
Yeah. Anything above that is considered above average, more effective than typical practise. And to give you a feel for it, another educator, Jeff Petty, he interpreted 0.5 as being like a one grade leap at GCSE level.
[Speaker 2]
Wow, okay, that's significant.
[Speaker 1]
Very. And an effect size of 1.0. Well, that's often linked to advanced achievement. Petty saw that as potentially a two grade leap.
[Speaker 2]
A two grade leap. That's huge. So these numbers really mean something tangible in terms of student progress.
[Speaker 1]
Absolutely. It's a powerful way to compare interventions.
[Speaker 2]
So what did Hattie find? What are some of the heavy hitters from his 2016 updates? Anything surprising?
[Speaker 1]
No, definitely. Some things are perhaps expected, like feedback that has a massive effect size, 1.13. Okay, good feedback matters.
[Speaker 2]
Makes sense.
[Speaker 1]
Yeah. A student's prior cognitive ability, what they bring to the table, that's also huge, 1.00. And instructional quality, just good teaching basically, is 1.00. Right, high impact.
[Speaker 2]
What about the lower end? Any surprises there?
[Speaker 1]
Well, computer-assisted instruction came in at 0.31, so below that average mark.
[Speaker 2]
Hmm, interesting. We hear so much about tech in the classroom.
[Speaker 1]
And here's the one that often gets people talking. Physical attributes of the classroom, specifically class size, actually had a negative effect size in his analysis, minus 0.05. Wait, negative?
[Speaker 2]
So smaller classes don't automatically mean better results according to this massive data set?
[Speaker 1]
That's what Hattie's synthesis suggests. It really flies in the face of common assumptions, doesn't it?
[Speaker 2]
It really does. Everyone assumes smaller is better.
[Speaker 1]
It implies that maybe just shrinking the class isn't the magic bullet. It's more about what the teacher does in that class, the quality of the teaching, the feedback, than the actual number of students.
[Speaker 2]
Fascinating. So it shifts the focus away from just structure and onto the actual teaching practise.
[Speaker 1]
Exactly.
[Speaker 2]
And it was this focus on effective practise, these high effect sizes, that led Sharon to zero in on a method promoted by another theorist, Robert Marzano.
[Speaker 1]
That's right. Marzano also focused on strategies with high effect sizes. For instance, he found that tasks asking learners to identify similarities and differences, he called them same or different tasks, had an effect size of 1.32. Whoa, 1.32, even higher than feedback. Impressive, yeah.
[Speaker 2]
And, crucially for Sharon's project and our discussion, graphic organisers came in with an effect size of 1.24. 1.24. Okay, so graphic organisers, we're talking things like mind maps, flow charts, Venn diagrams, comparison tables.
[Speaker 1]
Exactly. Those visual tools that help you organise information, connect new ideas to what you already know, maybe improve memory.
[Speaker 2]
Right. And the thinking is they work because our brains like structure and visuals help with that structure.
[Speaker 1]
That's the cognitive basis, yes. They're meant to align with how our mental processes work, making learning more efficient and sticky.
[Speaker 2]
So Sharon had this really strong theoretical backing. Graphic organisers should be super effective based on Marzano's findings and the concept of effect sizes. She had a powerful theory to put to the test.
[Speaker 1]
Precisely. The stage was set for her action research.
[Speaker 2]
So, armed with all this theory, these big numbers, and that drive to improve her own teaching, Sharon set up a project. What did it actually look like in her business administration classes?
[Speaker 1]
Well, she mentioned being concerned that learners were maybe just, you know, ticking boxes on their portfolios.
[Speaker 2]
Without really learning and retaining the material long-term.
[Speaker 1]
Exactly. So, she designed this project over eight weeks, spanning four different courses she was teaching consecutively, about 32 learners involved overall.
[Speaker 2]
Okay, four courses. How did she set up the comparison?
[Speaker 1]
It was quite neat. She decided two of the courses would run using her standard methods, which she said involved a lot of group discussion.
[Speaker 2]
Her control group, basically.
[Speaker 1]
Sorta, yeah. And then for the other two courses, she deliberately integrated these graphic organisers into the lessons.
[Speaker 2]
The experimental group.
[Speaker 1]
Right. And importantly, all four groups worked on the exact same project task. Planning an event for community learning.
[Speaker 2]
So, she kept the content the same, just vary the teaching method for two groups.
[Speaker 1]
Yeah.
[Speaker 2]
That makes the comparison fair.
[Speaker 1]
Yep. Comparing apples to apples, as you said earlier.
[Speaker 2]
And how did she measure the impact? What data did she collect to see if the graphic organisers actually made a difference?
[Speaker 1]
She used a mix, what researchers call a dual approach. Both qualitative and quantitative data.
[Speaker 2]
Okay, so numbers and observations.
[Speaker 1]
Pretty much. The qualitative stuff came from her own notes during the lessons. She'd observe engagement, participation, whether learners seemed to recall information spontaneously.
[Speaker 2]
It's your teacher's eye, basically.
[Speaker 1]
Exactly. And then for the quantitative, the numbers part, learners filled out short surveys at the end of their course.
[Speaker 2]
Asking them what?
[Speaker 1]
Things like, did they enjoy the resources? How well did they feel they could recall the information needed for the project? She also used some assessment questionnaires.
[Speaker 2]
So, a combination of her observations and the learners self-reported experiences and recall. It sounds like a well structured little experiment.
[Speaker 1]
Yeah, a deliberate attempt to see if that impressive 1.24 effect size for graphic organisers would actually show up in her classroom reality.
[Speaker 2]
Okay, the moment of truth then. After all the planning, the teaching, the data collection, what did Sharon actually find? Did the graphic organisers live up to their theoretical hype?
[Speaker 1]
Well, the results were definitely interesting, but maybe not the clear-cut win for graphic organisers that the theory might have suggested.
[Speaker 2]
Oh, how so?
[Speaker 1]
Sharon herself concluded that the project was, in her view, too small, just those 32 learners, to totally convince her to overhaul her entire teaching style based only on this.
[Speaker 2]
Okay, so not a slam-dunk.
[Speaker 1]
No, and she also reflected that her survey questions could have been better designed, which is a common thing to realise after your first action research cycle.
[Speaker 2]
Right, learning about the research process itself, but what did the data actually show, even if it wasn't definitive?
[Speaker 1]
Okay, so the quantitative surveys. Most learners, 30 out of the 32, said they enjoyed using the handouts, which included the graphic organisers in those two groups.
[Speaker 2]
That's pretty positive. What about the two who didn't?
[Speaker 1]
They apparently mentioned not being entirely sure what they were supposed to do with them, which is interesting in itself.
[Speaker 2]
Yeah, implies a need for clear instruction on how to use the tool.
[Speaker 1]
Right, but here's the kicker. When she looked at the overall outcomes, like did they pass the qualification and their ratings on things like recall, there was no significant difference between the graphic organiser groups and the discussion groups.
[Speaker 2]
No difference at all.
[Speaker 1]
Not a significant one, no. All 32 learners across all four courses achieved the qualification and apparently to a very good standard.
[Speaker 2]
Wow, so the method with the 1.24 theoretical effect size didn't actually produce better results than her standard discussion method in this particular case.
[Speaker 1]
That's what her data indicated, yes. Quite the plot twist, as you said.
[Speaker 2]
It really is. Okay, so the numbers were basically a wash. What about her qualitative data, her observations?
What did she see happening?
[Speaker 1]
Her notes showed that learners seemed fully engaged in all four courses, which is good.
[Speaker 2]
Regardless of method.
[Speaker 1]
Right, but she did write, and I'm quoting loosely here, that they seemed more participative during the group discussion.
[Speaker 2]
Oh, interesting. More talkative and involved in the discussion groups.
[Speaker 1]
That was her observation. She also noted that a couple of times she actually had to give more support to the learners using the graphic organisers.
[Speaker 2]
Hmm, maybe wrestling with the tool itself.
[Speaker 1]
Possibly. And regarding recall remembering the project information, she noted it was slightly better when graphic organisers were used.
[Speaker 2]
Ah, okay, so a slight edge there.
[Speaker 1]
But, she wrote, not enough to be detrimental to those during the discussions. So a very subtle difference, if any.
[Speaker 2]
So theory suggested a huge impact. But in practise, in this setting with these learners, it was minimal. And discussions even seemed slightly more engaging.
What did Sharon take away from all this?
[Speaker 1]
Well, one positive thing for her was realising she probably already used quite a few high-impact strategies in her regular teaching. Good feedback, clear instruction, things Hattie also highlights. That boosted her confidence.
[Speaker 2]
Right, maybe her control method was already pretty effective.
[Speaker 1]
That seems likely. But her main conclusion, the big takeaway, was that while one method might be great for one group, it won't necessarily work for everyone. There's no silver bullet.
[Speaker 2]
Ah, the diversity of learners.
[Speaker 1]
Exactly. Her ultimate conclusion was that you really need a blend of teaching methods to be truly effective and reach everyone.
[Speaker 2]
Mix it up, basically. Don't just rely on one super technique.
[Speaker 1]
That was her key learning about teaching.
[Speaker 2]
That feels like a really important practical insight. It's not about finding the one perfect method, but having a toolkit and using the right tool or combination of tools for the job.
[Speaker 1]
Precisely.
[Speaker 2]
And what about her own journey as a researcher? What did she learn about doing this kind of enquiry?
[Speaker 1]
She felt her strengths were in, like, digging into the theories and planning the project carefully.
[Speaker 2]
Okay.
[Speaker 1]
But she was quite honest about her weaknesses, feeling less confident about collecting the responses, designing really effective survey questions, the whole data collection side. That takes practise. For sure.
But importantly, she expressed a real desire to keep developing those skills, maybe do larger studies, involve other tutors. That's the spirit of action research, continuous improvement, not just for the teaching, but for the research skills, too.
[Speaker 2]
What a fascinating deep dive that was.
[Speaker 1]
Yeah.
[Speaker 2]
We've gone from action research as a concept to the power or potential power of effect sizes. And then seen a real-world test where the practise didn't quite match the theory, leading to some really valuable insights.
[Speaker 1]
Yeah. This project, even though it was small, is a perfect example of that gap that can sometimes exist between a compelling theory and the messy reality of a classroom. But it also shows the huge value in teachers doing this kind of reflective experimentation themselves.
It reminds us that even theoretically strong methods aren't magic.
[Speaker 2]
It's the thoughtful blend, the adaptation to your specific learners, that seems to matter most.
[Speaker 1]
Absolutely. You need that mix to cater to everyone.
[Speaker 2]
Sharon's study really hints that even the methods with the highest scores need that careful blended approach to really work for everybody. It makes you think, doesn't it? How does this apply beyond the classroom?
When you're trying to explain something complex or persuade someone, is one single approach ever enough?
[Speaker 1]
Probably not. Tailoring your message, using different angles, maybe visuals and discussion. It seems like that blend is key almost anywhere information is shared.
[Speaker 2]
A great thought to end on. It's not just about finding the best cool, but building the best toolkit.
[Speaker 1]
Well said.
[Speaker 2]
Thank you for joining us for this deep dive. We hope it sparked some thoughts about how we learn, how we teach, and how we can all get a little bit better at both.
[Speaker 1]
I hope so, too.
[Speaker 2]
We'll see you next time.