• Learning Science & Insights

    News & Media

News & Media

A new standard for measuring the efficacy of ed tech?

A new standard for measuring the efficacy of ed tech?

Four takeaways from the 2018 Impact Research Advisory Council Summer Summit

2018-08-17   |   Category: In the News

Designing impact research to give instructors the practical insights they need for what ed tech to use and how to use it is challenging, and incredibly rewarding. To support us on this endeavor, Macmillan Learning assembled the Impact Research Advisory Council (IRAC).

As a panel of recognized experts, the Council has been with us since the start - guiding us in developing a forward-looking strategy for measuring the efficacy of our digital products, critiquing the design of each study, and evaluating our analyses, findings, and claims.

Each Summer, the IRAC convenes at the Macmillan Learning office in New York City with members of the Learning Science and Insights Team to assess the past year, cue up some challenging topics, plan for the year ahead, and enjoy the company of educators and researchers who are passionate about empirically improving education. Here are some of the biggest takeaways:

1. Building understanding of how instructors choose to implement edtech and how students are engaging with it provides essential context. In the last year, our Impact Research team has worked on the ground with more than 90 instructors to systematically document their educational contexts, challenges, and if and how the solutions they are using are effective. When asked to critique our approach, the IRAC members unanimously agreed that starting with these contextual studies is not just important, it's essential. Dr. Lane commented, "although [randomized controlled trials] have been the gold standard it's time to start thinking outside of the box and what you've done here is just that. Well done."  Michael Feldstein commented, "this is how you really help instructors understand how [efficacy research] can really be meaningful to them".   

2. The research-based Learning Model underlying Macmillan Learning's new products holds promise for increasing efficiencies for instructors teaching and students learning. Underlying our next generation of products is a research-based learning model. This is a synthesis of "what works" in educational research and cognitive psychology, carefully operationalized and sequenced to give our product developers a blueprint for developing digital solutions that are engineered to drive greater student success. The IRAC members were overwhelmingly positive about the rigor and value of the Learning Model and its use in product design. Dr. Sara Finney enthusiastically commented that "[the Learning Model] is just what the Professors in my department need, as soon as I saw this I wanted to post it on our departmental website!" Dr. Finney remarked that she had recently increased her own course schedule from two days a week to three just to help her students develop effective study techniques, but that the Learning Model provided a more efficient and practical synthesis. "I would never have had to [add a day to my course schedule] if I had a product where these things were already built in. [Macmillan's Learning Model] will really increase efficiencies [for students and instructors]".  

3. Macmillan Learning's approach to Learning Science is "revolutionary" and its transparency provides instructors with evidence and insights they can have confidence in. At Macmillan, we've gone to great lengths to develop end-to-end approach to user-centered design, applying learning research, and using insights from analytics and impact evaluation to develop impactful products, and provide fast and meaningful insights during develop to improve and refine them. The IRAC members reviewed those insights reports and examples of they're being applied in real time to refining products. Dr. Thanos Patelis remarked, "I'm sorry to act hysterical about this, but this is really revolutionary".  The IRAC members identified additional data to capture over time, methods of analysis, and complementary insights to build on what they'd seen and provide transparency to instructors of an "efficacy black box".

4. There is much more work to be done. Three suggestions that the IRAC provided which the Impact Research team has committed to implementing this year are (1) identify instructors who are not partnering with us on studies to independently review and critique the instruments we use (surveys, rubrics, and performance assessments) and validate that the data we are capturing and the outcomes we are measuring are those that will help them the most, (2) work with more Centers for Teaching and Learning to expand the research with individual instructors to their institutions at large, and (3) share more our ethics and data privacy policies in our public reports and publications since, as Dr. Elana Zeide commented "you follow all of the proper protocols, so why not share that with your readers?".

The two day Summit was packed with intense discussions, lively debate around challenging topics, and a lot of laughs. What impresses me most about the members of this Council is that they're able to bring deep and diverse expertise, and collaboratively identify and refine practical solutions. And, along the way, we have a lot of fun.

Much like our Learning Science program, the IRAC grows stronger each year.  At the 2018 IRAC Summer Summit the energy in the room was palpable, and we all left inspired and ready to continue the important work we're doing to try to help educators and students to succeed.

For more information on the work of the Learning Science and Insights team head to our website. You can also find results of our most recent studies here.  If you would like more information on partnering on a study you can reach out to me directly at kara.mcwilliams@macmillan.com.