A new standard for measuring the efficacy of ed tech?

August 17th, 2018

Image Description Here

Designing impact research to give instructors the practical insights they need for what ed tech to use and how to use it is challenging, and incredibly rewarding. To support us on this endeavor, Macmillan Learning assembled the Impact Research Advisory Council (IRAC).

As a panel of recognized experts, the Council has been with us since the start - guiding us in developing a forward-looking strategy for measuring the efficacy of our digital products, critiquing the design of each study, and evaluating our analyses, findings, and claims.

Each Summer, the IRAC convenes at the Macmillan Learning office in New York City with members of the Learning Science and Insights Team to assess the past year, cue up some challenging topics, plan for the year ahead, and enjoy the company of educators and researchers who are passionate about empirically improving education. Here are some of the biggest takeaways:

1. Building understanding of how instructors choose to implement edtech and how students are engaging with it provides essential context. In the last year, our Impact Research team has worked on the ground with more than 90 instructors to systematically document their educational contexts, challenges, and if and how the solutions they are using are effective. When asked to critique our approach, the IRAC members unanimously agreed that starting with these contextual studies is not just important, it's essential. Dr. Lane commented, "although [randomized controlled trials] have been the gold standard it's time to start thinking outside of the box and what you've done here is just that. Well done." Michael Feldstein commented, "this is how you really help instructors understand how [efficacy research] can really be meaningful to them".

2. The research-based Learning Model underlying Macmillan Learning's new products holds promise for increasing efficiencies for instructors teaching and students learning. Underlying our next generation of products is a research-based learning model. This is a synthesis of "what works" in educational research and cognitive psychology, carefully operationalized and sequenced to give our product developers a blueprint for developing digital solutions that are engineered to drive greater student success. The IRAC members were overwhelmingly positive about the rigor and value of the Learning Model and its use in product design. Dr. Sara Finney enthusiastically commented that "[the Learning Model] is just what the Professors in my department need, as soon as I saw this I wanted to post it on our departmental website!" Dr. Finney remarked that she had recently increased her own course schedule from two days a week to three just to help her students develop effective study techniques, but that the Learning Model provided a more efficient and practical synthesis. "I would never have had to [add a day to my course schedule] if I had a product where these things were already built in. [Macmillan's Learning Model] will really increase efficiencies [for students and instructors]".

3. Macmillan Learning's approach to Learning Science is "revolutionary" and its transparency provides instructors with evidence and insights they can have confidence in. At Macmillan, we've gone to great lengths to develop end-to-end approach to user-centered design, applying learning research, and using insights from analytics and impact evaluation to develop impactful products, and provide fast and meaningful insights during develop to improve and refine them. The IRAC members reviewed those insights reports and examples of they're being applied in real time to refining products. Dr. Thanos Patelis remarked, "I'm sorry to act hysterical about this, but this is really revolutionary". The IRAC members identified additional data to capture over time, methods of analysis, and complementary insights to build on what they'd seen and provide transparency to instructors of an "efficacy black box".

4. There is much more work to be done. Three suggestions that the IRAC provided which the Impact Research team has committed to implementing this year are (1) identify instructors who are not partnering with us on studies to independently review and critique the instruments we use (surveys, rubrics, and performance assessments) and validate that the data we are capturing and the outcomes we are measuring are those that will help them the most, (2) work with more Centers for Teaching and Learning to expand the research with individual instructors to their institutions at large, and (3) share more our ethics and data privacy policies in our public reports and publications since, as Dr. Elana Zeide commented "you follow all of the proper protocols, so why not share that with your readers?".

The two day Summit was packed with intense discussions, lively debate around challenging topics, and a lot of laughs. What impresses me most about the members of this Council is that they're able to bring deep and diverse expertise, and collaboratively identify and refine practical solutions. And, along the way, we have a lot of fun.

Much like our Learning Science program, the IRAC grows stronger each year. At the 2018 IRAC Summer Summit the energy in the room was palpable, and we all left inspired and ready to continue the important work we're doing to try to help educators and students to succeed.

For more information on the work of the Learning Science and Insights team head to our website. You can also find results of our most recent studies here. If you would like more information on partnering on a study you can reach out to me directly at kara.mcwilliams@macmillan.com.

Image Description Here

Edupreneurship: Using Research and Evaluation to Devise Innovations that Drive Outcomes

Entrepreneurship in educational technology, also known simply as edupreneurship, is a hot topic-so hot that there are conferences devoted to it (including next week's ASUGSV), philanthropic efforts focused on it (such as the Chan Zuckerberg Initiative and Bill and Melinda Gates Foundation), and even graduate degrees concentrating on it (including the University of Pennsylvania's M.S. in Education Entrepreneurship).

April 9, 2019

Image Description Here

The Power of Active Learning: Does It Work?

“Active learning enhances motivation, deepens retention, promotes transfer, and builds collaborative skills useful in work and life.” Dr. Chris Dede, Harvard Graduate School of Education.

March 11, 2019

Image Description Here

Using student-centered design to create more engaging edtech: A SXSWedu Educator Workshop

At Macmillan Learning, we have been experimenting with and refining these techniques over the last two years to develop a next-generation or more effective and engaging digital experiences for students. We have learned that involving students from the very first steps and throughout the design process is fundamental to creating learning products that are empathetic, drive higher student engagement, and enable instructors to achieve better outcomes.

March 11, 2019

Learning Science News

Browse Our Archive

Educational Results

Science + Empathy + Data Insights

The result of this empathetic and scientific approach are learning solutions that are practical, deeply intuitive, highly impactful, and help students, instructors, and institutions to achieve their very best outcomes.

See Our Latest Educational Results