- Our Story
- College Success
- Electrical Engineering
- Environmental Science
- Music & Theater
- Nutrition and Health
- Philosophy & Religion
- Political Science
- Contact Us
A new standard for measuring the efficacy of ed tech?
August 17th, 2018
Designing impact research to give instructors the practical insights they need for what ed tech to use and how to use it is challenging, and incredibly rewarding. To support us on this endeavor, Macmillan Learning assembled the Impact Research Advisory Council (IRAC).
As a panel of recognized experts, the Council has been with us since the start - guiding us in developing a forward-looking strategy for measuring the efficacy of our digital products, critiquing the design of each study, and evaluating our analyses, findings, and claims.
Each Summer, the IRAC convenes at the Macmillan Learning office in New York City with members of the Learning Science and Insights Team to assess the past year, cue up some challenging topics, plan for the year ahead, and enjoy the company of educators and researchers who are passionate about empirically improving education. Here are some of the biggest takeaways:
1. Building understanding of how instructors choose to implement edtech and how students are engaging with it provides essential context. In the last year, our Impact Research team has worked on the ground with more than 90 instructors to systematically document their educational contexts, challenges, and if and how the solutions they are using are effective. When asked to critique our approach, the IRAC members unanimously agreed that starting with these contextual studies is not just important, it's essential. Dr. Lane commented, "although [randomized controlled trials] have been the gold standard it's time to start thinking outside of the box and what you've done here is just that. Well done." Michael Feldstein commented, "this is how you really help instructors understand how [efficacy research] can really be meaningful to them".
2. The research-based Learning Model underlying Macmillan Learning's new products holds promise for increasing efficiencies for instructors teaching and students learning. Underlying our next generation of products is a research-based learning model. This is a synthesis of "what works" in educational research and cognitive psychology, carefully operationalized and sequenced to give our product developers a blueprint for developing digital solutions that are engineered to drive greater student success. The IRAC members were overwhelmingly positive about the rigor and value of the Learning Model and its use in product design. Dr. Sara Finney enthusiastically commented that "[the Learning Model] is just what the Professors in my department need, as soon as I saw this I wanted to post it on our departmental website!" Dr. Finney remarked that she had recently increased her own course schedule from two days a week to three just to help her students develop effective study techniques, but that the Learning Model provided a more efficient and practical synthesis. "I would never have had to [add a day to my course schedule] if I had a product where these things were already built in. [Macmillan's Learning Model] will really increase efficiencies [for students and instructors]".
3. Macmillan Learning's approach to Learning Science is "revolutionary" and its transparency provides instructors with evidence and insights they can have confidence in. At Macmillan, we've gone to great lengths to develop end-to-end approach to user-centered design, applying learning research, and using insights from analytics and impact evaluation to develop impactful products, and provide fast and meaningful insights during develop to improve and refine them. The IRAC members reviewed those insights reports and examples of they're being applied in real time to refining products. Dr. Thanos Patelis remarked, "I'm sorry to act hysterical about this, but this is really revolutionary". The IRAC members identified additional data to capture over time, methods of analysis, and complementary insights to build on what they'd seen and provide transparency to instructors of an "efficacy black box".
4. There is much more work to be done. Three suggestions that the IRAC provided which the Impact Research team has committed to implementing this year are (1) identify instructors who are not partnering with us on studies to independently review and critique the instruments we use (surveys, rubrics, and performance assessments) and validate that the data we are capturing and the outcomes we are measuring are those that will help them the most, (2) work with more Centers for Teaching and Learning to expand the research with individual instructors to their institutions at large, and (3) share more our ethics and data privacy policies in our public reports and publications since, as Dr. Elana Zeide commented "you follow all of the proper protocols, so why not share that with your readers?".
The two day Summit was packed with intense discussions, lively debate around challenging topics, and a lot of laughs. What impresses me most about the members of this Council is that they're able to bring deep and diverse expertise, and collaboratively identify and refine practical solutions. And, along the way, we have a lot of fun.
Much like our Learning Science program, the IRAC grows stronger each year. At the 2018 IRAC Summer Summit the energy in the room was palpable, and we all left inspired and ready to continue the important work we're doing to try to help educators and students to succeed.
For more information on the work of the Learning Science and Insights team head to our website. You can also find results of our most recent studies here. If you would like more information on partnering on a study you can reach out to me directly at firstname.lastname@example.org.
Entrepreneurship in educational technology, also known simply as edupreneurship, is a hot topic-so hot that there are conferences devoted to it (including next week's ASUGSV), philanthropic efforts focused on it (such as the Chan Zuckerberg Initiative and Bill and Melinda Gates Foundation), and even graduate degrees concentrating on it (including the University of Pennsylvania's M.S. in Education Entrepreneurship).
April 9, 2019
“Active learning enhances motivation, deepens retention, promotes transfer, and builds collaborative skills useful in work and life.” Dr. Chris Dede, Harvard Graduate School of Education.
March 11, 2019
Learning Science News
Browse Our Archive
Learning Science: Taking It Step-By-Step | April 19, 2019
The Power of Active Learning: Does It Work? | March 11, 2019
Learning for Life: A Reality for Everyone | January 25, 2019
The Science Behind a Better Education | November 29, 2018
Beyond the hype of big data in education | September 7, 2018
Learning Models: Enabling Research-Based Approaches to Active Learning | August 27, 2018
A new standard for measuring the efficacy of ed tech? | August 17, 2018
‘My Class Is Different!’ | August 13, 2018
Macmillan Shares Learning Science Foundations | July 17, 2018
Macmillan Learning Announces Learning Research Advisory Council | June 26, 2018
Unveiling Macmillan Learning’s Student Co-designers | April 12, 2018
Learning Science’s Impact on Digital Products You Use | April 6, 2018
Good, better, best: helping to realize student potential | January 11, 2018
Unpacking the Black Box of Efficacy | November 29, 2017
Macmillan Launches Austin Learning Lab | November 13, 2017
Two mindsets every designer and researcher should embrace | September 18, 2017
Learning science, insights, and the journey to better outcomes | June 28, 2017
Macmillan Learning Redefines a Learner-Centered Strategy | May 10, 2017
Making efficacy research more actionable | May 8, 2017
Science + Empathy + Data Insights
The result of this empathetic and scientific approach are learning solutions that are practical, deeply intuitive, highly impactful, and help students, instructors, and institutions to achieve their very best outcomes.