Rasil Warnakulasooriya, Ph.D
VP of Analytics & Data
The promise of big data is the talk of the town these days. However, does big data in education mean big insights for educators and edtech developers? While there is potential for a wealth of illuminating and practical insights into teaching and learning that can be derived from data, that potential can only be realized by asking the right questions, creatively tackling noisy and incomplete data, and recognizing the limitations of any analysis. Simply put, gaining useful and reliable insights from educational data - big or small - is part science, and part art.
Why should technologists and educators be cautious about the promise of big data? We have identified four main challenges: 1. high variability among educational settings leading to limitations in data in any given context; 2. the reality of educational technologies only reflecting parts of the teaching and learning experience; 3. variability of educational content in both type and effectiveness; 4. the complexity and variability of conceptual domains (English composition, Economics, or Physics, and the conceptual differences within a given discipline.) These limitations also pose significant challenges in utilizing artificial intelligence techniques in education.
Although data from educational settings are now generated in large volumes, at high velocity, and with greater variety, the above limitations largely result in drowning analysts and educators in a sea of information that can obscure actionable insights or not be relevant for the most pertinent questions facing learners, teachers, and policy makers.
So how can we gain reliable insights from educational data to help advance teaching and learning? From decades of experience in analyzing data from educational technologies, we have developed a play book of best practices on how to be successful with educational data mining. In summary, we recommend data analysts:
1. Explore data from multiple angles
2. Visualize data in every stage of exploration
3. Avoid foregone conclusions
4. Understand how users engage with learning applications before conducting analyses
5. Do not reject outlier data without due consideration
6. Use point statistics cautiously
7. Isolate and adjust for confounding factors
8. Safeguard against overfitting statistical models
9. Strive to understand predictive and machine learning algorithms
10. Strive to obtain quality data
11. Align statistical thinking with scientific thinking
12. Consider how learning analytics can contribute to learning research
In the paper, Beyond the Hype of Big Data in Education, we provide more details for these guidelines, illustrate how they can be applied to a variety of practical examples in education, and share insights that we hope are of interest to curious educators.
We are entering an exciting age when learning analytics has the potential to become a major contributor to the improvement of education, when exercised with rigor and awareness.
We look forward to the journey ahead and hope this White Paper piques your curiosity and positively contributes to the discussion of how to use educational data responsibly and effectively to improve teaching and learning.
Mitchell, C., Dipetta, T. & Kerr, J. (2001). The frontier of web-based instruction. Education and Information Technologies, 6, 105-121.