The Efficacy-Effectiveness Pyramid

By Prof Natalia I. Kucirkova, Centre Director

The efficacy-effectiveness pyramid is a hierarchical framework that organises the various research components in a company’s evidence portfolio. It’s often depicted as a pyramid to illustrate how different types of evidence build upon each other over time, mapping onto product maturity and increasing confidence in findings as research progresses. While the pyramid model isn’t perfect—since it suggests a linear progression rather than the cyclical nature of research—it serves as a helpful concept for combining efficacy and effectiveness studies. This structure can guide EdTech companies in creating comprehensive, consistent research plans that evolve with their products, supporting an evidence base that grows in depth and rigour over time.

Efficacy and Effectiveness Outcomes As Impact Indicators

When we talk about "impact" in EdTech , we are referring to the practice of intentionally drawing both financial and educational returns. We are looking beyond traditional financial returns to include educational, social, and environmental outcomes. Impact is not about outputs—like tracking platform completions or counting quiz completions. The real impact of EdTech is seen in whether it drives substantial improvements, such as helping users increase their income or supporting measurable gains in children’s academic performance through third-party assessments.

Efficacy and effectiveness are two ways of looking at the outcomes.

The Efficacy- Effectiveness Distinction

Efficacy studies are traditionally rooted in quantitatively driven methodologies, particularly in fields like medical science, where there is a hierarchical approach to evidence. This hierarchy begins with foundational research, such as logic models and theories of change, progresses through correlational studies, and culminates in randomized controlled trials (RCTs) as the gold standard. This model forms the basis of the ESSA standards in the U.S. and is echoed in other frameworks, such as the Australian AERO standards.

Effectiveness studies, on the other hand, focus on the pedagogical quality of educational products as a starting point, guiding further evaluations. These assessments may include observing students' use of the tool, conducting replication studies across varied contexts, implementing longitudinal evaluations, and analysing cost-effectiveness in different implementations. Emphasising the variation within implementation, effectiveness studies often involve collaborative research and development within partnerships that span research, practice, and industry, allowing for a more inclusive approach to evaluation that adapts to real-world educational settings. You often find such approaches in EdTech testbed initiatives.

The Efficacy-Effectiveness Continuum

Since our Centre’s inception, we’ve advocated that efficacy and effectiveness are not in opposition or a strict sequence, but rather two complementary approaches that should be integrated in EdTech evaluations. Both efficacy and effectiveness have unique strengths, and both can drive innovation at any stage.

Efficacy and effectiveness studies can and should be combined for holistic edtech evaluations.

But what might this combined approach look like in practice? Using the models of efficacy and effectiveness organized by consolidated benchmarks—where different weights of evidence are assigned to various methodologies—we can identify three evidence levels: emerging, moderate and strong evidence. These levels map onto each other to form a progressive pathway, blending efficacy and effectiveness research into a continuous cycle of insight and development.

 In this cycle, each study informs the next, creating a succession of evidence that reflects a product’s maturity and an ongoing generation of insights. This accumulated evidence contributes to a company’s evidence portfolio, which remains dynamic and evolves with each new study. From ensuring that there is alignment between users’ views and literature insights, all the way to rigorous tests through randomization and replication of the positive results across contexts, research-based innovations can flourish. Through this approach, evidence collection becomes an ongoing learning process, employing diverse methodologies suited to different questions in different contexts. This integrated model not only strengthens the impact of EdTech products but also supports a culture of continuous improvement and adaptive innovation.

Kucirkova, 2024

Interested to learn more about our Centre’s approach to holistic evidence? Efficacy and Effectiveness are two of the five impact pathways we focus on. Check out our work on Ethics, Equity and Environment here.

Cite this article as: Kucirkova, N. (2024). The Efficacy-Effectiveness Pyramid, Insights: International Centre for EdTech Impact.