Selecting the right measurement tools for outcomes-based interventions in K12 EdTech

By Professor Natalia I. Kucirkova, Centre Director

Outcomes-based interventions in K12 EdTech focus on assessing whether educational innovations lead to tangible improvements in student learning and teaching practices. With increasing awareness among users and policymakers that many EdTech tools lack proven impact—alongside tightening education budgets and growing political scrutiny—the demand for such impact evaluations is surging. This shift is set to transform educational policy and practice, ensuring accountability, smarter resource allocation, and a stronger emphasis on what truly enhances learning in an increasingly competitive landscape.

Is your EdTech ready for an impact evaluation?

There are several methodological steps that need to be followed for impact evaluations to work well. One key step is to check whether the EdTech solution (typically a platform or learning product, provided with teacher professional development training and accompanying activities) is ready to be evaluated in this way. Conducing an impact evaluation study too early in the development process risks wasting precious evaluation resources that could have been allocated to design or development of the product.

Figure 1 illustrates how different stages in the maturity cycle of an EdTech tool align with specific types of studies appropriate for each phase. While this progression may vary across tools and the path from one study to another is rarely linear (as indicated by the grey arrows showing multiple pivots and iterations back to earlier stages), the overall trajectory represents a typical process for assessing whether an EdTech product is ready for impact evaluation.

Figure 1: Alignment between product maturity and type of studies over time, ICEI Copyright.

Outcomes-oriented interventions typically do not focus on gathering teachers' perspectives or engaging in co-design with stakeholders; rather, they occur at a more advanced stage of evaluation. By this point, the tools have already undergone testing for implementation feasibility, alignment with the context of use and design compatibility with users’ needs. These steps are crucial because if an intervention does not meet the practical requirements of its intended users ( for instance, if an educational app functions only online but the target community has unreliable internet access or limited data), achieving meaningful impact becomes unlikely.

Theory of Change as your North Star

Once the impact evaluation study has been identified as a suitable fit, the next crucial step is developing a Theory of Change for the intervention. The Theory of Change should outline how and why the intended effects will occur, drawing on existing literature to justify the expected changes. It is essential to clearly articulate the underlying assumptions and distinguish between short-term and long-term outcomes.

Importantly, outputs should not be mistaken for outcomes: for example, measuring the time children spend using an educational app provides insights into usage patterns but does not indicate meaningful learning gains. In contrast, outcomes reflect significant changes over time, such as improvements in foundational literacy or numeracy skills or socio-emotional development. The Theory of Change should be developed by domain experts to ensure its validity and to serve as a guiding framework for the evaluation design.

The Devil of Detail Is in the Design

A poorly designed evaluation can compromise the validity of a study, leading to unreliable or misleading results. A key consideration therefore is the actual design of the intervention and dedicating sufficient time to develop a robust study design. A robust study design needs to have appropriate sample size calculations to guide recruitment, as well as accounting for expected attrition rates based on prior research in the specific geographical and demographic context.

Ethics Boards will also require details on the duration and intensity of engagement with the EdTech solution to ensure that any observed effects are attributable to the tool usage. This involves determining the required dosage in terms of both frequency and intensity of use. Sporadic engagement with an app for just a few minutes, without consistency or structured content progression, is unlikely to produce meaningful learning outcomes. Therefore, establishing the optimal duration and frequency of use is a critical factor in designing robust impact evaluation studies.

Another critical aspect of an evaluation design is selecting the appropriate measurement tools. The measurement tools may be newly developed for the specific study; however, in such cases, they must undergo rigorous piloting and psychometric validation before implementation. Alternatively, standardized measures, which are widely recognized and validated assessment tools, are preferable as they enhance the generalizability of findings by ensuring that observed progress is meaningful beyond the specific intervention (and this increases the study’s value in broader educational and policy contexts).

The Education Outcomes Fund (EOF) has recently published a valuable resource that compiles key standardized measures for early childhood interventions. The EOF’s technical brief provides a comprehensive summary of these tools, detailing their quality characteristics and applicability in various study settings. You can watch the experts and scholars involved in developing the technical brief discuss the significance of these standardized measures in this accompanying video.

The true power of outcomes-based projects lies in their ability to align financial incentives with measurable impact—ensuring that innovation in education translates into meaningful, lasting change. As more organizations share their expertise in outcomes-based contracting, the field can move toward standardized processes that drive greater impact. This convergence enables comparative studies, a richer pool of meaningful data, and ensures that children and teachers participate in studies that are both relevant and beneficial. Moreover, it fosters a shift in EdTech development, pushing creators to design products that prioritize demonstrated learning outcomes rather than mere reach.  

Cite this article as: Kucirkova, N. (2025). Selecting the right measurement tools for outcomes-based interventions in K12 EdTech, Insights: International Centre for EdTech Impact.