Outputs are not outcomes

By Natalia I. Kucirkova, Centre Director

Many EdTech organisations believe they are measuring progress and learning by collecting data on user engagement and learning analytics. However, upon closer examination, we, researchers at the Centre, often find that these organisations are measuring outputs rather than outcomes. Understanding this distinction is crucial to assessing the real impact EdTech organizations have on learners. Let’s unpack this important difference.

Logic Model as a first step to model impact

Typically, organizations develop a logic model that outlines how their inputs lead to outputs and, ultimately, to impact. If you don’t have a logic model, it’s essential to design one and have it verified by researchers.

Here’s a simple example:

Platform A provides professional development for teachers and access to an app that offers exercises in English as a second language. The activities include in-app exercises, quizzes, teacher worksheets, and implementation tips for classroom use. The outputs—which platforms often track through their own data collection mechanisms—might include the number of active users. However, impact is measured through outcomes, both short-term and long-term. A short-term outcome could be improved English-speaking skills, while a long-term outcome might be increased confidence and the ability to participate fully as a citizen.

Crucially, outcomes are not about usage or engagement—they reflect what is actually achieved through that engagement.

Measuring outcomes

EdTech companies often claim that students are learning with their platforms— how can they know this? With the outputs and outcomes distinction comes also the differentiation between internal and external measures.

Internal Measures in an education game refer to assessments built within the platform that track a player’s progress and performance. These measures typically include in-game quizzes, level progression, completion rates, and accuracy in answering game-based questions. They help evaluate how well a student is engaging with and mastering the content within the game but do not necessarily indicate whether the knowledge can be applied outside the game environment.

External Measures are assessments conducted outside the game to evaluate whether the learning gained transfers to real-world applications. These may include standardized tests, teacher-administered evaluations, real-life problem-solving tasks, or independent assessments of skills beyond the game. Therefore, external measures provide a more robust indication of whether the game’s learning objectives lead to meaningful educational outcomes.

External measures can be integrated into EdTech platforms, but they must be rigorously validated for their psychometric properties and developed by experts in the relevant field (for example language development). Our researchers are often called in to design these assessments to ensure they measure real learning outcomes rather than just in-game progress.

For example, in one project, an EdTech platform tracked whether a child progressed from Level 1 to Level 2 in a literacy game and included a quiz to assess their understanding of Level 1 concepts before advancing. However, if the quiz only tested knowledge directly tied to the game, it merely confirmed that the child had mastered the game—not necessarily that they had acquired transferable skills. To address this, our researchers developed an external measure as part of the platform’s assessment system. Instead of focusing solely on content covered in the game (internal or proximal measures), this new assessment incorporated elements beyond the app’s direct instruction, aligning with broader learning goals (external or distal measures).

A distal measure evaluates whether children have acquired skills they should know regardless of whether they used the app. A good example is a standardized language test given to all children of a certain age, assessing their language proficiency against established developmental norms. These broader assessments provide a clearer picture of true learning outcomes. In a controlled experiment, if the usage of an app can show improvement on such a distal measure, it is a rare but robust finding because it means that it really improves the child’s outcomes.

Because most countries have standardized assessment tests to measure children’s progress and define educational curricula, education outcomes can often be aligned with these established metrics. However, they still need to be adapted to the specific organization or EdTech company contributing to them.

For example, in one project, an appropriate external measure did not exist in a standardized format for the target language (Arabic). As a result, the project first had to develop and standardize the assessment before integrating it into the app. This was a significant achievement, as the app can now be used to assess children's language skills at a national level, something that has attracted strong interest from the government, which is considering a nationwide rollout.

Is the Focus on Outcomes Only for Mature Companies?

A common question from organizations is whether outcomes-based measurement applies only to mature companies, with some assuming it is too ambitious for startups. This is partially true. The key factor is not the company’s size but how rigorously its solution has been designed and tested for impact. Startups with a smaller scale of implementation will naturally show different results than large-scale solutions that include extensive resources and additional services. Generally, the more comprehensive a company's intervention, the more likely it is to detect its impact with external assessment measures.

Regardless of company size, evaluating tools through the lens of the science of learning is essential—considering how children learn, typical developmental progress rates, and the expected added value of the tool. The effectiveness of any solution also depends on the optimal "dosage" or intensity of use, as well as implementation factors like how teachers or parents integrate the tool into learning. Fortunately, education has long tackled these challenges, and many established measurement frameworks exist. These models not only enhance education impact investing but also offer solutions to the broader impact investment sector, which has traditionally struggled with inconsistent or overly individualized metrics. EdTech outcomes-based contracting presents a powerful new approach to impact investing, offering insights that could help standardize and improve impact measurement across the social, environmental and educational sectors.

Cite this article as: Kucirkova, N. (2024). Outputs are not outcomes, Insights: International Centre for EdTech Impact.