Measuring Change in Education Systems – Summary

Summary

Measuring changes to education systems, let alone performance, is complex. With education systems consisting of many moving parts, there are multiple approaches to assessing education system performance across the world, reflecting the reality of this complexity. This summary explores global practice for measuring education system change.

Whole article:

Measuring Change in Education Systems – Summary

This short summary accompanies a longer report, Measuring Change in Education Systems: A Review, produced by the Education Evaluation Centre (Te Ihuwaka) of New Zealand’s Education Review Office.

 

Key findings

Looking at a range of multilateral and country-specific approaches to measuring education system change, overall, we found six key findings.

  1. There is variation in approaches at both the multilateral and country level.
  2. There is more monitoring and less evaluation.
  3. Emphasis appears to be placed on different aspects of the system (for example, inputs, outputs, and outcomes) with relatively less focus on learner outcomes.
  4. The purpose of the frameworks we looked at reflects the context of the multilateral organisation or the focus of the countries at that point in time.  
  5. Frameworks also reflect either an emphasis on tracking state-of-play/status information or an improvement focus.
  6. The underlying structures and incentives that prompt multilaterals and countries to focus on the things that they do is hard to pin down. Context matters and this varied focus may reflect a pragmatic approach that reflects the capability, capacity, and resources available to the country at the time, along with the needs of the project or educational system.

How do multilaterals measure education system changes?

In this review, we looked at the following multilaterals:

  • UNESCO (United Nations Educational, Scientific and Cultural Organization)
  • UNICEF (United Nations International Children’s Emergency Fund)
  • OECD (Organisation for Economic Co-operation and Development)
  • World Bank

Multilaterals use a range of conceptual models and approaches to describe and assess the performance of education systems. Across all multilaterals, there is no single conceptual approach that emerges as more apt for measuring change in educational outcomes at a system level.

Multilateral frameworks demonstrate a strong focus on the whole education system

There are cross-cutting domains (for example, policy levers) and themes (for example, sectors) central to all multilateral frameworks from an education systems perspective.  The difference is in the inclusion/exclusion or level of emphasis placed on domains and themes and the emphasis on collecting data or information relating to each.  For example, the UNCIEF and UNESCO approaches place an emphasis on social development goals and a focus on educational infrastructure, while the OECD frameworks tend to focus on learning and educational outcomes of students alongside inputs-outputs and contextual factors.

The mixed approaches affirm the complexity inherent in education systems

This complexity is a result of the interplay between inputs, outputs, and outcomes. If learning outcomes for students are the key results of a well performing education system, then it is essential to understand how the education infrastructure (for example, policy, curriculum frameworks) and inputs and activities (for example, teacher training, pedagogical practice) all come together to help influence such outcomes. This is why multilateral frameworks seek information at different levels from different perspectives (for example, students, policy makers, and education stakeholders).

A mix of tools and methods appears to provide triangulation of results through the broad monitoring and evaluation activities undertaken

These included bespoke (for example, Programme for International Student Assessment, Multiple Indicator Cluster Surveys) and routine surveys (for example, European Union Household Labour Force Survey), thematic deep dives and stakeholder feedback, and administrative data generated by institutions and government agencies. The use of a mixed methods approach enables different insights to be generated and for results from different sources to be checked.

There is a split between multilaterals, depending on whether they primarily work in a developed or developing country context

The split between a developed and developing/emerging country-focus reflects the context and framework the respective multilateral agencies are working within.

In a developing country context, for example, the UNESCO and the World Bank frameworks focus on participation and enrolment in compulsory schooling, or proportion of learners assessed as meeting a specific grade level, or the number of qualified trained teachers. 

The Global Education Policy Dashboard (GEPD) framework includes five outcome measures, 11 indicators of practices (or service delivery), 18 policy levers, and five indicators for politics and bureaucratic capacity. There are, however, also enduring areas of education interest (for example, school governance and conditions).

Differences can also be seen at the project level. For example, the OECD-South East Asia’s use of a case study-like approach in Indonesia and Malaysia is a pragmatic approach that reflects the goals of the in-country review projects and the resources available.

Theoretical frameworks such as Human Capital Approach are referred to by some multilaterals

Both the OECD and World Bank-GEPD refer to the Human Capital Approach, but this is not the dominant lens for overall analyses. In this approach, labour market outcomes such as employment, earnings, and return on education investment are routinely monitored as part of the human capital approach.

There is no framework that comprehensively focuses on and collects learning outcomes for learners

There are multiple learner outcomes – learning progression, social and emotional skills, and academic achievement. Learner outcomes however in themselves are seldom the focus of most frameworks. Many frameworks are focused on a range of inputs and outputs (for example, policy and sectoral areas of education, and school conditions).

The frequency of the monitoring and evaluation approaches varies

The frequency of many of the evaluation approaches varies from annually to once every few years, and some of the monitoring data is not current. In addition to this, for some frameworks and associated reports using the frameworks, there is little visibility on the quality of the data collated. This can occur for a wide range of reasons including pragmatic trade-offs around timeliness of reporting and cost, capability and capacity to undertake monitoring and evaluation work, and participant sensitivities about the information.

How do individual countries measure education system changes?

In this review, we looked at Cambodia, Canada, Finland, India, Jamaica, Malaysia, Mauritius, and Scotland.

The most common way that countries evaluated their education system was through tracking status and progress against specific goals and priorities

Thematic areas were based on enduring areas of importance to the education system and the development of core education infrastructure, and then evaluated on a periodical basis.

Alongside evaluations of thematic areas, advanced countries had extensive statistical indicator programmes and, in some instances like Canada, had a programme exclusively dedicated to a suite of learning assessments including the PISA.

Scotland had the closest to providing an annual overview of the education system – with focus on what’s happening across the school system on six key improvement drivers.

Evaluation and monitoring efforts undertaken covered specific parts of the system

Monitoring and evaluation goals change over time and countries were more likely to have monitoring and evaluative information on their priority areas at any given time.

Countries have some whole system information, but there are few countries that evaluated all parts of their education system year on year.

Differences between developed and developing countries

As expected, there are some broad differences between developed and developing or emerging jurisdictions in terms of data collection methods and sources.  The differences probably reflect the stage of development of the education system in the country and the country’s internal capability and capacity.

Developed countries

The developed countries tend to have system-wide frameworks for assessing change and progress. For example, Finland in 1999 produced a map of the entire education system – placing the aspects of this under evaluation criteria such as efficiency, effectiveness, and economy. Scotland uses a National Improvement Framework (2016) with the child at the centre of this framework.

Developing countries

Administrative data was the mainstay for the developing countries in this study. Very few focused on learning outcomes and most monitor proxy markers such as school leaving qualifications.

Most developing countries had a simple monitoring dashboard of indicators on inputs-outputs and some outcomes in this review. For example, Cambodia had a score card approach on key areas, including inputs such as management education for school leaders and rates of higher teacher qualifications. India has a Performance Grading Index of 70 parameters from administrative and achievement data. Malaysia has a framework that covers a mix of input-output priorities such as access to education, quality of education, equality in education, building unity through education, and maximising efficiency.

Considerations for evaluating education system change

This review provides an overview of frameworks or principles for assessing education system measurement at a country-level. We recommend three things are considered in supporting countries to monitor education performance.

1. Developing a checklist to assess what countries are doing for system-level measurement

A checklist could assess what specific countries are currently doing in terms of system-level measurement. This could serve to identify areas for measuring system capability where support is required. Using a checklist will also provide insights into which areas of monitoring and evaluation countries prioritise and why.

2. Encouraging governments to invest in national level education management information systems

Encouraging individual countries to take more systematic approaches to measuring educational change will be useful to understand progress. Partner agency supported systems such as Systems Approach for Better Education Results (SABER) and the UNESCO Institute of Statistics can help collate data that is collected.

3. Use multilateral frameworks as a starting place for country-specific education system evaluation

Based on the review we have undertaken, Te Ihuwaka suggests that frameworks such as those from the OECD or UNESCO would make a good starting point for thinking about where a country is at in terms of education system performance. For many countries, responding to different funding agency reporting requirements can be resource intensive.  Adoption of a common multilateral reporting framework by key funding agencies could be of benefit as it would provide opportunities for comparative analysis and reduce the burden on recipient countries. However, this is not always straight forward to achieve because of the context within which the funding agencies are operating.

This research was supported by the New Zealand Aid Programme, Ministry of Foreign Affairs and Trade, New Zealand. The views expressed in this publication do not necessarily reflect those of MFAT or the New Zealand Government.

This work draws on multiple sources. For a list of references please see our full report Measuring Change in Education Systems: A Review.