This work explores global best practice for measuring education system change. It considers the variation found between different approaches to measuring education system change, looking particularly at how select individual countries and multilaterals each approach this. This work will also consider what can support countries to monitor education performance.
Whole article:Measuring Change in Education Systems: A Review
List of abbreviations
CMEC: Council of Ministers of Education, Canada
DHS: Demographic and Health Surveys
DoSEL: Department of School Education and Literacy
DWMS: Development World Management Survey
EC: Children Under Five
ERO: Education Review Office
FCF: Child Functioning
FINEEC: Finnish Education Evaluation Centre
FL: Foundational Learning
GECDD: Global Early Child Development Database
GEM: Global Education Monitoring
GEPD: Global Education Policy Dashboard
GTP: Government Transformation Programme
ICILS: The International Computer and Information Literacy Study
ICT: Information and Communications Technology
INES: Indicators of Education Systems
MERL: Monitoring, Evaluation, Research and Learning
MFAT: Ministry of Foreign Affairs and Trade
MICS: Multiple Indicator Cluster Surveys
MICS-EAGLE: Multiple Indicator Cluster Surveys – Education Analysis for Global Learning and Equity
MITD: Mauritius Institute of Training & Development
MMT: Mass Media and ICT
OECD: Organisation for Economic Co-operation and Development
PCAP: The Pan-Canadian Assessment Program
PCEIP: Pan-Canadian Education Indicators Program
PEMANDU: Performance Management and Delivery Unit
PGI: Performance Grade Index
PIAAC: Programme for the International Assessment of Adult Competencies
PIRLS: The Progress in International Reading Literacy Study
PISA: Programme for International Student Assessment
PR: Parental Involvement Module
SABER: Systems Approach for Better Education Results
SALSUS: Scottish Schools Adolescent Lifestyle and Substance Use Survey
SDG: Sustainable Development Goal
SEN: Special Education Needs
SSLN: Scottish Survey of Literacy and Numeracy
TALIS: Teaching and Learning International Survey
TEDS-M: The Teacher Education and Development Study in Mathematics
TIMSS: The Trends in International Mathematics and Science Study
UIS: UNESCO Institute for Statistics
UNESCO: United Nations Educational, Scientific and Cultural Organization
UNICEF: United Nations International Children's Emergency Fund
USAID: United States Agency for International Development
This research was supported by the New Zealand Aid Programme, Ministry of Foreign Affairs and Trade, New Zealand.
The views expressed in this publication do not necessarily reflect those of MFAT or the New Zealand Government.
This review was undertaken by ‘Te Ihuwaka’, the Education Review Office’s (ERO) Education Evaluation Centre. It explores global best practice for measuring change, currently used by multilateral and government institutions outside the Pacific, to inform New Zealand Ministry of Foreign Affairs and Trade’s (MFAT) education focused work. This review used a solely desk-based research approach to draw on evidence about education measurement approach practices.
As would be expected, the review found substantial variations in the approaches. Across all multilaterals, there is no single conceptual model or approach that emerges as distinct and more apt for measuring change in educational outcomes at a system level. For example, the UNESCO and UNICEF frameworks have an emphasis on the Social Development Goals, the World Bank similarly has a development lens, and the OECD framework reflects the economic maturity of the advanced countries that it serves. While all approaches straddle key domains of the education system, they privilege some domains that are reflective of their specific mandates.
There are some broad differences between developed and developing jurisdictions in terms of data collection methods and sources. The differences, we suggest, probably reflect the stage of development of the education system in the country and the country’s internal capability and capacity. Typically, administrative data seems to be the mainstay for the developing countries in this study. Very few focussed on actual learning outcomes and mostly monitor proxies such as school leaving qualifications. The most common way that countries evaluated their education system was through the lens of tracking status and progress against specific goals or priorities.
Alongside status tracking and evaluations of thematic areas, advanced countries had extensive statistical indicator programmes and, in some instances like Canada, had a programme exclusively dedicated to a suite of learning assessments including the PISA. Scotland focused on what’s happening across the school system on six key improvement drivers so that it could direct improvement efforts accordingly.
Most developing countries had a simple monitoring dashboard of indicators on inputs-outputs and some outcomes in this review. India has, in recent years, developed a performance grading system (70 weighted indicators across five key domains) that groups various states that enable improvement focus. Most developing countries may be expected to align some level of reporting to meet multilateral commitments; however, we could not sight such reporting during this research on selected countries.
Overall, the review shows that:
- there is variation in approaches and practices at both levels
- there is more monitoring and less evaluation
- emphasis appears to be placed on different aspects of the system: inputs, outputs and outcomes, with relatively less on learner outcomes
- the purpose of the frameworks reflects the context of the multilateral organisation and the focus of the countries at the point in time
- frameworks also reflect either an emphasis on tracking state-of-play/status information or an improvement focus emphasis
- the underlying structures and incentives that prompt multilaterals and countries to focus on the things that they do is hard to pin down as context matters. At best, they appear to focus on what is fit for purpose in their contexts. This, in our experience, probably reflects a pragmatic approach that reflects the capability, capacity and resources available to the country at the time, along with the needs of the project or educational system.
One option for improving our understanding is to undertake further research to obtain insights about the gaps in knowledge, and to understand the appetite for a more cohesive approach to monitoring education system performance by funding recipient countries that MFAT operates with.
Overall, there are several useful things to consider based on this review. The review provides an overview of frameworks or principles for assessing education system measurement at a country-level. This could be further developed into a generic framework, or detailed at the level of inputs-outputs-outcomes as necessary for a given country’s context and development need. This can help with mapping education measurement systems and gap analysis in the first instance.
A checklist could be developed to assess what specific funding or recipient countries are currently doing or aspiring to do in terms of system level measurement. Such an approach could serve to identify areas for measuring system capability and where support is required or could be provided.
Using a mapping or checklist approach with a select list of funding countries will also provide insights into which areas of monitoring and evaluation they prioritise and why (for example, national interests, international commitments, resources).
Partner agency supported systems such as SABER and the UNESCO Institute of Statistics can only help collate the data that is available . However, these systems can be used to encourage national governments to invest in more robust national level educational management information systems.
Part 1: Introduction
The New Zealand Aid Programme is managed by the Pacific and Development Group of MFAT. The aid investments on education total about NZ$30-40 million per year, with nearly 60 percent going towards the Pacific region. The investments are aligned to MFAT’s Strategic Frameworks and Strategic Results Frameworks  and usually spread over three to five years. These investments directly relate to SDG 4: ‘ensure inclusive and equitable quality education and promote lifelong learning opportunities for all’. The monitoring and evaluation to date tends to focus mostly on outputs (such as the number of teachers who received training) and is mostly undertaken by implementing partners.
The purpose of this review is to understand the various measurement approaches used across developed and developing jurisdictions. MFAT is committed to understanding the impact of its education investments in the Pacific. This review undertaken by Te Ihuwaka, ERO’s Education Evaluation Centre, explores global best practice approaches for measuring change, currently used by multilateral and government institutions outside the Pacific, to inform MFAT’s education focused work.
How to use this document
This report is laid out in two parts and is supported by two detailed appendices.
The first part outlines the objectives for this research and the methodological approach.
The second part outlines the key findings from this review of the approaches of a set of multilaterals and selected countries on assessing educational outcomes and system performance. We pose pointers for future consideration by MFAT in the use of the frameworks. The analysis is based on the material presented in the two substantive appendices.
Appendix 1 presents a detailed description and brief commentary on the frameworks developed by multilateral organisations.
Appendix 2 focuses on frameworks used by selected countries identified as of interest to MFAT. Similarly, a description and brief commentary is presented for each country.
Objectives for this review
The project focuses on global best practice for measuring change in education system outcomes. The review covers:
- education outcomes measurement systems rather than programmes or specific education investments or policy choices
- frameworks and approaches for measuring change in school education systems among global agencies that have an education focused mandate (for example, OECD, UNESCO)
- frameworks and approaches that help monitor equity of education outcomes
- practices and approaches among selected countries for measuring changes in school education systems
- innovative approaches to systems level monitoring and evaluations
- analysis of the different systems and approaches reviewed.
The project scope excludes:
- project, programme, and output focus
- Monitoring, Evaluation, Research and Learning (MERL) tools and approaches for evaluating education investments
- attribution and causality
- Pacific and development context.
In discussion with MFAT, Te Ihuwaka undertook a desk-based literature research review focused on identifying examples of education system performance and outcomes frameworks used by multilateral institutions and countries. A desk-based research approach is not without limitations. Limitations for desk-based research could include: not being able to connect frameworks with subsequent monitoring work, the rationale behind frameworks and alignment to indicators, prioritisation of indicators and subsequent changes to reporting, and the reasons why, as well as the quality of the data and the value of insights. Without communicating with in-country experts or key informants or access to a wide range of related data and documentary sources, the insights from a desk-based review can be limited.
Standard academic literature search databases were used to help identify key documents for review. These included the following databases:
- ERIC – an excellent range of scholarly research relating to all areas of education. Includes surveys, descriptions and evaluations of programmes, curriculum and teaching guides, instructional materials, position papers and resource materials.
- Education Research Complete - a database focused on all aspects and levels of education. This includes journal and magazine articles, book chapters, and conference papers. Geographic coverage is international, including New Zealand.
Websites searched included:
- OECD - http://www.oecd.org/education/
- UNESCO - https://en.unesco.org/themes/education
- Education | UNICEF
- World Bank - Education (worldbank.org) 
Key words such as education frameworks, outcomes, monitoring, were used to identify candidates. Other criteria were then developed to select countries to review, these included:
- an existing education outcomes measurement system to review
- material about the country’s education outcomes measurement system was available in English and could be accessed on the internet
- a basic level of education system infrastructure in place and system maturity to assess education system activity and performance
- a spread of geographic locations.
Institutions and countries
Using the criteria, the frameworks and approaches of the following institutions and countries were identified and included for review:
- multilateral: UNESCO, UNICEF, World Bank and OECD Directorate of Education
- countries: United Kingdom, Finland, Canada, Mauritius, Malaysia, Cambodia, Caribbean (Jamaica or Cuba) and India.
Approach to analysis of the frameworks
The review work was organised in three sequential parts comprising a scan of multilateral agency approaches, a country scan, and thematic analysis. A focus of the study was on understanding the different ways of thinking about measuring change in education system outcomes. We considered:
- the context, type and focus of the theoretical or conceptual framework used, such as the OECD and UNICEF education frameworks. This analysis also looked at the extent to which measurement systems exist as a pillar of the wider education system and/or education policy system
- the degree to which the framework or outcomes monitoring approach followed good outcomes measurement practice at a system level. For example, to what extent did the framework identify:
- core components
- add-on components
- standard tools and monitoring instruments to implement the framework
- innovative elements
- institutional arrangements to support the implementation of the framework (for example, systematic review points, benchmarking)
- what the outputs of the frameworks looked like in practice.
Part 2: Key findings and discussion
What frameworks do multilaterals use?
Multilaterals use a range of conceptual models and approaches to describe and assess the performance of education systems. These are outlined in the following table and described in detail in Appendix 1– How multilateral institutions measure change in education systems.
Key findings from Te Ihuwaka analysis of the table and Appendix 1 are listed below.
- Across all multilaterals, there is no single conceptual model or approach that emerges as distinct and more apt for measuring change in educational outcomes at a system level. Each has been developed in the context of their organisational focus and tends to emphasise different components. For example, the UNESCO and UNICEF frameworks have an emphasis towards the Social Development Goals, the World Bank similarly has a development lens, and the OECD framework reflects the economic maturity of the advanced countries that it serves.
- The UNESCO SDG 4 Monitoring framework has a set of global indicators (outputs) as well as thematic indicators (inputs and outputs) representing context (for example, national context, policy priorities, technical capacity, and data availability).
- The World Bank SABER system has a conceptual model of inputs linked to outcomes with a focus on outputs. This system looks at namely policies and institutions of education systems with emphasis for their quality.
- The GEPD framework (adapted from the 2018 World Development Report conceptual framework) has learning outcome indicators at the centre, ringed by indicators for institutional practices and the wider political context and bureaucratic capacity of the system.
- The OECD has the most elaborate and comprehensive inputs-activities-outputs-outcomes-impact framework bounded by contextual factors.
OECD Indicators of Education Systems (INES) Framework
Performance of national systems as a whole; allows for benchmarks and country comparisons
Organising framework/ model
Human capital approach; encompasses multiple levels of inputs-outputs-outcomes and key actors; contextual factors (demographic, socio-economic and political)
Participation and learning outcomes; school settings; wider enabling conditions
Each element and their interplay; (e.g. quality of education outcomes, equality and equity; adequacy, efficiency of resources; relevance of policy measures)
INES drawn from OECD and Eurostat databases, plus tailored surveys (e.g. PISA)
OECD Education Policy Outlook
Assess education policy systems
Organising framework/ model
Focus on policy issues, strategy
Country reviews, and OECD bespoke survey and data
OECD Education Policy Reviews
Country specific reviews; cross-country studies
Country reviews, and OECD bespoke survey and data
Adapted to developing and emerging country contexts
GEM monitoring towards SDG4 progress and education in other goals; comparison across countries
Organising framework/ model
Alignment to SDG4 priorities
Development centric: covers 11 global and 43 thematic indicators; three sectors
Household surveys, learning assessments, and administrative data (supported by work to harmonise data and build capability)
World Bank - SABER
Reforming education systems at country level; build high-quality knowledge-base for global systems
Organising framework/ model
Policy areas framework
Evaluating quality of policies and institutions (13 areas) – rubrics and indicators (four-point scale)
Surveys through a principal country investigator
World Bank - GEPD
Speedily identify areas and gaps for improving learning outcomes
Organising framework/ model
Learning outcomes; enabled by practices and politics/bureaucracy capacity
Dynamic progress dashboard (across the three areas)
Three surveys: schools; policy; and public officials
Improving learning outcomes and equity through building national capacity and global data foundation
Organising framework/ model
Equity lens (seven areas): gender, socio-economic status, ethnicity etc
Development context: data analysis linked to further analysis of policy issues
MICS6 Survey (sole data source)
Are there any key gaps?
The analysis showed that there were two key gaps.
- Learner outcomes are focused on a range of inputs and outputs (for example, policy and sectoral areas of education, and school conditions). There is no framework that comprehensively focuses on and collects learning outcomes for learners.
- The frequency of many of the monitoring and evaluation approaches appear to vary from annual to once every few years and some of the monitoring data is not current. In addition to this, for some frameworks and associated reports using the frameworks, there is little visibility on the quality of the data collated. This can occur for a wide range of reasons including pragmatic trade-offs around timeliness of reporting and cost, capability, and capacity to undertake monitoring and evaluation work, and participant sensitivities about the information.
Likely rationale for framework selection
The multilateral frameworks demonstrate several aspects as below.
- A strong focus in tracking across the whole education system. There are cross-cutting domains (for example, policy levers) and themes (for example, sectors) central to all of them from an education systems perspective. The difference is in the inclusion/exclusion or level of emphasis placed on domains and themes and the emphasis on collecting data or information relating to each. For example, the UNCIEF and UNESCO approaches place an emphasis on Social Development Goals and a focus on educational infrastructure (i.e. ‘inputs’), while the OECD frameworks tend to focus on learning and educational outcomes of students alongside inputs-outputs and contextual factors.
- The mixed approaches affirm the complexity inherent in education systems and that impacts result from the interplay of inputs-outputs-outcomes; hence they seek information at different levels from different perspectives and actors (for example, students, policy makers, and education stakeholders).
- The use of a mix of tools and methods appears to provide triangulation of results through the broad monitoring and evaluation activities undertaken. These included bespoke (for example, PISA, MISC) and routine surveys (for example, EU HLFS), thematic deep dives and stakeholder feedback, and administrative data generated by institutions and government agencies. The use of a mixed methods approach enables different insights to be generated and for results from different sources to be checked. Tools such as PISA and MISC are established data collection tools and methods, and are internationally acknowledged as representative of best research practice.
- The approaches reflect a general split between developed and developing/emerging countries in line with country development context.
- Within the UNESCO and the World Bank frameworks, the focus is on participation and enrolment in compulsory schooling, or proportion of learners assessed as meeting a specific grade level, or the number of qualified trained teachers.
- The GEPD framework includes five outcome measures, 11 indicators of practices (or service delivery), 18 policy levers, and five indicators for politics and bureaucratic capacity. There are, however, also enduring areas of education interest (for example, school governance and conditions) and signs of adaptation (for example, PISA-D survey for developing jurisdictions).
- The OECD-SEA’s use of a case study approach in Indonesia and Malaysia is a pragmatic approach that reflects the goals of the in-country review projects and the resources available.
- The split between developed and developing/emerging country-focus reflects the context and framework the respective multilateral agencies are working within. Differences can also be seen at the project level, and the monitoring activities reflect the specific purpose of the review projects.
- Theoretical frameworks such as Human Capital Approach are referred to by the OECD and World Bank-GEPD but are not necessarily the dominant theme for overall analyses. In this approach, labour market outcomes such as employment, earnings, and return on education investment are routinely monitored as part of the human capital approach. In contrast, Table 1 and the discussion in Appendix 1 show that the UNICEF and UNESCO frameworks are explicitly set within the Social Development Goal framework, which places a greater emphasis on establishing core building blocks for education delivery. These building blocks include, for example, rooms, teaching resources, developing appropriately trained teachers, and establishing a monitoring system.
What frameworks do individual countries use?
At the country level a range of frameworks for measuring educational system activity and educational outcomes have been reviewed. These are described in detail in Appendix 2 – Practices in select countries.
The following table presents a summary of the range of approaches that have been taken. Commentary key insights are provided below the table. Of the examples reviewed, the Scottish framework is the closest to ERO’s model.
Finland Education evaluation plans
Determine education objectives and priorities for the next four years
Plan for evaluation projects
Assessments of learning outcomes, thematic and system evaluations, international evaluations
The plan guides evaluation activities for that four year period. Certain areas are evaluated rather than the whole system
Finland Evaluation Framework
Tool designed to assist national level evaluations of educational outcomes in Finland
Framework with educational outcomes at the top, evaluation dimensions in the middle, and evaluation objects below
The evaluation dimensions are efficiency, effectiveness and economy
A wide view of the whole education system. No evidence could be found of this framework being used in Finland to measure educational outcomes
Improve education system in Scotland through collecting information and conducting and monitoring improvement activities
Drivers of improvement around the child, parents, school, local and national levels
Four key priority areas (learner outcomes, closing the gaps, health and wellbeing, employability, and skills) and six key drivers (leadership, teacher professionalism, parental engagement, assessment, school improvement, and performance information) of improvement
Each year a new plan is created, updating on the progress since previous year and stating the activities going forward
Canada - Framework for Statistics of Learning and Education in Canada
Tool for organising information within learning and education to assist with prioritisation
Inputs, processes, outputs, and outcomes across learner, provider, and jurisdiction levels and education levels
The framework document suggests a wide range of data sources to use
It looks across the whole system – a wide view. No evidence of this framework being used to evaluate education systems can be found
Canada - Learning Assessment Programmes
Education in Canada
Canada runs a set of programmes to assess academic achievement across the different provinces and compare these to international outcomes
A list of assessment programmes they run or take part in
This is a way Canada can compare education academic outcomes across their country and to other countries. It only focuses on assessment outcomes
Performance Grading Index (PGI) to grade States and Union Territories on education system performance
Framework encompassing five domains and 70 indicators
Learning outcomes and quality, access, infrastructure and facilities, equity, and governance processes
Enables grading without ranking; helps sub-national units with continuous improvement to focus on domains that need grade improvement
Show data on education for previous year
Inputs and outputs (for example, public expenditure, enrolment numbers, and exam results)
This dashboard provides very limited information
Evaluating education system and vision of where they want to be (monitoring and tracking against goal)
A loosely organised framework, reporting on progress annually
Five key areas - access, quality, equality, unity, efficiency – reported on annually
Education outcomes are monitored in relation to progress against their clearly set out goals
Cambodia - Strategic Plan
Their strategic plan is in line with the SDG 4 (two key policies with indicators below)
Indicator framework only
Two policy areas (indicators): (inclusive and equitable quality education (eight), effective leadership and management (two)
They create a new plan every four years which identified progress from previous plan and shows goals going forward
Cambodia - Education Road Map
Strategy to improve education through identifying and tracking policy priorities
Indicators sitting below policy priorities and expected results
They have identified five priority areas (access for all learners to quality ECE, basic, upper secondary, vocational/tertiary education and literacy and numeracy for learners in all age groups)
A long-term plan for improving education through policy priorities
Plan to be a developed country with a goal or national outcome specifically related to education
More of an indicator framework below the national outcomes
Monitoring against targets
Long-term plan which touches on educational outcomes
Key findings from this analysis of the above table and Appendix 2 are listed below.
- Across the countries reviewed, in general there are no universal coherent frameworks used by countries for measuring education outcomes at a system level that approach the comprehensive nature of the multilateral frameworks – with one exception, Finland.
- The developed countries tend to have system-wide frameworks for assessing change and progress. However, national or annual level reporting is not consistently aligned to frameworks.
- For example, Finland in 1999 produced a map of the entire education system (1999) including lifelong learning and placed the map under evaluation criteria such as efficiency, effectiveness, and economy. However, the recent Finnish four-year national evaluation outputs plan does not appear to align with the comprehensive 1999 map.
- Similarly, Canada’s multi-dimensional framework covers not only the pillars of learning (early childhood, elementary-secondary, postsecondary, adult Learning) but also the information levels (learner, provider, jurisdiction) and the dynamic elements (inputs, processes, output and outcomes). However, it is unclear how the Canadian Federal Framework aligns with actual provincial review activity, and with established tools and instruments for generating information to populate the framework.
- Scotland uses a National Improvement Framework (2016) with the child at the centre of this framework. As the framework suggests, it is guided by an improvement focus on a mix of priority goals and system improvement drivers at the level of schools and institutions. Improving labour market outcomes is one of the four priorities. The Scottish framework has similarities to the ERO ‘Learners’ model that informs ERO’s review services in their evaluation of school performance.
- Of the developing countries, India and Malaysia have relatively more substantial frameworks. India’s PGI stems from the rationale that there must be constant monitoring of inputs, outputs, and outcomes, and fast course correction in order to create an efficient, inclusive, and equitable school education system. This is similar to the approach of some multilaterals. Malaysia has a framework subsumed within its education National Blueprint. The relevant section ‘Current Performance’ covers a mix of input-output priorities such as access to education, quality of education, equality in education, building unity through education, and maximising efficiency.
- Mauritius, Cambodia, and Jamaica have no explicit or implicit frameworks.
Overall, what can we say about selected countries?
As expected, there are some broad differences between developed and developing or emerging jurisdictions in terms of data collection methods and sources. The differences, we suggest, probably reflect the stage of development of the education system in the country and the country’s internal capability and capacity.
- Typically, administrative data seems to be the mainstay for the developing countries in this study. Very few focused on actual learning outcomes and most monitor proxies such as school leaving qualifications. In Te Ihuwaka’s experience, to make the move in monitoring activity requires a considerable increase in capability, capacity, and resourcing, which may not be available for some of the countries reviewed.
- The most common way that countries evaluated their education system was through the lens of tracking status and progress against specific goals and priorities.
- Thematic areas were based on enduring areas of importance to the education system and the development of core education infrastructure, and then evaluated on a periodical basis. This was the case for a range of countries and tends to reflect political cycles or project development cycles.
- Alongside evaluations of thematic areas, advanced countries had extensive statistical indicator programmes and, in some instances like Canada, had a programme exclusively dedicated to a suite of learning assessments including the PISA.
- Scotland had the closest to providing an annual overview of the education system – with focus on what’s happening across the school system on six key improvement drivers but with no reference to the wider system settings (for example, workforce supply).
- Most developing countries had a simple monitoring dashboard of indicators on inputs-outputs and some outcomes in this review. Cambodia had a score card approach on key areas, for instance 10 core breakthrough indicators. These included inputs such as management education for school leaders and higher teacher qualifications. India has had a performance grading system (five domains and 70 indicators) since 2017-18 and Malaysia has a comprehensive baseline about key parts of the system including student outcomes, however, how this will be assessed periodically is unclear.
- Most developing countries may be expected to align some level of reporting to meet multilateral commitments. However, we could not sight such reporting during this research on selected countries.
- Generally, of the countries reviewed, the type of evaluation and monitoring efforts undertaken only covered specific parts of the system and:
- these monitoring and evaluation goals can change over time
- these countries have some whole system information (inputs and outputs and outcomes)
- there are few countries that evaluated all parts of their education system year on year. India’s new PGI reporting is an attempt at systematic annual measurement of the education system. In Te Ihuwaka’s experience, this type of activity is very resource intensive. Its usefulness depends on what the national priorities are and how sensitive the priorities are to changes in the system
- countries are more likely to have monitoring and evaluative information on their priority areas at any given time.
Conclusions and next steps
Education systems are complex with elements of linearity and non-linearity . The presence of multiple approaches to assessing education system performance reflects the reality of the complexity.
The purpose of the review was to understand the various approaches, across developed and developing jurisdictions, to measuring the performance of education systems.
Within the scope of the review, Te Ihuwaka has provided insights about the types of approaches and practices at multilateral and country levels to educational system performance.
Overall, the review shows that:
- there is variation in approaches and practices at both levels
- there is more monitoring and less evaluation
- emphasis appears to be placed on different aspects of the system - inputs, outputs and outcomes with relatively less on learner outcomes
- the purpose of the frameworks reflects the context of the multilateral organisation and the focus of the countries at the point in time
- frameworks reflect either an emphasis on tracking state-of-play/status information or an improvement focus emphasis
- the underlying structures and incentives that prompt multilaterals and countries to focus on the things that they do is hard to pin down as context matters. At best they appear to focus on what is fit for purpose in their contexts. This, in our experience, probably reflects a pragmatic approach that reflects the capability, capacity and resources available to the country at the time, along with the needs of the project/educational system.
There are some gaps that remain in our understanding of what countries do.
- The links between strategy documents and annual or periodical reporting is often unclear. There are several possible reasons for this, which could be explored as part of another project.
- Alignment between frameworks and various indicator dashboards and reporting schedules varies.
- The presence or absence of institutional or system level accountabilities and capabilities for producing various indicator or evaluation outputs appears to be a common issue. There are several possible reasons for this, including a lack of resourcing.
- How country efforts are linked to multilateral reporting obligations and the quality of such reporting is not always clear. For many countries, responding to different funding agency reporting requirements can be resource intensive. Adoption of a single multilateral reporting framework across countries by key funding agencies, could be of benefit as it would provide opportunities for comparative analysis and reduce the burden on recipient countries. However, this is not always straight forward to achieve because of the context that the funding agencies are operating within themselves.
One option for improving our understanding is to undertake further research to obtain insights about the gaps in knowledge, and to understand the appetite for a more cohesive approach to monitoring education system performance by funding recipient countries that MFAT operates with.
Overall, there are several useful things to consider based on this review.
- The review provides an overview of frameworks or principles for assessing education system measurement at country-level. This could be further developed into a generic framework or detailed at the level of inputs-outputs-outcomes as necessary for a given country’s context and need. This can help with mapping education measurement systems and gaps analysis in the first instance.
- A checklist could be developed to assess what specific funding/recipient countries are currently doing or aspiring to do in terms of system level measurement. Such an approach could serve to identify areas for measuring system capability and where support is required or could be provided.
- Using a mapping or checklist approach with a select list of funding countries will also provide insights into which areas of monitoring and evaluation they prioritise and why (for example, national interests, international commitments, and resources).
- Partner agency supported systems such as SABER and the UNESCO Institute of Statistics can only help collate the data that is available . However, these systems can be used to encourage national governments to invest in more robust national level educational management information systems.
Based on the review we have undertaken, Te Ihuwaka suggests that the OECD or UNICEF frameworks would make a good starting point for thinking about where a country is at in terms of education system design. These frameworks provide a systems-level view for thinking about the educational system as a whole and what appropriate monitoring and evaluation activities could look like for any given type of educational intervention in the system. They have components that align well with the Social Development Goals and have well established resources and instruments for data collection that could be flexibly used depending upon the focus of the education system development or intervention.
Appendix 1– How multilateral institutions measure change in education systems
OECD organising framework of education systems evaluation
The OECD approach to education systems evaluation is laid out in Education at a Glance 2020: OECD Indicators. According to the OECD, their framework offers a rich, comparable, and up-to-date array of indicators that reflect a consensus among professionals on how to measure the current state of education internationally. The indicators provide information on the human and financial resources invested in education, how education and learning systems operate and evolve, and the returns to investments in education.
They are organised thematically, each accompanied by information on the policy context and interpretation of the data. The indicators are organised within a framework that distinguishes between the actors in education systems, groups them according to the types of issues they address, and examines contextual factors that influence policy. In addition to these dimensions, the time perspective makes it possible to visualise dynamic aspects of the development of education systems.
Actors in education systems
The OECD Indicators of Education Systems (INES) programme seeks to gauge the performance of national education systems as a whole, rather than to compare individual institutional or other subnational entities. However, there is increasing recognition that many important features of the development, functioning and impact of education systems can only be assessed through an understanding of learning outcomes and their relationships to inputs and processes at the level of individuals and institutions.
To account for this, the first dimension of the organising framework distinguishes the three levels of actors in education systems. These are:
- education systems as a whole
- providers of educational services (institutions, schools), as well as the instructional setting within those institutions (classrooms, teachers)
- individual participants in education and learning (students). These can be either children or young adults undergoing initial schooling and training, or adults pursuing lifelong learning programmes.
The second dimension of the organising framework is the education indicators, which are grouped into three sections. The indicators are dynamic year-to-year and are tailored to fit an overarching focus. For 2020, the indicators focus on vocational education and the transition into work.
Indicators on the output, outcomes, and impact of education systems
Output indicators analyse the characteristics of those exiting the system, such as their educational attainment. Outcome indicators examine the direct effects of the output of education systems, such as the employment and earning benefits of pursuing higher education. Impact indicators analyse the long-term indirect effects of the outcomes, such as the knowledge and skills acquired, contributions to economic growth and societal wellbeing, and social cohesion and equity.
The following indicators on outputs, outcomes and impact are used by the OECD in 2020.
- A1: To what level have adults studied?
- A2: Transition from education to work: Where are today's youth?
- A3: How does educational attainment affect participation in the labour market?
- A4: What are the earnings advantages from education?
- A5: What are the financial incentives to invest in education?
- A6: How are social outcomes related to education?
- A7: To what extent do adults participate equally in education and learning?
Indicators on the access, participation, and progression within education entities
These indicators assess the likelihood of students accessing, enrolling in, and completing different levels of education, as well as the various pathways followed between types of programmes and across education levels.
The following indicators on access, participation, and progress are used by the OECD.
- B1: Who participates in education?
- B2: How do early childhood education systems differ around the world?
- B3: Who is expected to complete upper secondary education?
- B4: Who is expected to enter tertiary education?
- B5: Who is expected to graduate from tertiary education?
- B6: What is the profile of internationally mobile students?
- B7: How do vocational education systems differ around the world?
Indicators on the input into education systems or the learning environment
These indicators provide information on the policy levers that shape the participation, progression, outputs, and outcomes at each level. Such policy levers relate to the resources invested in education, including financial, human (such as teachers and other school staff), and physical resources (such as buildings and infrastructure). They also relate to policy choices regarding the instructional setting of classrooms, pedagogical content, and delivery of the curriculum. Finally, they analyse the organisation of schools and education systems. This includes governance, autonomy, and specific policies to regulate participation of students in certain programmes.
The OECD splits indicators on ‘inputs’ into two categories, consisting firstly of ‘financial resources invested in education’, and secondly of ‘teachers, the learning environment and the organisation of schools’. The following indicators are used when considering financial resources.
- C1: How much is spent per student on educational institutions?
- C2: What proportion of national wealth is spent on educational institutions?
- C3: How much public and private investment in educational institutions is there?
- C4: What is the total public spending on education?
- C5: How much do tertiary students pay and what public support do they receive?
- C6: On what resources and services is education funding spent?
The following indicators are used when considering teachers, the learning environment, and the organisation of schools.
- D1: How does time spent by students in the classroom vary over the years?
- D2: What is the student-teacher ratio and how big are classes?
- D3: How much are teachers and school heads paid?
- D4: How much time do teachers and school heads spend teaching and working?
- D5: Who are the teachers?
Contextual factors that influence policy
Demographic, socio-economic, and political factors are all important national characteristics to take into account when interpreting indicators. For example, the 2008 financial crisis had a significant impact on public funds available to education. The characteristics of the students themselves, such as their gender, age, socio-economic status or cultural background, are also important contextual factors that influence the outcomes of education policy.
Indicator analysis using the framework
Analysis of each element of the framework and the interplay between them contributes to understanding a variety of policy perspectives, including:
- quality of education outcomes and education opportunities
- equality of education outcomes and equity in education opportunities
- adequacy, effectiveness, and efficiency of resources invested in education
- relevance of education policy measures to improve education outcomes.
The OECD INES indicators and framework use a wide range of data sources. The main data source is the OECD and Eurostat databases which includes extensive information on OECD member states. They also use a range of tailored surveys and other data-collection tools, such as:
- annual National Labour Force survey
- annual INES Network Survey or Teachers and the Curriculum
- EU statistics on income and living conditions
- PISA (Programme for International Student Assessment)
- PIAAC (Survey of Adult Skills)
- TALIS (Teaching and Learning international survey)
- Adult Education Survey for European OECD countries
- many others.
In recent years, the INES indicators have only used limited evidence of student education outcomes provided through PISA (an international study of student education attainment) and TALIS (an international survey on teaching and learning).
Education Policy Outlook
In addition to INES indicators, the OECD also evaluates education system outcome changes through their Education Policy Outlook. Compared to INES, the Education Policy Outlook has a stronger focus on academic attainment (versus a human capital approach in INES).
The aim of the Education Policy Outlook is to provide an updated comparative perspective of policy continuity and policy change as part of education policy ecosystems, as well as provide the evidence available on their implementation outcomes. This overview of policy priorities and policies can serve as a source of inspiration for other education systems that share similar challenges and contextual characteristics
The Education Policy Outlook mainly utilises OECD and Eurostat data for statistics. The main sources of OECD data include Education at a Glance reports (discussed above), the Programme for International Student Assessment (PISA), and the OECD Teaching and Learning International Survey (TALIS). In some cases, where no OECD or Eurostat data is available, national data is consulted. For country-level policy information, the Education Policy Outlook National Survey for Comparative Policy Analysis is used, along with various OECD publications of country-based analysis.
The Education Policy Outlook uses an analytical framework to examine education policy ecosystems. Drawing on OECD work with countries on education policy, this framework serves as a lens through which readers can review education systems from the point of view of students, institutions, and systems. This framework is provided below and explores policy issues (and their respective levers and evidence) at three levels: students, institutions and systems.
Through its global network UNESCO supports education policy development, and in-country, regional, and global policy analyses, and national strategic planning processes. It does this all within the framework of the global education agenda, Sustainable Development Goal (SDG) 4 - Education 2030 and the targets of SDG4.
SGD 4: Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all.
A key role of UNESCO is monitoring the progress towards the education targets in the 2030 Sustainable Development Agenda. The Global Education Monitoring (GEM) report is the global mechanism for monitoring and reporting on SDG4 and on education in the other SDGs. One half of the GEM report is dedicated to this monitoring function.
The GEM report also reports on the progress of implementation of national, regional and international strategies to help hold all relevant partners accountable for their commitments as part of the overall SDG follow-up and review. This content is included in the ‘thematic’ half of the GEM report, with overarching themes changing annually. For 2020, the theme was inclusive education. The themes of previous reports have been migration, accountability, and people and planet. The thematic chapters are separate from the monitoring framework but can also be used to consider the broad components to include in evaluation of education systems. These are:
- laws and policies
- data collection
- governance and finance (including education ministries, ministries/agencies in other related sectors, subnational education authorities, and NGOS)
- curricula, textbooks, and assessments
- students, parents, and communities.
Indicator framework to monitor SDG4 targets
At the international level, UNESCO uses a set of indicators to monitor progress against achieving SDG4. These include 11 global indicators (blue rows in Table 3) and 43 thematic indicators (white rows in Table 3). These indicators were developed by the Inter-Agency and Expert Group on Sustainable Development Goal Indicators based on inputs from countries, international and regional organizations, civil society, academia, and the international community.
The 11 global indicators represent the minimum set of indicators proposed to countries for the global monitoring of the SDG4 targets. The broader set of 43 internationally comparable thematic indicators serve to chart global progress on education and to monitor the SDG4 education targets more comprehensively across countries, allowing the possibility to identify challenges regarding concepts of the targets that are not reflected well by the global indicators. The thematic indicators represent a recommended set of additional indicators that countries may use to monitor based on the national context, policy priorities, technical capacity, and data availability.
Target 4.1 - Primary and secondary education
By 2030, ensure that all girls and boys complete free, equitable and quality primary and secondary education leading to relevant and effective learning outcomes
- Proportion of children and young people (a) in Grade 2 or 3; (b) at the end of primary education; and (c) at the end of lower secondary education achieving at least a minimum proficiency level in (i) reading and (ii) mathematics, by sex
- Administration of a nationally-representative learning assessment (a) in Grade 2 or 3; (b) at the end of primary education; and (c) at the end of lower secondary education
- Gross intake ratio to the last grade (primary education, lower secondary education)
- Completion rate (primary education, lower secondary education, upper secondary education)
- Out-of-school rate (primary education, lower secondary education, upper secondary education)
- Percentage of children overage for grade (primary education, lower secondary education)
- Number of years of (a) free and (b) compulsory primary and secondary education guaranteed in legal frameworks
Target 4.2 - Early Childhood
By 2030, ensure that all girls and boys have access to quality early childhood development, care and pre-primary education so that they are ready for primary education
- Proportion of children under 5 years of age who are developmentally on track in health, learning and psychosocial well-being, by sex
- Participation rate in organized learning (one year before the official primary entry age), by sex
- Percentage of children under 5 years experiencing positive and stimulating home learning environments
- Gross early childhood education enrolment ratio in (a) pre-primary education and (b) and early childhood educational development
- Number of years of (a) free and (b) compulsory pre-primary education guaranteed in legal frameworks
Target 4.3 - Technical, vocational, tertiary and adult education
By 2030, ensure equal access for all women and men to affordable quality technical, vocational and tertiary education, including university
- Participation rate of youth and adults in formal and non-formal education and training in the previous 12 months, by sex
- Gross enrolment ratio for tertiary education by sex
- Participation rate in technical-vocational programmes (15 to 24-year-olds) by sex
Target 4.4 - Skills for work
By 2030, substantially increase the number of youth and adults who have relevant skills, including technical and vocational skills, for employment, decent jobs and entrepreneurship
- Proportion of youth and adults with information and communications technology (ICT) skills, by type of skill
- Percentage of youth/adults who have achieved at least a minimum level of proficiency in digital literacy skills
- Youth/adult educational attainment rates by age group, economic activity status, levels of education and programme orientation
Target 4.5 – Equity
By 2030, eliminate gender disparities in education and ensure equal access to all levels of education and vocational training for the vulnerable, including persons with disabilities, indigenous peoples and children in vulnerable situations
- Parity indices (female/male, rural/urban, bottom/top wealth quintile and others such as disability status, indigenous peoples and conflict-affected, as data become available) for all education indicators on this list that can be disaggregated
- Percentage of students in primary education whose first or home language is the language of instruction
- Extent to which explicit formula-based policies reallocate education resources to disadvantaged populations
- Education expenditure per student by level of education and source of funding
- Percentage of total aid to education allocated to least developed countries
Target 4.6 – Literacy
By 2030, ensure that all youth and a substantial proportion of adults, both men and women, achieve literacy and numeracy
- Proportion of population in a given age group achieving at least a fixed level of proficiency in functional (a) literacy and (b) numeracy skills, by sex
- Youth/adult literacy rate
- Participation rate of illiterate youth/adults in literacy programmes
Target 4.7 - Sustainable development and global citizenship
By 2030, ensure all learners acquire knowledge and skills needed to promote sustainable development, including among others through education for sustainable development and sustainable lifestyles, human rights, gender equality, promotion of a culture of peace and non-violence, global citizenship, and appreciation of cultural diversity and of culture’s contribution to sustainable development
- Extent to which (i) global citizenship education and (ii) education for sustainable development, including gender equality and human rights, are mainstreamed at all levels in: (a) national education policies, (b) curricula; (c) teacher education; and (d) student assessment
- Percentage of schools that provide life skills-based HIV and sexuality education
- Extent to which the framework on the World Programme on Human Rights Education is implemented nationally (as per the UNGA Resolution 59/113)
- Percentage of students by age group (or education level) showing adequate understanding of issues relating to global citizenship and sustainability
- Percentage of 15-year-old students showing proficiency in knowledge of environmental science and geoscience
Target 4.a – Education facilities and learning environments
Build and upgrade education facilities that are child, disability and gender sensitive and provide safe, non-violent, inclusive and effective learning environments for all
- Proportion of schools with access to: (a) electricity; (b) internet for pedagogical purposes; (c) computers for pedagogical purposes; (d) adapted infrastructure and materials for students with disabilities; (e) basic drinking water; (f) single-sex basic sanitation facilities; and (g) basic handwashing facilities (as per the WASH indicator definitions)
- Percentage of students experiencing bullying, corporal punishment, harassment, violence, sexual discrimination and abuse
- Number of attacks on students, personnel and institutions
Target 4.b - Scholarships
By 2020, substantially expand globally the number of scholarships available to developing countries, in particular least developed countries, small island developing States and African countries, for enrolment in higher education, including vocational training, information and communications technology, technical, engineering and scientific programmes in developed countries and other developing countries
- Volume of official development assistance flows for scholarships by sector and type of study
- Number of higher education scholarships awarded, by beneficiary country
Target 4.c – Teachers
By 2030, substantially increase the supply of qualified teachers, including through international cooperation for teacher training in developing countries, especially least developed countries and small island developing States
- Proportion of teachers in: (a) pre-primary education; (b) primary education; (c) lower secondary education; and (d) upper secondary education who have received at least the minimum organized teacher training (e.g. pedagogical training) pre-service or in-service required for teaching at the relevant level in a given country, by sex
- Pupil-trained teacher ratio by education level
- Proportion of teachers qualified according to national standards by education level and type of institution
- Pupil-qualified teacher ratio by education level
- Average teacher salary relative to other professions requiring a comparable level of qualification
- Teacher attrition rate by education level
- Percentage of teachers who received in-service training in the last 12 months by type of training
UNESCO data sources
As the primary responsibility for monitoring SDG4 lies at the national level, UNESCO expects countries to establish effective monitoring and accountability mechanisms which are adapted to national contexts and priorities, in consultation with all stakeholders. At the global level, the UNESCO Institute for Statistics (UIS) remains the official source of cross-nationally comparable data on education and supports countries to strengthen national education data systems.
Three data sources are key for monitoring progress on SDG 4 indicators: household (and other) surveys, learning assessments, and administrative data.
Household and other surveys are the foundation for disaggregating global education indicators by individual characteristics. Examples include completion (4.1), early childhood education participation (4.2), adult education participation (4.3), youth and adult information and communication technology skills (4.4), and adult literacy (4.6). Such surveys are also the basis for calculating global indicator 4.5.1, the parity index, by gender, location and wealth. The main international household survey programmes are the Demographic and Health Surveys (DHS) [USAID] and the Multiple Indicator Cluster Surveys (MICS) [run by UNICEF and discussed in more detail below].
Surveys should be frequent, their questions comparable and their data publicly available to allow open discussion. These conditions are met for 59 percent of countries, corresponding to 87 percent of the population. Northern Africa and Western Asia has the lowest coverage in population terms (46 percent). Oceania has the lowest coverage in country terms (29 percent). There are a range of projects that aim to harmonise international data, as well as international organisations who support statistical capacity development programmes, in the aim of having high-quality and comparable survey data across the world.
Learning assessments are the source of information on global indicator 4.1.1., but also a potential source of information on selected thematic indicators, including knowledge of environmental science (4.7.4) and bullying (4.a.2). While many countries opt to report results from their participation in cross-national assessments, national assessments are also used, for instance, for data on reading skills in countries such as China (lower secondary education) and India (primary education). The UIS database shows 26 percent of countries in Africa have reported reading skills data for the early grades of primary education since 2014, corresponding to 28 percent of the population. Considerable capacity and financial constraints need to be overcome to ensure African countries carry out nationally representative, sample-based national or cross-national assessments every three to five years that meet quality standards.
Administrative data provide information on teacher-related indicators, for example, global indicator 4.c.1 - the percentage of trained teachers. Administrative data is somewhat patchy. About 58 percent of countries in sub-Saharan Africa have reported data on primary education since 2016 but only 25 percent on upper secondary education. Data interpretation suffers from lack of clarity in the definition of trained teachers and in the distinction between trained and qualified teachers. For example, teachers can be qualified, trained, both or neither. In practice, this distinction may not be straightforward depending on country context, or may clash with established terminology whereby a qualified teacher is one who has been trained. This is a challenge the UIS will tackle with a new international standard classification for teachers.
The 2020 GEM report notes that progress has been made in formulating, endorsing, and refining an expanded SDG monitoring framework, but much more effort is needed to ensure that countries report on the global indicators across the SDGs. Custodian agencies need to communicate indicators’ meaning, significance, and methodologies to national authorities. National authorities need to collect data and build their capacity to analyse, report, and use them. Funders need to coordinate their data collection and capacity development programmes.
The World Bank utilises two projects for measuring education system performance. These are SABER (Systems Approach for Better Education Results) and the GEPD (Global Education Policy Dashboard).
SABER: Systems Approach for Better Education Results
The World Bank Group has focused its efforts in education in two strategic directions: reforming education systems at the country level and building a high-quality knowledge base for education reforms at the global level.
SABER is the primary tool used by the World Bank to achieve these goals. At the country level, it is intended to provide education systems analyses, assessments, diagnosis, and “opportunities for dialogue”. At the global level, it is intended to improve the education systems knowledge base and use this information to implement effective reforms.
Using diagnostic tools and detailed policy information, SABER produces comparative data and knowledge about education system policies and institutions. It evaluates the quality of those education policies against evidence-based global standards, with the aim of helping countries systematically strengthen their education systems.
When it was established, SABER focused exclusively on the quality of policies and institutions. It has now grown to include the quality of policy implementation (in some policy areas). SABER does not focus on the quality and quantity of education delivered, this is covered in other World Bank projects (such as the GEPD, discussed later).
The SABER programme collects comparable data on the policies and institutions of education systems around the world and benchmarks them against good practice. The World Bank states that SABER’s aim is “to give all parties with a stake in educational results a detailed, objective, up-to-date, easy-to-understand snapshot of how well their country's education system is oriented toward delivering learning”. This is based on measures that can be easily compared across education systems around the world.
For each policy area, the World Bank has undertaken substantial research to develop a “What Matters Framework Paper”. From the evidence gathered in these papers, rubrics were developed (for every policy area) to assess the performance of education systems (as well as individual schools). A rubric is a framework that sets out criteria and standards for different levels of performance and describes what performance would look like at each level.
The rubrics consist of various indicators alongside a scoring table. For each indicator, an education system (or a school) can receive one of the following scores: 1) Latent, 2) Emerging, 3) Established and 4) Advanced.
High-level indicators in each SABER policy area (adapted from World Bank)
1. Levels of education
Early childhood development
- Establishing an enabling environment
- Legal Framework
- Inter-sectoral coordination
- Implementing Widely
- Scope of programs
- Monitoring and assuring quality
- Quality standards
- Compliance with standards
- Strategic framework
- Articulating a strategic direction
- Fostering a demand-driven approach to workforce development
- Strengthening critical coordination for implementation
- System oversight
- Ensuring efficiency and equity in funding
- Assuring relevant and reliable standards
- Diversifying pathways for skills acquisition
- Service delivery
- Enabling diversity and excellence in training provision
- Fostering relevance in training programmes
- Enhancing evidence-based accountability
- Regulatory environment
- Quality assurance
- Ensuring the basic conditions for learning
- Monitoring learning conditions and outcomes
- Overseeing service delivery
- Budgeting with adequate and transparent information
- Providing more resources to students who need them
- Managing resources efficiently
- Setting clear expectations for teachers
- Attracting the best into teaching
- Preparing teachers with useful training and experience
- Matching teachers’ skills with students’ needs
- Leading teachers with strong principals
- Monitoring teaching and learning
- Supporting teachers to improve instruction
- Motivating teachers to perform
School health and school feeding
- Health-related school policies
- Safe school environment
- School-based health and nutrition services
- Health education
- Policy framework
- Financial capacity
- Institutional capacity and coordination
- Design and implementation
- Community roles – reaching beyond the school
School autonomy and accountability
- The level of autonomy in the planning and management of the school budget
- The level of autonomy in personnel management
- Role of the school council in school governance
- School and student assessment
- School accountability
Engaging the private sector
- Encouraging innovation by providers
- Holding schools accountable
- Empowering all parents, students, and communities
- Promoting diversity of supply
Assessment (classroom assessment, examinations, and system-level assessment)
- Enabling context
- System alignment
- Assessment quality
Education management information systems
- Enabling environment
- System soundness
- Quality data
- Utilisation for decision making
5. Complementary inputs and cross-cutting themes
Information and communication technologies
- Vision and planning
- ICT Infrastructure
- Skills and competencies
- Learning resources
- EMIS (Education Management Information Systems)
- Monitoring and evaluation, assessment, research and innovation
- Equity, inclusion and safety
Equity and inclusion
- Establishing an enabling environment and providing adequate resources
- Ensuring that all children and ready to learn and in school
- Ensuring all children and especially vulnerable groups learn in school
- Education in adversity
- Assets and engagement
- Relevant school and community support
- Aligned education system support
SABER primarily uses bespoke data collection instruments for gathering the required information to score an education system against each of the indicators. These are generally surveys intended for a single respondent — an experienced principal investigator in the country — to fill out using information from key informants, documents, and other sources.
In the typical model, an experienced principal investigator will collect the policy information and data necessary to fill out the data-collection instrument by drawing on his or her knowledge of the system and on government contacts. Data collection can usually be completed within a few weeks in this way. An alternative approach, used in some domains, is to convene a workshop of experts, including government officials, and use that group process to collect the evidence and code data. In either case, data sources are clearly identified and made public when the data are posted.
SABER domain teams use the gathered data to analyse “how developed the country’s education policies and institutions are”, from the perspective of achieving key education goals. In the process, it will also generate benchmarks of progress in those specific areas against other countries or provinces. In some domains, these evaluations will be embedded in a more in-depth report discussing policy options and relevant experiences from other countries.
GEPD: Global Education Policy Dashboard
The second World Bank project is the GEPD. According to the World Bank, the GEPD collects and presents data on the specific areas where countries need to act to improve learning outcomes, using indicators that can show progress relatively quickly.
The dashboard tracks progress in three areas – practices (or service delivery), policies and politics. The World Bank states that the GEPD indicators are comprehensive (in that they holistically cover the most important drivers of learning at scale) but also focused (so that they can focus stakeholders’ attention on what is most important). With these indicators, the dashboard 1) highlights gaps between what the evidence suggests is effective in promoting learning and what is happening in practice in each system; and 2) allows a way for governments to track progress as they act to close those gaps.
The structure/framework of the dashboard was adapted from the 2018 World Development Report conceptual framework (see Figure 6: World Bank GEPD framework).
At the centre are the outcome indicators capturing learning for all (meaning learning combined with access).
These are encircled by indicators representing the four main school-level service-delivery factors. These factors labelled as “practices” include: prepared learners, capable teaching, appropriate inputs and infrastructure, and capable school management. All these factors bring the other factors together to produce learning. The next set of indicators are markers for the policies that affect each of these areas, and the final set captures the political context and bureaucratic capacity of the system. Sustained system-wide improvement in learning will likely depend on better performance in these policy and politics domains. In summary, this framework considers the politics, policies and practices which all have an impact on learning outcomes.
In selecting and developing the indicators for the dashboard, the World Bank applied three main criteria, listed below.
- First, each indicator should predict better learning and access outcomes (based on sound empirical evidence or a strong conceptual presumption to support the relationship).
- Second, with concerted effort, it should be possible to improve the indicator over a relatively brief period — one to two years, say —so that the indicator can serve as a marker of progress for a government committed to the longer-term challenge of improving learning.
- And third, it should be possible to generate the data for the indicator every two years at a reasonable cost.
The list includes 39 indicators distributed among the four levels symbolized by Figure 6. These include five outcome measures, 11 indicators of practices (or service delivery), 18 policy levers, and five indicators for politics and bureaucratic capacity. Figure 7 provides a summary of all the indicators that are included. Underneath each indicator sit a range of sub-indicators. For example, for the indicator “basic infrastructure”, one sub indicator is the availability of internet connectivity.
The dashboard project collects new data in each country using three new instruments: a School Survey, a Policy Survey, and a Survey of Public Officials. Data collection involves school visits, classroom observations, legislative reviews, teacher and student assessments, and interviews with teachers, principals, and public officials. In addition, the project draws on some existing data sources to complement the new data it collects.
School Survey: The School Survey collects data primarily on practices (the quality of service delivery in schools), but also on some de facto policy indicators. It consists of streamlined versions of existing instruments—including Service Delivery Surveys on teachers and inputs/infrastructure, Teach on pedagogical practice, Global Early Child Development Database (GECDD) on school readiness of young children, and the Development World Management Survey (DWMS) on management quality—together with new questions to fill gaps in those instruments.
Policy Survey: The Policy Survey collects information to feed into the policy indicators. This survey is filled out by key informants in each country, drawing on their knowledge to identify key elements of the policy framework (as in the SABER approach to policy-data collection that the Bank has used over the past seven years). The survey includes questions on policies related to teachers, school management, inputs and infrastructure, and learners.
Survey of Public Officials: The Survey of Public Officials collects information about the capacity and orientation of the bureaucracy, as well as political factors affecting education outcomes. The survey includes questions about technical and leadership skills, work environment, stakeholder engagement, impartial decision making, and attitudes and behaviours.
While most GEPD indicators are derived from data collected using the three instruments described above, the team also draws on existing data for a small number of indicators. This is particularly key for outcome data e.g. school participation and learning. Similarly, because factors outside the education system also affect education outcomes, the dashboard also includes a few indicators based on existing data from other sectors. For example, many factors that affect whether children are in school and ready to learn lie outside the education system. Thus, policy levers for this practice area include indicators like the rate of children that are well-nourished and the share of children that are fully immunised, among others. These indicators draw on non-dashboard data sources.
UNICEF: MICS-EAGLE (Multiple Indicator Cluster Surveys – Education Analysis for Global Learning and Equity)
UNICEF launched Multiple Indicator Cluster Surveys (MICS) in 1995 to monitor the status of children around the world. Over the past 25 years, this household survey has become the largest source of statistically sound and internationally comparable data on women and children worldwide. MICS was a major data source for the Millennium Development Goals indicators and continues to inform more than 150 Sustainable Development Goals (SDG) indicators in support of the 2030 Sustainable Development Agenda.
UNICEF launched the MICS-EAGLE (Education Analysis for Global Learning and Equity) Initiative in 2018 with the objective of improving learning outcomes and equity issues in education by addressing two critical education data problems – gaps in key education indicators, as well as lack of effective data utilization by governments and education stakeholders.
MICS-EAGLE is designed to:
- build national capacity for education sector situation analysis and sector plan development, and leverage the vast wealth of disaggregated education data collected by MICS6 (or future iterations)
- build on the global data foundation provided by MICS6 to conduct analysis of disaggregated data at the national, regional, and global level that yields insights as to how barriers to education opportunities can be reduced so each child can reach his or her full potential.
MICS-EAGLE will analyse the following areas through an equity lens (gender, socio-economic status, ethnicity, etc.).
The MICS-EAGLE initiative also offers activities at the national, regional, and global levels. National level activities seek to identify specific education issues using MICS data linking to policy discussions, and key education data gaps that require additional studies or data sources. Global and regional level activities are geared towards capacity building through increased foundational knowledge regarding the use of household data analysis to inform education sector policy discussions.
The MICS6 survey is the source of evidence for MICS-EAGLE. MICS surveys are conducted by trained fieldworkers who perform face-to-face interviews with household members on a variety of topics.
MICS6 was launched in 2017 with data innovations to better reflect measurement of the SDGs and to address data availability issues. Three of the key education innovations involve assessment of early grades learning that will greatly enhance global understanding of educational outcomes and the factors contributing to learning outcomes, collection of data on parental involvement, and collection of data on ICT skills among youth and adults.
The Foundational Learning Skills (FL) module assesses learning outcomes in reading and numeracy skills expected for Grade 2 or 3 levels, covering both in-school and out-of-school children aged between 7-14 years old, which provides data for SDG4.1.1.a. The Parental Involvement (PR) module collects data about the learning environment at home and the extent to which parents are involved in the child’s education at school. The Mass Media and ICT module collects data on ICT skills covering nine ICT activities among youth and adults who are older than 15 years of age. These new modules help shed light on foundational learning and parental participation in education through a global household survey programme for the first time.
Furthermore, MICS6 offers modules on child disability (Child Functioning) and other critical information which helps assess children’s skills and identify education issues around learning and equity for the most marginalized and vulnerable individuals. These changes reinforce the value of MICS6 as a leading household survey in the field of education globally.
These and other education data collected by MICS6 play a critical role in calculating SDG indicators, including 4.1.1.a (learning), 4.2.1 (early childhood development), 4.2.2 (access to pre-primary education), 4.4.1 (ICT skills) and 4.5.1 (equity with a focus on disability).
Corresponding MICS module
Proportion of children and young people in Grade 2 or 3 in (i) reading and (ii) mathematics, by sex
Foundational Learning Skills (FL) – new module in MICS6
Proportion of children under 5 years of age who are developmentally on track in health, learning and psychosocial wellbeing, by sex
Children Under Five (EC)
Participation rate in organized learning (one year before the official primary entry age), by sex
Children Under Five (EC)
Proportion of youth and adults with information and communications technology (ICT) skills, by type of skill
Mass Media and ICT (MMT) –
new module in MICS6
Parity indices (female/male, rural/urban, bottom/top wealth quintile and others such as disability status, indigenous peoples and conflict-affected, as data become available) for all education indicators on this list that can be disaggregated
Child Functioning (FCF) – new module in MICS6
Disaggregation and regression analysis of the above mentioned SDG4 indicators with parent’s participation in education Data available in the PR module
Parental Involvement Module (PR) –
new module in MICS6
Commentary on MICS-EAGLE
Unlike other institutions, MICS-EAGLE only gathers data on outcomes for learners. The survey is firmly set within the development context and the survey is designed to specifically to answer indicators on SDG4 targets. However, UNICEF further analyses the survey findings to identify specific education issues and link this to policy discussions. Another potential limitation is that MICS-EAGLE is based on a single, self-contained survey, whereas other institutions utilise a range of data sources.
Appendix 2 – Practices in select countries
Education Evaluation Plan
In Finland, the Finnish Education Evaluation Centre (FINEEC) is responsible for the evaluation of education. The FINEEC, established in 2014, brought together the evaluation work of the Finnish Higher Education Evaluation Council, the Finnish Education Evaluation Council, and the Finnish National Agency for Education, to centralise and establish a coherent approach to education system evaluation.
Every four years Education Evaluation Plans are developed by the Evaluation Council of FINEEC and approved by the Ministry of Education and Culture. The Education Evaluation Plans are used to determine education objectives and priorities for the following four-year period. Finland’s Education Evaluation Plan for 2016-2019 details the evaluation projects and guides the development of the evaluation system in 2016–2019.
The projects include audits of quality systems in higher education institutions, assessments of learning outcomes in vocational education and training, and assessments of learning outcomes in basic education. National assessments are sample-based and carried out according to focus areas selected in the plan, generally covering a range of subjects. In addition, the plan sets out thematic and system evaluation projects, such as initial teacher education for certain subjects and the self-evaluation and quality management procedures for basic and general upper secondary education.
The individual evaluation projects appear to be the outputs of this plan and the education system performance seems to be measured at this level. There are some details about each evaluation project in the four year plan, but details such as indicators or how evaluations will be conducted are not included. The evaluation plan notes that halfway through the plan validity period, FINEEC and the Ministry meet to discuss the completed evaluations and their results as well as any revisions, additions, and specifications to the evaluation plan.
A framework for evaluating education outcomes in Finland
In addition to this, there appears to be a comprehensive framework designed for evaluating educational outcomes in Finland. This was created in 1999 by the National Board of Education. This framework does not appear to be used or referred to in the National Education Evaluation Plan 2016-2019 in relation to any of the evaluation projects. The framework states it is designed primarily for national-level evaluations but suggests evaluations need to take place at every level of the education system. The same concepts and approaches can be applied to evaluations at all levels and there is also a section about conducting school-based self-evaluations.
No evidence of this framework being used for evaluating educational outcomes at the national or local level could be found during this research. National evaluation indicators are also mentioned as a work in progress within the framework but no further evidence of these being published could be found.
Evaluating the state of the Finnish education system: Results of the Finnish education evaluation
In addition to these documents there is a further report: Evaluating the state of the Finnish education system: Results of the Finnish education evaluation. It is a summary and synthesis of evaluations conducted between 2014-2018. It highlights four key themes: 1) Trust, 2) Equity, 3) Inclusion and Wellbeing and 4) Society renewed by competence. These are different to the themes identified in the evaluation plan. This summary report makes no reference to the education outcomes framework or evaluation plan for the same time period and those documents also make no reference to this report. We assume that this summary uses information from the evaluation projects planned and conducted in accordance with the evaluation plans for the corresponding time.
In summary Finland has a few different ways in which the education system could be monitored and evaluated. The comprehensive outcomes framework created 20 years ago doesn’t seem to have been used in any official documents. The main way the education system appears to be evaluated is through the evaluation projects set out in the Education Evaluation Plans. It appears most likely that these projects are them synthesized into the summary report about the education system set under four key themes.
National Improvement Framework for Scottish Education
In 2016 the Scottish government published a National Improvement Framework for Scottish Education identifying four key priorities and six key drivers for improvement.
The four key priorities are:
- improvement in attainment, particularly in literacy and numeracy
- closing the attainment gap between the most and least disadvantaged children
- improvement in children and young people’s health and wellbeing
- improvement in employability skills and sustained, positive school leaver destinations for all young people.
The six key drivers for improvement are:
- school leadership
- teacher professionalism
- parental engagement
- assessment of children’s progress
- school improvement
- performance information.
In 2016 the government produced an evidence report in relation to the National Improvement Framework. It is an overview of what is known about the Scottish education system and provides information about the context in which the children and young people learn. It synthesises evidence from a wide range of sources with the goal to learn from good practice and produce plans for improvement where needed. Figure 14 shows the way they visualize the drivers of improvement within the framework.
This report brings together information on pupils’ achievement, attainment, and health and wellbeing from a range of existing data sources, such as the new Achievement of Curriculum for Excellence Levels return (based on teacher professional judgements), Scottish Survey of Literacy and Numeracy (SSLN), Pupil Census, Teacher Census, Attendance, Absence and Exclusions, Growing Up in Scotland, 27-30 month Child Health Reviews, the OECD’s Programme for International Student Assessment (PISA), Scottish Health Survey, the Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS) and Her Majesty's Inspectorate of Education inspections.
The Education Act 2016 requires Ministers to review the National Improvement Framework and publish a plan on an annual basis. These National Improvement Framework plans appear to have been created every year since 2016. They give an update on education but also continue to focus on the drivers of improvement, updating the available evidence and providing detailed improvement activities under each driver. There is a summary of the ongoing and completed activities from the previous year’s plan.
The 2021 plan states certain sources of information were unavailable due to disruptions caused by Covid-19. It suggests the national plan is, therefore, a summation of what schools across Scotland have told us they want to focus on, informed by local consultation and evidence. The plan has also been informed by the data in the NIF Interactive Evidence Report, evidence from school inspections, and local authority self-evaluation reports, and recommendations and advice from the International Council of Education Advisers.
In addition to these annual plans which report on progress, the National Improvement Framework is referenced in the Education Scotland Annual Report, giving an update on performance in many areas including National Improvement.
In summary, Scotland has a clear framework for evaluating and monitoring their education system. They use a range of sources to inform their reports. Their system seems a lot more linked up than other systems and various different reports reference the existence of each other, showing they are used and functioning as intended.
A Framework for Statistics on Learning and Education in Canada
In 2008, there was a joint declaration between provincial and territorial Ministers of education to use the Learn Canada 2020 Framework to enhance Canada’s education systems, learning opportunities, and overall education outcomes. This framework identified four pillars of lifelong learning:
- early childhood learning and development
- elementary to high school systems
- postsecondary education
- adult learning and skills development.
The framework also listed the following key priorities and activity areas.
- Literacy: Raise the literacy levels of Canadians.
- Aboriginal Education: Eliminate the gap in academic achievement and graduation rates between Aboriginal and non-Aboriginal students.
- Postsecondary Capacity: Enhance and stabilize the long-term capacity of postsecondary systems to meet the training and learning needs of all Canadians seeking higher education learning opportunities.
- Education for Sustainable Development: Raise students’ awareness and encourage them to become actively engaged in working for a sustainable society.
- International and National Representation: Speak effectively and consistently for education and learning in Canada in both pan-Canadian and international settings.
- Official Languages: Promote and implement support programs for minority-language education and second-language programs that are among the most comprehensive in the world.
- Learning Assessment Programs and Performance Indicators: Support the implementation of national and international learning assessment programs and performance indicators for education systems.
- Education Data and Research Strategy: Create comprehensive, long-term strategies to collect, analyze, and disseminate nationally and internationally comparable data and research.
The Canadian Education Statistics Council created a strategic plan for 2010-2020 to assist them in meeting the goals of Learn Canada 2020. To help achieve the last key activity area, Education Data and Research Strategy, the Canadian Education Statistics Council created A Framework for Statistics on Learning and Education in Canada.
The Framework was designed as a tool to assist with deciding the information priorities of the Canadian Education Statistics Program. It is a way of organising the information within the learning and education environment in Canada. It highlights the relevance of information along with the relationships between information. In addition, the framework addresses reporting to the Organization for Economic Cooperation and Development (OECD) on education and learning. It covers not only the pillars of learning (early childhood, elementary-secondary, postsecondary, adult learning) but also the information levels (learner, provider, jurisdiction), and the dynamic elements (inputs, processes, outputs, and outcomes).
As an appendix to the general framework there is a comprehensively populated framework specific to the Canadian context at the time. It is made up of four matrices, one for each pillar of learning. This is a summary of the information about the education system at the time. The matrices identify the existing sources of information for each dynamic element, as well as providing limitations of the sources, but do not provide details on the technical or procedural aspects of collecting or reporting information. These are all details to be worked out in the plan to implement the framework. There are a wide range of sources suggested here for ways to gather this information.
There is no apparent evidence of this framework being used in any annual education report produced by the provinces or anyone using it for evaluating the education system at any level. The Learn Canada 2020 framework also doesn’t appear to be referred to in the annual reports either.
Council of Ministers of Education, Canada learning assessment programs
The aim of national and international large-scale assessments is to make information about academic achievement available to policy makers, administrators, teachers, and researchers so they can gain insight about how the education system functions and identify areas for improvement. Such assessment is not designed to provide academic achievement information at the student, school, or school-board level.
The Council of Ministers of Education, Canada (CMEC) list the learning assessment programs they are part of the:
- Pan-Canadian Assessment Program (PCAP)
- Programme for International Student Assessment (PISA)
- Programme for the International Assessment of Adult Competencies (PIAAC)
- International Computer and Information Literacy Study (ICILS)
- Teacher Education and Development Study in Mathematics (TEDS-M)
- Progress in International Reading Literacy Study (PIRLS)
- Trends in International Mathematics and Science Study (TIMSS).
Looking at the PCAP more closely, it was developed in 2007 by the provinces and territories, coordinated by the CMEC and built on its predecessor, the School Achievement Indicators Program. It assesses the skills and knowledge of Grade 8 (Secondary II in Quebec) students in reading, mathematics, and science on a three-year cycle. It is to be used primarily by provincial/territorial ministries of education to assess their education systems. From 2019 the PCAP was conducted online, being responsive to both technological developments and special education needs students.
Education data and research
The Council of Ministers of Education, Canada along with Statistics Canada produce yearly reports called Education Indicators in Canada: An International Perspective. These reports are produced as part of the Pan-Canadian Education Indicators Program (PCEIP). They present data for the provinces and territories allowing comparisons between them, and with data for OECD countries.
The main goal of the PCEIP is to provide statistics that give insight about education and learning in Canada with the aim of supporting evidence-based policy making. PCEIP produced its first set of education indicators in 1996 and has been publishing every year since.
There are several ways in which the Canadian education system can be evaluated but there doesn’t seem to be any coherent links between these. None of the reports above make references to any of the other reports mentioned and make it hard to piece together how they come together in informing the overall education system. Both the PCAP and the PCEIP appear to provide data on how provinces perform compared to each other but there is no cross-reference between these reports.
The Indian education system is one of the largest in the world. The Department of School Education and Literacy (DoSEL), Ministry of Education is currently focused on improving the quality of education across the system. In order to achieve this, DoSEL has designed a tool called the Performance Grading Index (PGI) to incentivise transformational change in school education.
The PGI score is derived using databases on 70 parameters across five domains. These are access, equity, governance processes, infrastructure and facilities, and learning outcomes. These data are mostly self-reported by the States and Union territories but vetted by the Centre. The total weighted score is a maximum of 1000. The data for 54 parameters come from a range of administrative data and for the remaining 16, scores from National Achievement Survey (NAS) 2017 conducted by the National Council of Educational Research and Training (NCERT) have been used. Find below an example of how the indicators are assembled for Domain 1 – Learning Outcomes and Quality.
The first PGI report was published in 2019 for the reference year 2017-18. The goal of the PGI is to motivate States and Union Territories to undertake multi-pronged interventions that improve education outcomes. The PGI can also help States and Union Territories to identify the gaps in outcomes and prioritize areas for intervention. The PGI is also a good source of best practice information for States and Union Territories to follow.
The PGI provides grades to the States and Union Territories, as opposed to ranking. This system allows multiple States and Union Territories to sit at the same level. It removes the requirement for one State and Union Territory to fall in position before another can improve. The PGI stems from the rationale that there must be constant monitoring of inputs, outputs and outcomes, and fast course correction in order to create an efficient, inclusive, and equitable school education system.
It was reported that the PGI system has been effective since it was put in place. Many States and Union Territories have made substantial improvements to their scores and grades for many of the outcome parameters in 2019-20.
The Ministry of Education, Tertiary Education, Science and Technology create an ‘Education Card’ every year that presents the key figures around their education system. They collect and present data on pre-primary, primary, secondary (general and pre-vocational), technical and vocational, education in specialized institutions as well as tertiary level education.
Below are the areas they present statistics and graphics on.
- Brief country profile
- Educational indicators
- Public expenditure on education
- Government expenditure on education by sector
- Number of schools and enrolment - pre-primary
- Number of schools and Enrolment - primary
- Number of schools and enrolment - secondary general
- Number of schools and enrolment - secondary pre-vocational
- Number of centres and enrolment in Mauritius Institute of Training & Development (MITD)
- Number of schools and enrolment in Special Education Needs (SEN)
- Enrolment in tertiary institutions by gender & type of providers
- Enrolment in tertiary institutions by level and field of study
- Examination results - Primary School Achievement Certificate (school candidates only)
- Examination results - School Certificate (school candidates only)
- Examination results - Higher School Certificate (school candidates only)
- ICT in pre-primary, primary and secondary Schools
The 2019 Education Card provides information on 2018 and 2019, allowing the reader to see any changes across this time. Figure 17 gives an example of the type of statistics presented. This report provides very basic information on the inputs and outputs in the system. It doesn’t provide any deeper evaluation or analysis. These are similar to the broad UNESCO indicators.
National Education Blueprint 2013-2025
In 2011, the Malaysian Ministry of Education conducted a review of the education system in Malaysia to improve educational outcomes. The government were responding to parental and public concerns about the Malaysian education system’s ability to prepare students for the challenges they would face in the 21st century, and keep up with rising international education standards. The National Education Blueprint is the document capturing the extensive research and public engagement undertaken to evaluate the performance of Malaysia’s current education system along with a vision of where they want it to go. The report includes 11 shifts that would be required to achieve their vision. They collected information from many sources.
The objectives of the review were to:
- understand the current performance and challenges of the Malaysian education system, with a focus on improving access to education, raising standards (quality), closing achievement gaps (equity), fostering unity amongst students, and maximising system efficiency
- establish a clear vision and aspirations for individual students and the education system over the next 13 years
- outline a comprehensive transformation programme for the system, including key changes to the Ministry which will allow it to meet new demands and rising expectations, and to ignite and support overall civil service transformation.
The National Blueprint is a long document covering:
- context and approach
- vision and aspirations
- current performance
- student learning
- teachers and school leaders
- Ministry transformation
- system structure
- delivering the road map.
The part that is evaluating the education system is the Current Performance section and it covers the following areas:
- access to education
- quality of education
- equality in education
- building unity through education
- maximising efficiency.
This section is an analysis of how students are performing over time and in comparison to other countries. It is designed to be an objective fact base to establish a baseline of how the Malaysian education system is performing. They comment that access, quality, equality, unity, and efficiency have been the priority outcomes as far back as 1956 but acknowledge that only by conducting an examination of where they currently sit in term of achieving those outcomes, can they truly move towards improvement.
Multiple perspectives were gathered from various experts and international agencies to evaluate and assess the performance of Malaysia’s education system. These included the World Bank, the United Nations Educational, Scientific, and Cultural Organisation (UNESCO), the Organisation for Economic Co-operation and Development (OECD), and six local universities. The Ministry also worked with other government agencies to ensure alignment with other public policies related to education. For example, the Ministry has worked closely with the Performance Management and Delivery Unit (PEMANDU) to develop the Government Transformation Programme (GTP)2.0 initiatives on education so they reflect the priority reforms in the Blueprint from 2013 to 2015.
In addition, over 55,000 Ministry officials, teachers, school leaders, parents, students, and members of the public across Malaysia, were engaged via interviews, focus groups, surveys, national dialogue townhalls, open days and roundtable discussions. More than 200 memorandums were submitted to the Ministry and over 3000 articles and blog posts were written on the issues raised in the Blueprint. The Ministry also appointed a 12-member Malaysian panel of experts, and a four-member international panel of experts to provide independent input into the review findings.
They have three waves to their education transformation plan over the 13 years with key outcomes noted for each wave. There doesn’t appear to be a system noted in this document about monitoring of these outcomes, however, on their website there is an annual reports section. The reports that this research could find in English appear to give updates on the progress towards the five outcome goals. Administrative data appears to be the main source of information for these annual reports but it’s not totally clear.
In summary, a wide range of sources were used to gain a baseline picture of what the Malaysia education system performance looked like across five key areas. They report on the progress on these areas each year in their annual reports. Monitoring data is main source rather than results of individual thematic evaluations that have been carried out.
Education Strategic Plan 2019-2023
In Cambodia, the education system sits under the authority of the Ministry of Education Youth and Sport MoEYS. Most of the website is not in English so it’s possible there is something else more appropriate on their website, but this is what this research could find that appears to be most relevant.
The 2019- 2023 report begins with a section on the performance of the education sector from 2014-2018, which is the timeframe for the previous Education Strategic Plan. It comments on the progress made during the plan’s implementation period. A midterm review had been conducted in 2016 (mid-way between 2014 and 2018) and the report notes the changes this review resulted in. There were two medium term education policies identified (these are in line with SGD 4).
- Policy 1: Ensure inclusive and equitable quality education and promote life-long learning opportunities for all.
- Policy 2: Ensure effective leadership and management of education officials at all levels.
There are 10 core breakthrough indicators that have been identified in relation to these policies and the progress on them is presented. This section of the report also goes through how both policy objectives are being reached in each subsector (early learning, primary etc). The report identified eight indicators in relation to the policies for the plan implementation period. The report also sets out a plan for how these will be achieved in each of the sub-sectors.
Education Road Map 2030
In 2019, the Education Road Map 2030 was produced as a strategy to improve education. Its aim is to ensure inclusive and equitable quality education and promote lifelong learning opportunities for all and defined five policy priorities, listed below.
- All girls and boys have access to quality early childhood care and education and pre-primary education, and complete free, equitable and quality basic education (primary and lower-secondary) with relevant and effective learning outcomes.
- All girls and boys complete upper-secondary education with relevant learning outcomes, and a substantial number of youth have increased access to affordable and quality technical and vocational education.
- Ensure equal access for all women and men to affordable and quality technical, vocational and tertiary education, including university.
- All youth and adults achieve literacy and numeracy, and learners in all age groups have increased life-long learning opportunities.
- Governance and management of education improves at all levels.
There is a reporting framework that organizes the policy priorities with expected results and sector indicators sitting beneath each. This framework was created to help monitor the progress against these priorities. The framework displays the 2018 baseline along with the targets for the plans covering the period 2019-23 and 2024-28 and the target for 2030.
Vision 2030 Jamaica and three-yearly Medium-Term Socio-economic Policy Frameworks
Jamaica has a document called Vision 2030 Jamaica which is a 21-year plan about how Jamaica will progress to become developed. This long-term plan:
- identifies long-term goals and outcomes
- presents national and sector strategies for achieving the outcomes
- proposes outcome indicators to measure performance over the long term
- identifies key strategies and actions for years one to three
- presents the framework for its implementation, monitoring, and evaluation.
It encompasses all aspects of becoming a developed country including a focus on education. There are four key national goals and 15 national outcomes that sit under these goals. The Jamaican education system doesn’t appear to have a specific framework for evaluating education outcomes, but it can be evaluated through the national level tracking and evaluation of the goal related to education. The following aspects focus on education.
Goal 1: Jamaicans are empowered to achieve their fullest potential
Below this goal there is a National Outcome 2: World class education and training.
Next are the National Strategies linked to the National Outcome above:
- Ensure that children zero to eight years old have access to adequate early childhood education and development programmes.
- Enable a satisfactory learning environment at the primary level.
- Ensure that the secondary school system equips school leavers to access further education, training, and/or decent work.
- Accelerate the process of creating and implementing a standards-driven and outcomes-based education system.
- Develop and establish financing and management mechanisms for schools.
- Ensure a physical environment in all schools that is safe and conducive to learning at all levels of the school system.
- Ensure that adequate and high-quality tertiary education is provided with an emphasis on the interface with work and school.
- Expand mechanisms to provide access to education and training for all, including unattached youth.
- Promote a culture of learning among the general populace.
- Establish a National Qualification Framework.
- Strengthen mechanisms to align training with demands of the labour market.
Within the Vision 2030 Jamaica document National Outcome Indicators are proposed for each National Outcome, including Outcome #2: World class education and training.
As part of the implementation plan there are three-yearly medium-term socio-economic policy frameworks produced. These report on the specific indicators and targets which can be used for results-based monitoring and evaluation. The 2018-2021 plan reports on each of the national strategies identifying the sector strategies that fall under each national strategy. It highlights the actions to be taken under the sector strategy and the main implementing entity responsible or the action.
First Quarter Performance report
In addition to this, the Jamaican Ministry of Education, Youth and Information produce quarterly performance reports (only available report was April-June 2020). There is however very little overlap between this document and the Vision or the Medium-Term goal report.
In summary, Jamaica appear to have a clear plan for evaluating and monitoring their education system progress towards identified priorities. However, it is it mostly reliant on administrative data to assess performance. They have a plan taking them through to 2030 which outlines most of their goals and in addition they have three-yearly reporting frameworks that specify more details.
Council of Ministers of Education. (2008). Learn Canada 2020: Joint Declaration Provincial And Territorial Ministers Of Education. https://www.cmec.ca/Publications/Lists/Publications/Attachments/187/CMEC-2020-DECLARATION.en.pdf
Council of Ministers of Education. (n.d). Learning Assessment Programs. https://www.cmec.ca/131/Learning_Assessment_Programs.html
Council of Ministers of Education. (n.d). The Pan-Canadian Assessment Program. https://www.cmec.ca/240/Pan-Canadian_Assessment_Program_(PCAP).html
Council of Ministers of Education. (n.d). The Pan-Canadian Education Indicators Program (PCEIP). https://www.cmec.ca/259/Programs-and-Initiatives/Education-Data--Research/Indicators/index.html
Council of Ministers of Education & Statistics Canada. (2010). A framework for statistics on learning and education in Canada. http://www.cmec.ca/Publications/Lists/Publications/Attachments/257/cesc-data-framework-sept2010.pdf
Education Scotland. (2019). Annual Report and Accounts 2018-19. https://education.gov.scot/media/jpran3nk/es-annual-accounts2018-19.pdf
Finnish Education Evaluation Centre. (2016). National Plan For Education Evaluation 2016–2019. https://karvi.fi/app/uploads/2016/06/National-Plan-for-Education-Evaluations-2016-2019.pdf
Finnish Education Evaluation Centre. (2019). Evaluating the state of the Finnish education system: Results of the Finnish education evaluation. https://karvi.fi/app/uploads/2019/10/KARVI_T1419.pdf
Government of India. (2021). Performance Grading Index (PGI) 2019-20, For States and Union Territories, Department of School Education and Literacy, MOE pgi_2019_20_en.pdf (education.gov.in)
Ministry of Education Malaysia. (2013). Malaysia Education Blueprint 2013 - 2025 (Preschool To Post-Secondary Education). https://www.ilo.org/dyn/youthpol/en/equest.fileutils.dochandle?p_uploaded_file_id=406
Ministry of Education Malaysia. (2018). 2018 Annual Report: Malaysia Education Blueprint 2013 – 2025. https://www.padu.edu.my/wp-content/uploads/2019/07/AR2018-Eng.pdf
Ministry of Education, Tertiary Education, Science and Technology. (2019). Education Card 2019. https://education.govmu.org/Documents/downloads/Documents/Statistics/Education%20%20Card%20%202019.pdf
The Ministry of Education, Youth and Sport. (2014). Education Strategic Plan 2014-2018. https://www.globalpartnership.org/sites/default/files/2015_02_cambodia_education_sector_plan.pdf
The Ministry of Education, Youth and Sport. (2016). Mid-Term Review Report in 2016 of the Education Strategic Plan 2014-2018 and Projection to 2020. https://www.globalpartnership.org/sites/default/files/2016-11-cambodia-mid-term-review-education-sector-plan.pdf
The Ministry of Education, Youth and Sport. (2019). Cambodia’s Education 2030 Roadmap Sustainable Development. https://www.moeys.gov.kh/index.php/en/planning/2901.html#.YK8TQ4fiuUk
The Ministry of Education, Youth and Sport. (2019). Education Strategic Plan 2019-2023. https://www.globalpartnership.org/sites/default/files/2019-10-education_sector_plan-cambodia.pdf
Ministry of Education, Youth & Information. (2020). First Quarter Performance Report April -June 2020. https://drive.google.com/drive/folders/1yQ05bLGe5euMyBNF0x1ZTClrbuI_kuSi
National Board of Education. (1999). A Framework for Evaluating Educational Outcomes in Finland. https://karvi.fi/app/uploads/2014/09/OPH_0499.pdf
OECD. (2019). Education Policy Outlook 2019. https://www.oecd-ilibrary.org/content/publication/2b8ad56e-en
OECD. (2020). Education at a Glance 2020: OECD Indicators. https://www.oecd-ilibrary.org/content/publication/69096873-en
Planning Institute of Jamaica. (2018). Vision 2030 Jamaica – National development plan: medium term socio-economic policy framework 2018 – 2021 http://www.vision2030.gov.jm/Portals/0/MTF/MTF%202018%20-%202021%20-%20%20March%202019.pdf
Planning Institute of Jamaica. (2009). Vision 2030 Jamaica: National Development Plan. http://www.vision2030.gov.jm/Portals/0/NDP/Vision%202030%20Jamaica%20NDP%20Full%20No%20Cover%20(web).pdf
The Scottish Government. (2016). National Improvement Framework For Scottish Education 2016 Evidence Report. https://www.gov.scot/publications/national-improvement-framework-scottish-education-2016-evidence-report/
The Scottish Government. (2020). Achieving Excellence and Equity 2021 National Improvement Framework and Improvement Plan https://www.gov.scot/publications/2021-national-improvement-framework-improvement-plan/
UNESCO (2015). Incheon Declaration and SDG4 – Education 2030 Framework for Action. https://iite.unesco.org/publications/education-2030-incheon-declaration-framework-action-towards-inclusive-equitable-quality-education-lifelong-learning/
UNESCO (2020). Global Education Monitoring Report 2020: Inclusion and education: All means all. https://gem-report-2020.unesco.org/
UNICEF. (2020). MISC-EAGLE: Implementation Manual 2020: Leveraging Data and Building Capacity. https://data.unicef.org/wp-content/uploads/2019/07/MICS-EAGLE-manual-FINAL-10March.pdf
UNICEF. (n.d). The MICS-EAGLE initiative: Leveraging data and building capacity. https://data.unicef.org/wp-content/uploads/2019/07/MICS-EAGLE-Initiative-Brochure_Revised_20210118.pdf
World Bank. (2013). The What, Why and How of the Systems Approach for Better Education Results (SABER). http://wbgfiles.worldbank.org/documents/hdn/ed/saber/supporting_doc/Background/SABER_Overview_Paper.pdf
World Bank (2021). DRAFT Global Education Policy Dashboard (Provided in confidence by World Bank).
World Bank. (n.d.). About SABER and Methodology. Systems Approach for Better Education Results (SABER) (worldbank.org)