Schools: Evaluation for Improvement Approach: implementation case studies

Acknowledgements

I would like to acknowledge the collegial support and guidance of Dr Ro Parsons and Sandra Collins. 

Special thanks to the senior leaders of the six schools who generously shared their time speaking with me about their experiences. While it was not possible to conduct in-person case studies with each school due to Covid-19, every school was extremely supportive of the evaluation process.

Evaluation Summary

About the Case Studies

• Case studies of six schools were undertaken to understand and document school experiences with ERO’s Schools: Evaluation for Improvement approach. All schools were at an early phase of the evaluation process and focused their comments on their experience to date.
• Selection of schools was based on principles of diversity in the size of the school, in the type of school, and in location. Two schools were selected from each region.

Strengths of the Evaluation Approach

• All six schools were positive about the philosophy of collaboration and partnership ways of working that underpinned the approach to school evaluation. They believed that a high trust model was the most appropriate way for ERO to work with schools to support evaluation for improvement. From their perspective a strong collaborative base will enable ERO to get a more accurate perspective about what is working and not working across schools in New Zealand. 
• School leaders contrasted the previous ERO evaluation approach with the new approach. They valued ERO’s emphasis on providing ongoing evaluation support, rather than relying on a one-off or episodic evaluation of the school. The previous school evaluation approach was seen as narrow, superficial, and judgemental and based on limited understanding of the school context.
• Schools see evaluation as important, and a key responsibility of schools for learning and for accountability. Five of the six schools believed their knowledge and skills in internal evaluation will be strengthened through the new approach. Some of the internal school stakeholders whose views were canvassed in this external evaluation expressed genuine excitement about learning more about evaluation through working with their evaluation partner.
• The evaluation focus and/or evaluation plan in five of the six schools aligned well with the school’s strategic direction, adding value and relevance to the evaluation process, and avoiding duplication of effort. This alignment was seen to enhance the usefulness of the evaluation process and ERO’s work within the school.
• Alignment of the evaluation plan with the school’s existing priorities as captured in their strategic plan or charter avoids duplication of effort and strengthens the relevance of formal evaluation mechanisms.
• Schools appreciated the evaluation partner working alongside them to develop an appropriate evaluation focus for the school. The development of a trusting, collegial relationship with the evaluation partner appeared to be a critical condition in supporting an effective evaluation planning process. The relational capabilities of the evaluation partner were regarded as important as their educational knowledge or evaluation skills. 
• Co-design was seen to be a central feature of the approach. While schools were not yet clear about expectations for external reports, they indicated that reports would be developed through a process of co-design, rather than prepared in isolation by ERO.
• School representatives found it difficult to nominate formal improvements to the approach as they were still in the early phases of the evaluation. They indicated that they will better be able to assess the approach after they have progressed through one entire evaluation cycle.
• The strengths of the new approach identified in the survey findings from 66 of the 75 principals involved in Group 1 implementation were reinforced in the deeper-dive case studies.  

Concerns about the Approach

• School representatives highlighted the importance of the evaluation partner gaining a good understanding of the school context, and the school’s strategic priorities to shape a useful evaluation plan with the school.
• Internal stakeholders within two schools noted the importance of continuity in the relationship with the evaluation partner to the success of the approach.
• Some of the principals questioned the viability of maintaining collaborative ways of working as the evaluation partner takes on a greater number of schools.
• A small number of school stakeholders noted a concern about continuity of process and understanding of the school context if their evaluation partner were to leave their position.

Sharing Lessons Learnt to inform school improvement

• Representatives of five of the six schools indicated it was important for ERO to share learnings and direct examples about the attributes of good internal evaluation based on their work with schools. Schools are keen to have more guidance from ERO about good practice to support them in improvement. It is clear they understood this to be a feature of the new approach. ERO possesses a wealth of information about what works and what doesn’t work in particular contexts, which could be usefully shared to improve the effectiveness and efficiency of improvement efforts in schools.

Key Questions for consideration by ERO

The synthesis of the case studies reveals five key questions for deliberation by ERO, as presented below.

1. How will ERO manage school expectation within existing resource constraints?
2. How can evaluative capacity be extended across schools and in classrooms?
3. How will ERO align its work with other partners who are also working with schools to progress improvement?
4. How will ERO build and maintain its internal capability in evaluation?
5. How will ERO support schools with internal evaluation and support external accountability?
6. How will ERO know it made a difference at the school and system level?

These questions are elaborated in the synthesis section of this report with recommendations for consideration by ERO.

Each section of this report builds an understanding of school experience and the potential implications of each of the questions for ERO.

Overview of this Report

This report represents the final phase in the external evaluation of the initial implementation of the approach with the initial group of 75 schools. The intended audience for this report is ERO senior leadership team. It is intended that this report will be used formatively to consider opportunities for refinement and improvement of the approach.

Section 1 of this report provides an overview of the Schools: Evaluation for Improvement Approach.

Section 2 presents a description of the case study methodology and the schools that participated in the case study of implementation.

Section 3 describes the current status of the evaluation process in the schools and identified areas of focus.

Section 4 is the major section of this report and provides key messages identified from the cross-case analysis of the participating schools.

Embedded case profiles from each school in the case study are included to add depth and detail to key messages. Reading the profiles in conjunction with the presentation of the cross-case analysis will allow the reader to understand the experience of individual schools during the initial implementation of the approach. Quotes from stakeholders interviewed across the six schools are integrated into each section to evidence key claims.

Section 5 presents a series of key questions for consideration by ERO, their implications and recommendations for consideration by ERO.  

Section 1:  Introduction

There are over 2500 schools in New Zealand providing education to approximately 850,000 students. Most schools are self-managing state or state-integrated schools, including 112 full Māori-medium kura and 83 mixed Māori language kura.

The Education Review Office (ERO) is the Government’s external education evaluation agency. ERO’s internal and external function encompasses: accountability (including compliance with regulatory requirements), support for improvement and knowledge generation. ERO contributes to the knowledge base about what works in which schools and community contexts to support equity and excellence for all learners. At a system level, ERO conducts research and evaluation to inform priorities and actions for change.

While most schools in New Zealand are providing positive educational experiences for students, inequities remain particularly for Māori and Pacific learners. A central point of inquiry in every school evaluation is the extent to which the school is addressing the needs of Māori and Pacific students, and the impact on equity in terms of learning outcomes.

In response to recommendations in the Review of Tomorrow’s Schools in 2019, ERO shifted its external evaluation approach to a more participatory, collaborative model with increased emphasis on evaluation capacity building and school improvement. Central to the new approach is a focus on ensuring that all schools are on an improvement trajectory.

The structural and operational aspects of the Schools: Evaluation for Improvement Approach were developed in 2020.  

1.1 Schools: Evaluation for Improvement Approach

ERO undertook a research and development process to track alongside the initial implementation of the approach with 75 schools across the country. Fifteen evaluation partners were each allocated 5-6 schools to work through the phases of the evaluation process.

For the evaluation partners the approach was new and reflected a move away from a summative judgement of the school’s performance at the conclusion of a three- or four-day on-site process to an improvement-oriented approach based on collaboration, embedded within the school’s own context that evolve over the course of an improvement cycle.

The Schools: Evaluation for Improvement Approach in a nutshell

• A shift from event-based external reviews to an evaluation approach that supports continuous improvement
• A shift to a more participatory, collaborative approach to external evaluation built on an understanding of the school context, culture and needs.
• A shift to a more adaptive and responsive evaluation approach.

1.2 The Principles of Practice

ERO developed a set of principles to guide educational evaluation within schools. They are based on improvement-oriented evaluation theory, evidence and practice. The eight principles of practice articulate the characteristics of effective evaluation that should be observed in schools and supported through the Schools: Evaluation for Improvement Approach.

The eight principles are:

1. Ko te Tamaiti te Pūtake o te Kaupapa: The child, the heart of the matter - a focus on the learner and equity and excellence in outcomes
2. The integration of internal and external evaluation
3. A participatory and collaborative process
4. A culturally and contextually responsive approach
5. Technical rigour in design, data collection, analysis, synthesis and reasoning
6. Promotion of evaluation use
7. Developing evaluation capacity
8. Promoting external accountability and strengthening internal accountability.

Section 2: External Evaluation of the Initial implementation with Schools

An external evaluation was integrated into ERO’s Research and Development process to support implementation of the Schools: Evaluation for Improvement Approach. The purpose of the evaluation was formative to document experiences to date and identify opportunities for improvement.

The external evaluation consisted of three phases.

• Phase 1: Experiences of Evaluation Partners/Review Officers with initial implementation (completed in January 2021)
• Phase 2: Key internal stakeholder views of the approach (completed in February-March 2021)
• Phase 3: A survey of all principals involved in the initial implementation (75 schools) (completed in June 2021), and
• Phase 3a: Conduct of six case studies of schools involved in the first implementation of the approach (conducted August to December[1] 2021 and completed in February 2022).

Reports were produced from each phase to guide improvement. This third report focuses on presentation of key findings from the case studies of six schools (Phase 3a) involved in the initial implementation.

2.1 The Case Studies

The objectives of the case studies of school experience were to:

1. Develop a deeper understanding of how the approach worked in practice in the six schools
2. Identify conditions that facilitated or inhibited implementation of the approach in the schools, and
3. Identify implications for ERO and opportunities for improvement.

Each region was asked to nominate 4 to 6 schools to inform case selection for this external evaluation. Regions were asked to consider elements of diversity, such as school type (primary, intermediate, contributing, secondary), level of experience of the principal, and school size. Regions were also asked to consider inclusion of schools where partnership approaches appeared to be working well and those where the approach was not working so well.  

Two schools were selected from each region using principles of variation as criteria for case selection.

2.2 Case Study Schools

Once selection of the six schools had been made, the external evaluator made contact with the Evaluation Partners/Review Officers assigned to each of the six schools. The external evaluator reiterated the purpose of the case study and asked to share any additional background information that evaluation partners were able to share with the external evaluator (for example, the current draft of the evaluation plan), that may be helpful in supporting understanding of the school context.

2.3 Characteristics of the Case Study Schools

The characteristics of each of the schools in the case study component of the external evaluation are outlined in Table 1.

Table 1: Characteristics of Schools

Region: Te Tai Raki (Northern)

School Name: Tauraroa Area School

General Characteristics: Area School, School roll is 500

Students: 26% Māori, including whānau roopu classes

 

School Name: Kaitaia Intermediate School

General Characteristics: Intermediate School, School roll is 261

Students: Predominantly Māori student population

 

Region: Te Tai Pūtahi Nui (Central)

School Name: Kowhai School, Hastings

General Characteristics: Special school with a base school and 4 satellites. High proportion of students on Ongoing Resourcing Scheme

Students: 43% Māori students, 11% Pacific heritage, 46% Pakeha

 

School Name: Waihi East School, Waihi

General Characteristics: Contributing school roll is 200. Early career principal (2 years in role)

Students: 31% Māori students

 

Region: Te Tai Tini (Southern)

School Name: Parklands School, Motueka

General Characteristics: Primary school Years 1-8, School roll is 206. Early career principal

Students: 40% Māori students, 5% Pacific heritage

 

School Name: St Bernard’s College, Lower Hutt

General Characteristics: Boys school offering years 7-13 with additional intake at year 9

Students: 26% Māori students, 20% Pacific heritage, 44% Pakeha

 

2.4 Initiating Contact with Schools

Principals of the six schools were contacted, the purpose of the case studies explained, and a date scheduled for the school visit with the principal. Case studies were originally scheduled face to face, but due to Covid restrictions the two case studies in Northland had to be conducted with key stakeholders via individual or small group interviews on the Zoom platform. Data collection and analysis was undertaken between August and November 2021.

The focus of the case studies was explicitly on schools’ experiences with the Schools: Evaluation for Improvement Approach. Questions of focus were designed to obtain an understanding of

• the school context
• experiences with previous ERO review processes
• principles underpinning the new approach
• experiences of the new approach in practice, and
• suggestions for improvement of the new approach.

The findings presented in this report are based on semi-structured interviews with the principal, and the deputy principal. In three schools’ interviews with a Board of Trustees representative, teachers and small groups of students were also conducted[2]. 

Each school was asked to provide any additional secondary documentation (for example, findings from consultations with whānau, results of surveys, the school charter) to assist the external evaluator to understand the school context and internal evaluation conducted within the school. 

2.5 The Nature of Evidence in the Case Studies

The case studies are illustrative. While the six schools are diverse, their views and experiences of the Schools: Evaluation for Improvement Approach have a number of common themes about the merit of the new approach. While no claims are made about the generalisability of findings, lessons learned from the experiences shared by these six schools may be transferable across a wider range of schools. 

It should be noted that one of the schools represents an outlier case in terms of experience of the approach. They were disappointed with the new approach and expressed reservations about the value of the approach to the school. This school’s experience provides insights into conditions that support or inhibit the success of ERO’s work with schools.

The limitations of the case studies are that there was no scope to triangulate interpretations with the evaluation partners working with these schools. Their perspective would have provided further additional insights and enabled some initial verification of information (for example, school context and key dates). 

In the first phase of the external evaluation all ERO’s evaluation partners involved in the initial implementation were asked for feedback about the approach, but the focus was on their experience overall; feedback about their work with individual schools was not sought.

2.6 Analysis and Synthesis

Interviews with school stakeholders were digitally recorded and transcribed. A reflexive thematic analysis (Braun & Clarke, 2022) was undertaken of each individual school with the school experience of the new approach being the central organising domain.

Data generated from interviews was organised according to interview questions. In this way the external evaluator was able to assess similarities and differences in views or perspectives in each school. A comprehensive case report of each school was then developed. These ranged in length from 10-12 pages.

The cross-case analysis then commenced. Each school was compared and contrasted with other schools in terms of the questions of focus. This cross-case analysis contributed to the development of analytic domains (identified as key messages highlighted in section 4 of this report). 

A case profile of one school was also assigned to elaborate the key message and provide richer insights into school experience on the ground. While several schools expressed very similar messages about the value of a collaborative platform and the skills of their evaluation partner, a decision was made about the best fit of each profile to the central messages.

Case profiles were returned to the principal of each school for review. Minor factual errors were corrected (such as length of time a principal had worked at the school), but no substantive changes were made to the case profile.

Approval was also sought for inclusion of photographs taken during the case study visit or retrieved from documentation provided by the school. This final report includes the approved case profiles including some photographs to bring the profiles to life.

Section 3: Overview of Schools: Evaluation for Improvement Approach

The Evaluation for Improvement Approach to school evaluation represents a shift in how ERO’s role and function is operationalised in practice in the school context.

Evaluation Partners/Review Officers and schools work together during the evaluation process. While each part of the evaluation process includes some core tasks, how the process plays out varies in each context. Evaluation tasks may overlap, and there may be a need to return to initial discussions to reinforce key aspects (for example, to re-engage and re-negotiate evaluation parameters).  

This more collaborative way of working with schools is designed to foster increased ownership by schools of evaluation processes and to generate evidence for improvement plans and action. Schools are encouraged to use that evidence to inform decisions and activity within the school that will ultimately lead to better outcomes for all students.  

Schools may build from one evaluation to another to understand the effectiveness of educational strategies or to generate an evidence base about what is required to improve outcomes for students within their school. Evaluation of the results of improvement actions is a critical part of the school evaluation process. 

The new approach means that ERO is not only focused on evaluation capability but also supporting schools with planning and implementation activities. The knowledge and skills of evaluation partners required for this role include,

• knowledge about the conditions and practices or school improvement
• Understanding and commitment to the implications of Te Tiriti o Waitangi
• knowledge of educational theory and practice
• knowledge of evaluation theory and technical requirements for robust evaluation, and
• relational skills in working with diverse school stakeholders.

Evaluation partners are supported by teams and an infrastructure that supports them in their work with schools.

3.1 Case Study Schools – Progress Across Evaluation Phases

All schools involved in the case studies were at relatively early phases of the evaluation process.

The 15 evaluation partners supporting the first group of 75 schools had begun to work with schools between September-November 2020.

By September 2021 four of the six schools had agreed an evaluation focus and a draft evaluation plan had been prepared. The two remaining case study schools were still in discussions with their evaluation partner about the evaluation focus and plan. The research and development process that underpinned the initial implementation with the first group of schools meant that evaluation partners were involved in shaping the practical implications of each phase as they worked with the schools. Evaluation partners were also involved in professional learning and team-based discussions with colleagues in their region and across the country, most of which occurred over Zoom.

It appears that evaluation partners spent a lot of initial time with schools communicating the rationale and philosophy of the new way of working with schools. They worked hard to build a collaborative foundation. The time taken in the initial set up of the evaluation process may also reflect some uncertainty by the evaluation partners of specific evaluation requirements or tasks, due to the developmental nature of the process.

While it was anticipated that implementation of the phases would progress more quickly, discussions and meetings were hindered by Covid-19 uncertainty and school lockdowns.

A summary of each school’s agreed evaluation focus is presented in Table 2 on the following page. The evaluation focus statement and key evaluation questions were taken directly from draft evaluation plans or working documents.

Table 2: Current Status of School Evaluation – October 2021

School: Tauraroa Area School

Status: Designing completed. Draft Evaluation Plan

Evaluation Focus:

Strengthening teacher’s assessment data literacy and rationalising assessment to provide reliable and timely assessment information

Evaluation Questions (as framed in the evaluation plan):

1. How effective is our assessment in supporting teaching and learning that contributes to engagement, equity, and excellence for all learners?
2. How effective are we in responding to evaluation and assessment for improving outcomes for all students?

 

School: Kaitaia Intermediate School

Status: Exploring and focusing

Evaluation Focus:

How effectively are school processes and teaching practice raising student achievement and building a learning focused culture that holistically promotes the wellbeing, language, culture, and identity of all students?

Evaluation Questions (as framed in the evaluation plan):

Not applicable

 

School: Kowhai School

Status: Designing

Evaluation Focus:

To strengthen internal evaluation and the school’s ability to improve

Evaluation Questions (as framed in the evaluation plan):

Evaluate the effectiveness of Kowhai TEC transition processes and practices and how they are supporting students to successfully transition from school to community (including a number of evaluation questions).

 

School: Waihi East School

Status: Designing

Evaluation Focus:

How effective are the use of the school’s values in the curriculum, supporting equity and excellence, wellbeing, and improved student outcomes?

Evaluation Questions (as framed in the evaluation plan):

1. To what extent are the school values visible in student learning?
2. In what ways do the school values support students in their learning at school and at home?
3. How well do the values support the graduate profile outcomes?
4. How does the deliberate teaching of school value impact on wellbeing?

 

School: St Bernard’s College

Status: Exploring and focusing

Evaluation Focus:

Equity and excellence for Māori and all other priority learners: How do we know?

Evaluation Questions (as framed in the evaluation plan):

1. Developing an evaluative practice model to support effective relationships for learning between teachers and students.
2. Building evaluative practice to strengthen effective relationships for learning between teachers and students.
3. Impact of ongoing evaluation practice that supports effective relationships for learning between teachers and students.

 

School: Parklands School

Status: Designing

Evaluation Focus:

Responsive curriculum and opportunity to learn: Reading in years 4 to 6.

Evaluation Questions (as framed in the evaluation plan):

The purpose of the evaluation is to explore the effectiveness of reading practices in years 4-6 (including effective teaching and assessment practices for reading, which promote student engagement, high expectations and continuity of learning.

Section 4: Findings

4.1 Views and Experiences of the Approach

As all schools were still at early phases in planning and designing the evaluation, they were not in a position to assess the approach as a whole at the time data for the case studies was collected. To date, the experiences in five of the six schools has been positive[3]. 

Principals and senior leaders welcomed the philosophy of participation and collaboration that underpinned the new approach. While ERO had communicated the elements of the approach publicly, some principals and teachers remained wary about how it would work in practice. Schools are frequently exposed to new models, frameworks, and approaches from a range of sources, and for some school stakeholders these frameworks do not support meaningful change; they are perceived as ‘window dressing’ and there is an expectation that nothing may change in practice.

At the beginning of most evaluations, stakeholders may feel uneasy because they do not have full oversight of the evaluation process, or may be anxious about the potential burden of the evaluation. Historically, ERO evaluation processes were associated with external judgement of the school. In the new approach the school and the evaluation partner make decisions together about the areas of focus of improvement efforts, given an understanding of the school context and strategic plan, the purpose and scope of the evaluation, the roles and responsibilities of each party and how evidence will be used for planning and evaluation of improvement actions.

Once the schools met with their evaluation partner some initial questions or concerns about the new approach were addressed. Through discussion the school developed an understanding of the practical implications of the approach. School representatives considered that the evaluation partners described the philosophy behind the new approach clearly.

When asked what terms they associated with the approach school representatives used terms such as, ‘collaborative’, ‘co-design’ and ‘partnership for improvement’ to describe the key features of the new approach. Most interviewees anticipated that the partnership established with the evaluation partner will ensure that the evaluation process remains collaborative throughout the evaluation cycle, from design, data collection, analysis, synthesis through to reporting.

4.2 The Value of a Collaborative Platform for School Evaluation

While it has become common in the educational evaluation literature to recommend collaborative approaches, the theory needs to translate into practice. This requires a number of questions to be addressed:

• What does collaboration mean in the context of ERO working with schools to support evaluation for improvement?
• Who provides the leadership for collaboration? How are collaborative processes maintained?
• What are the limits of collaborative practices for co-design and partnership throughout the evaluation process, including reporting?

Schools were unanimous in their support for ERO’s move to a more collaborative approach to school evaluation. Collaborative ways of working were seen to have more integrity; the school works with ERO over time on evaluation processes that will inform improvement. 

One of the principals referred to the shift they had already observed in the evaluation process:

“This (new approach) does not seem like a tick box thing. The school is not being scored at a particular level. It’s not judgement focused or performance management. It’s working together for improvement through open channels of communication and trust.  We don’t feel like we need to put on a show for our reviewer.”

While schools were not sure about the parameters of the approach in its entirety (particularly data collection and reporting requirements) or the implications for the school, they were confident that the collaborative platform established with the evaluation partner will allow understandings of the scope of the approach to unfold over time. As one principal put it,

“I would imagine that if we are working from a truly collaborative basis that there will be no surprises. We will be working it out together as we go along.”

Schools indicated that the prior review model often felt rushed. It was a snapshot judgement of the school by an external agency with a limited knowledge of the school context. Interviewees indicated that the new approach will be of value to the school, and more value to ERO because of the collaborative base. One principal explained the differences that a collaborative foundation makes to the potential for greater transparency and learning:

“I think it’s far more of an open relationship and allows a greater look into the depths of how the school’s working, what we’re working on, how we’re changing things, where the gaps are, how we’re planning to address them. Before it was this tiny snapshot that they would have during the week or so that they would come in, and the whole school felt the stress and pressure. That is nothing to do with the reviewers that came in, they were always very nice, but it was just that stigma that was always attached to ERO: ‘They’re here, be on your best behaviour, make sure you’ve got everything up to date,’ and you’d see people crossing the Ts and dotting the Is. Whereas this new model is far more relaxed, people feel comfortable to have conversations and to talk about actually what’s genuinely going well, as opposed to, “What can we think of that they’re going to want to hear, that we could show evidence of?”

The following case profile illustrates the value of the collaborative platform to school evaluation in increasing the utility and transparency of evaluation within the school. The case profile begins with a description of the school context and important areas of focus for the school, and then describes their experience of the benefits of EROs collaborative approach to evaluation to date.

Comments on the evaluation process were based on reflections to the initial meetings with the evaluation partner and discussion of the approach, the development of evaluation focus and the drafting of the evaluation plan.

Case Profile: Waihi East School - A collaborative platform for evaluation

School Context

Waihi East School was established in 1907. Waihi East is a co-educational contributing school catering for children in the 5 to 11 year age range (Years 1- 6). At the end of Year six transfer to Waihi College (Years 7-13). The school roll is approximately 200 students and 32% of the students identify as Māori. Prior to the implementation of the new approach to school evaluation by ERO, the school had not experienced an ERO review for seven years.

A number of the students attending Waihi East School experience disadvantage. During Covid-19 lockdowns, the challenges some of the students’ families faced were exacerbated. These experiences influence the level of students’ engagement in learning.

The wellbeing of students and of staff is a key focus for the school. Partnerships within the school, and between the school and whānau, and the wider community are emphasised in planning and in decision making. The focus of the school team on wellbeing is also based on the view that unless students are settled and safe, they will not be prepared for learning.

The school focus on ‘holistic wellbeing’ of students reflects the view that it is important to emphasise what 'grows and glows’ students. While the leadership team focus on progress in academic outcomes, they regard life skills as equally important.

Te reo Māori is a part of classroom culture and Māori tikanga (customs) are integrated into curriculum areas. The school offers a range of tikanga activities such as kapa haka, and mihi whakatau, marae visits, te reo kori, waiata tawhito (local songs) and tikanga time. The whānau roopu created a waiata for the school and the song is performed at school gatherings.  Regular whānau roopu huis are held at the school.

The five working documents that support Māori and Pasifika success within the school are, Ka Hikitia, Tataiako, Tau Mai Te Reo, Te Tiriti o Waitangi (Treaty of Waitangi) and the Pasifika education plan.

Waihi East School has three strategic aims:

1. Tiriti o Waitangi – working to ensure that plans, policies and local curriculum reflect the partnership, participation and protection obligations we strive to achieve for all.
2. Partnerships – working to ensure that plans, policies and local curriculum reflect local tikanga and Te Ao Māori.
3. Raising Student Achievement – accelerate learning for priority and at-risk students, improve teacher pedagogy and improve student outcomes.

The principal has been in her role for the past two years. However, while a relatively new principal, they are an experienced teacher and local resident, having lived for over 15 years in the area. They have taught at Waihi East School and other schools over that period of time. The principal has taken on a range of leadership roles including a role within the Kāhui Ako alongside the deputy principal.

A number of staff interviewed during the collection of data for this case study spoke very positively about the school culture and the ethic of care that extends to all areas of the school.

Working with ERO - A collaborative foundation

ERO’s commitment to collaborative ways of working with schools on evaluation was welcomed by the principal and deputy principal. They support a tailored approach to evaluation that reflects the school context and is able to document the ‘school’s story’. For Waihi East a one size fits all evaluation approach with schools will not allow an accurate or comprehensive representation of the unique challenges and opportunities schools may experience. 

Understanding the context of each school is a critical element in educational evaluation, and school representatives from Waihi East felt that their evaluation partner was developing a solid understanding of their school community.

The opportunity to collaborate with the evaluation partner in developing and implementing evaluation within the school was identified as a strength. Senior leaders believe that this approach will grow capability within the school and build on priorities for student wellbeing and achievement. The principal commented,

“We are all here to support students to reach their potential…If you collaborate then schools are going to improve because the school can be more open and it’s based on a relationship. It means there is trust on both sides.”

ERO’s prior approach to school evaluation

Waihi East teaching staff interviewed for this case study contrasted their experience of the new approach with previous review processes. While the new approach is explicitly collegial and interactive, the prior approach felt judgemental and superficial. Teachers indicated that while the reviewers seemed respectful and friendly, ERO’s focus on judgement of school performance outweighed the interpersonal and relational attributes of the reviewers.

Teachers felt like they were ‘under the microscope’ and felt pressure to demonstrate their competence. One of the teachers in the school suggested that the previous approach communicated a level of professional disrespect for the teaching role. 

“For me understanding pedagogy and practice is not about flicking through a teacher’s plan. It’s way more than that. We are professionals, but we always felt like we were being personally assessed.” 

The synthesis process and public reports missed capturing the unique characteristics and features of the school. A teacher with experience of multiple ERO reviews suggested that the outcome of the review (the report) did not comprehensively reflect the school’s story or what had been shared with reviewers:

“Two to three days’ work and this would all be translated into a three-page report. We would produce all this information and include butcher’s paper across the whole wall of the library, and this was shortened into a summary that for us didn’t reflect the conversation. The essence of the school was not there.”

The potential of the new approach

For this school collaborative ways of working foster trust; the relationship is primary in building trust. The evaluation partner and the school work together on evaluation and improvement.  Reports are produced through a negotiated process that is grounded in the collection and analysis of a range of evidence over time.

Working alongside the school means that the school’s accountability for improvement is not limited to the end; it is integral throughout the entire evaluation process.

“Having the evaluation partner work alongside the school is really valuable. There is a much greater opportunity for an external pair of eyes to see things that we may not have seen.  It is way less stressful and more real. I get the sense that it is not about comparing one school against another school on some narrowly defined set of criteria, but really focusing on getting an understanding of the school, its context and students.”

The new approach has the potential to create reports that are more relevant and readable, and that may be useful resources for the school and for the wider community. The principal understands that public reports provide the community with information that may be important for families in selecting schools for their children. However, in her view public reports need to reflect the character of the school, as well as its performance.

Waihi East School leaders identified several benefits of the new approach, both for their school and for ERO. They believed that the new approach would provide ERO with an evidence base about what works and in which contexts across regions and schools. Sharing this information with schools will facilitate school improvement efforts as actions can be made that have worked in similar and different school contexts.

“ERO will get a much more accurate picture and a better understanding of different schools and what is working and not working, which would be a great resource for us if it was more widely shared.”

4.3 Strengthening evaluation capability

Internal stakeholders in five of the six schools believed that working with the evaluation partner over a period of time will improve their skills in internal evaluation. In their view, evaluation provides the evidence base to drive planning and delivery of change. It is not supporting evaluation capability for the sake of doing evaluation.

The evaluation partner from ERO is external to the school, and in this role is able to ask critical questions and facilitate discussion about areas for improvement. They are able to ask the ‘sticky’ questions and focus in on the most critical mechanisms that will foster school improvement. Their knowledge of school conditions and curriculum that support improvement, and their evaluation skills are a resource to the school. The unrelenting focus on equity and excellence and school improvement in line with these outcomes supports schools to “keep on track” with their improvement agenda.

Schools are accountable to students, their families, the school Boards, and to the wider community and saw the process of working with ERO as an opportunity to strengthen both internal and external accountability.

School leaders indicated that the skills they will learn through working with their evaluation partner will be sustainable, and applicable for subsequent evaluations. Schools have the opportunity to draw on a knowledgeable evaluation partner to develop questions, identify data collection mechanisms and produce useful insights. The tools that the evaluation partner shares with the school will become ongoing resources for other evaluations, this lessening the need for ongoing capability building.

A member of the senior leadership team at Kowhai School highlighted the opportunities for learning for the school that will benefit the school over time.

“We hope we will be able to evaluate in the future once we have developed the knowledge and skills about how that will look from this process. In the past, the model has been just to write down and record what you do. It was not a model that was about learning how to evaluate or how to improve.”

It is clear that schools will have different levels of existing evaluation capability and capacity, and evaluation partners will need criteria to assess the level of support required. Schools saw this as a strength of the new approach as the way the evaluation partner works with the school can be tailored to their needs.

The capacity to tailor the process to school need raises two questions for consideration:

1. How will assessments of existing school capability and capacity in evaluation be made?
2. Is the evaluation partner able to provide the level of support expected by schools, given practical constraints on resourcing and existing skill sets?

The following case profile from Parklands School provides some insight into the role the evaluation partner plays in building evaluation capability.

 

Case Profile: Parklands School: Strengthening Evaluation Capability

Parklands School is a primary school for students in Years 1 to 8 located in Motueka. Forty percent of the students identify as Māori, and there are an increasing number of students from diverse cultural backgrounds enrolling in the school. The student population is 186 students with a large number of these students (up to 73) on a needs register, who require additional support with their learning.

Pou whenua at Parklands School, bearing the school’s identity and values

The school is situated on a site with an attached technology centre catering for Years 7 to 8 students across the district. A family service centre, playcentre, early childhood centre and community oral health clinic are also located on the school site. A school social worker, strengthening families coordinator, a resource teacher of Māori and a highly respected kaumātua kuia (Māori elder) are also based at the school.

A member of the staff team created the carvings that stand at the school entrance and these pou whenua host the school values of KAHA: Kotahitanga (working together); Ako (learning); Haepapa (responsibility) and Aroha (empathy). At their completion a blessing was held, which was attended by over 100 people, including local Iwi, parents and whānau.

The school has strong Iwi involvement - both manawhenua – Te Atiawa and Ngāti Rarua and connection to Te Awhina (the local marae). Representatives from local Iwi participate in strategic school meetings.

The current principal began in her role in 2020, but is not new to the school or to the community. They have an extensive history with the school, having begun as a teacher at Parklands in 1997. As principal, they are eager for the school to become a strong cultural hub within the community.

The school is focused on preparing students to achieve in Māori and in English. Parklands has a Māori bilingual unit made up of 3 bilingual classes. The school offers Immersion level 2 (60% of curriculum taught in te reo Māori for more than 12.5 hours a week). The school aims to build up all classrooms in te reo Māori immersion level 4 (12% to 30% of curriculum taught in te reo Māori for more than three hours a week) across the school. The vision of Parklands is that students leave the school with:

• pride in themselves, their culture, their school, and community
• a strong sense of who they are and their potential
• the knowledge, skills, and mindset to engage in further education, and
• confidence in their ability to learn, change, adapt, and grow.

Working with ERO on evaluation

The principal believed that work with the evaluation partner was contributing to improved school capability in evaluation. They recognised that as a principal they needed to engage in monitoring and evaluation, but initially did not know where to begin with the process. They were enthusiastic about working with the evaluation partner and getting additional support for internal evaluation.

“When the new model started up and I heard that it would help the school build its evaluation knowledge and skills I thought, ‘yes.’ Before then I didn’t know where to start.  As a new principal there were all these folders with evaluation information in them... I knew I would be able to do evaluation, but the guidance from [our evaluation partner] has been invaluable.”

The principal explained ERO’s new approach to the staff team after the initial meeting with the evaluation partner. They felt that some teachers were initially sceptical that the new approach would be different from the prior approach. These teachers had experienced a number of prior reviews that appeared to be ‘narrow and focused only on what ERO wanted to know, not the full school context.’

Building and Sustaining School Capability in Evaluation

The school and the evaluation partner worked together to develop the plan in their second meeting. The evaluation planning discussion began with reference to the school’s strategic plan and the goals of the school. 

Literacy had been identified as the key focal area of improvement in the strategic plan. The principal engaged with the reading recovery specialist in the school to contribute her ideas and insights to the evaluation plan. The purpose of the evaluation is to explore the effectiveness of reading practices in years 4-6 (including effective teaching and assessment practices for reading, which promote student engagement, high expectations and continuity of learning). For the principal there was strong alignment between the strategic directions of the school and the evaluation focus, which adds value to the school.

Students at Parklands School

ERO’s work with the school presents an opportunity to maintain the school’s momentum on improvement and sustains their focus on what they are trying to achieve. The principal commented on the utility of the evaluation to the school.

Literacy had been identified as the key focal area of improvement in the strategic plan. The principal engaged with the reading recovery specialist in the school to contribute her ideas and insights to the evaluation plan. The purpose of the evaluation is to explore the effectiveness of reading practices in years 4-6 (including effective teaching and assessment practices for reading, which promote student engagement, high expectations and continuity of learning). For the principal there was strong alignment between the strategic directions of the school and the evaluation focus, which adds value to the school.

ERO’s work with the school presents an opportunity to maintain the school’s momentum on improvement and sustains their focus on what they are trying to achieve. The principal commented on the utility of the evaluation to the school.

“I like working smart. And this isn’t something else on top or doubling up on what we are already doing. The conversations started with our strategic plan. It is an action plan that reflects our strategic plan and strategic directions. Our strategic plan is about things that will make our school better. We worked backwards to identify gaps and evidence, and then develop an action plan. The evaluation plan is doing work for me. I can show our Board this.”

The principal explained the way the evaluation partner facilitated development of the evaluation plan through discussion and note taking on butcher’s paper. The principal keeps the butcher’s paper on the back of her office door as a reminder of agreements and areas of focus. 

The school believes the final evaluation plan is ‘their’ plan, not ERO’s plan. The plan will provide guidance to the school for other evaluative work. The process of developing the plan also enables the school to build its capability in evaluation, which can be extended to other domains of inquiry and improvement.

It is likely that subsequent planning phases may be streamlined as the school will have the knowledge and skills needed to progress their own plans for improvement. The principal stated,

“The evaluation plan was built together. We sat here and we brainstormed some ideas, and the evaluation partner wrote it down… Part of our thinking is now that we have this evaluation plan, we can replicate it in other areas. We can apply the learning to other year levels, and also use the evaluation plan to plan other evaluations.” 

After the evaluation plan was drafted, the principal shared it with other members of the senior leadership team in an open forum. The principal asked teachers from years 4-6 for their reflections and feedback on the plan, and to consider the implications of the plan for them. 

Student voice and perspectives will also contribute to understanding experiences students have with reading in years 4-6 and the evaluation partner is going to assist with capturing student voices over an upcoming two-day visit. Māori whānau perspectives and voice will be gathered as part of the evaluation through consultation and a survey. 

Parklands School evaluation plan discussions noted on butcher’s paper and used as a reminder of agreements and areas of focus.

The evaluation partner as a Change Leader

The principal shared her view that the evaluation partner supports the school to ask critical questions. While the previous approach appeared to be focused on accountability without an appreciation of context, the new approach is intended to be about learning for improvement. 

The school recognises the value of strong internal evaluation as a ‘matter of good practice.’ EROs approach builds opportunities for more open exchange about what is working or not working within the school, and practical ways of generating evidence to inform improvement.  It is envisaged that evaluation findings may also be used to advocate for additional resourcing and support for school and student needs.

In the principal’s view the perceived shift from regulation and inspection to collaboration and co-design with ERO is significant. It is indicative of a shift in mindset about what will contribute most in supporting improvement in schools.

“ERO is now interested in the school context and looking at our data and hearing our story. There seems to be more flexibility built into the approach. They (ERO) are not coming in to ‘check’ on us in the same way as before (the previous review approach).  The conversations we have with our evaluation partner are great. They are not always easy conversations, and they challenge us and ask us to think more critically about what we are doing. Working with ERO now is like having a change leader in evaluation work”.

4.4 Construction of the Evaluation Plan

ERO’s evaluation approach provides an opportunity to develop an evaluation plan that is tailored to the school and builds internal capability in evaluation. The plan is not only for evaluation, but for evaluation and generation of evidence that will support planning and improvement actions. Schools understood that the evaluation plan, and the actions that stem from the plan and from evaluation, will lead to school improvement. Qualitative and quantitative data is powerful and the potential to collect rich sources of evidence throughout the evaluation was seen to be a strength.

The focus on the relevance of the plan to the school is a stark contrast to the previous review approach. For school stakeholders the previous approach had focused on what the school had done and school performance, rather than strengthening evaluation knowledge and skills that are grounded in an appreciation of school and community context. 

While schools recognised that ERO still has to verify what the school has done in terms of public accountability, they believed that the final report will have elements of co-construction. It was clear at the time the case studies were conducted that school stakeholders were largely unaware of the formal requirements for reporting or the required process for generating them.  At that stage, reporting requirements were still under development.

The case profile below illustrates the experience of the evaluation planning process within Kowhai School. The case highlights the ownership the school has over the evaluation plan, and the difference school leaders believe this will make to their improvement efforts.

Case profile: Kowhai School: ‘It’s our school evaluation plan’

Kowhai School was established in 1975 and has a focus on providing integrated educational experiences for all students. The network of base and satellite classes offer pathways through lower primary, upper primary, intermediate, and high school. 

Students are able to attend Kōwhai School from age 5 until age 21. Currently, the school roll is approximately 100 students, with growing demand from the community for places in the school.

Determining the evaluation focus – School transitions

The evaluation focus for the school was co-constructed with the evaluation partner, the deputy principal and another two members of the school leadership team[4]. In the initial meeting with the evaluation partner the group discussed potential areas requiring further development.

They were aware that settling on the evaluation focus requires consideration of a range of issues, including clarity about the purpose and intended use of the findings.

Support for effective transitions had been identified as a key direction for the school for some time and was selected as the evaluation focus. Issues of scope were also discussed in the evaluation planning session. While there had already been a lot of work to support students’ transitions, the school recognised it needed to do more work to support students in their educational pathway and post-school services and employment in the community.

The alignment of the evaluation plan with the school’s strategic directions has made the evaluation process ‘incredibly useful’ to the school. For the principal and the leadership team the process of chunking the evaluation into stages or phases helped build confidence in their capability to complete the evaluation. While separating the evaluation into phases made the process more manageable, it was the evaluation partner’s knowledge of the school and community context together with her evaluation knowledge and relational skills that reinforced their confidence in the evaluation process. The principal explained:  

“We are directing the evaluation, but with [the evaluation partner’s] support and guidance. They are really good at putting us at ease and validating what is important for us as well. They understand] the school context, and they know the evaluation work. They ask us critical and often challenging questions, but in a way that is not threatening to us.”

As a follow up action, the school hosted an evaluation forum to progress work on their evaluation plan. The school used the forum as an opportunity to seek feedback from students, parents, whānau and agencies about their transition experiences. They invited school leavers from 2018, current students and families as well as some agency representatives.  

The forum was well attended, which the school attributed in part to a personalised invitation and follow up phone call.

School staff posed four or five key questions to the group during the forum. Post-it notes and butcher’s paper were used to capture the feedback from participants, parents, whānau and transition agency representatives who attended the forum. The school collated feedback, including photos from the event. The school plans to share the report with their evaluation partner on her next visit.

A group of students at the feedback forum – Kowhai School

From the school’s perspective collecting data over time will allow them to review aspects of their practice on an ongoing basis, to keep the focus on how the school can improve transition experiences better for students and their families.

The deputy principal commented on the usefulness of the feedback and its use for school planning and improvements:

“We wrote it. We built it together. We sat here and the evaluation partner took notes and then went away and typed it up. It captured what we had discussed. It wasn’t her plan, or ERO’s plan. It is our plan, our school’s plan.”

4.5 The Need for Differentiated Evaluation Support

ERO’s evaluation approach is based on tailoring evaluation support to schools based on their needs. There may be value in ERO exploring criteria for providing differentiated support schools for evaluation, providing more intense support for some schools and more of an oversight evaluation role in others. Differentiation will need to be based on a diagnostic discussion with the school and with any agency partners involved in providing support to the school. In schools where evaluation capability and capacity are already strong, ERO’s role could offer a ‘lighter touch’ to monitor progress and outcomes.

There are also school contexts when the ongoing involvement of the evaluation partner may not be warranted because the school already has external groups or agencies working with them on strategies for school improvement. In these contexts, the evaluation partner will need to carefully tailor the level of support to avoid duplication and confusion. 

The following quote from a Student Achievement Function (SAF) practitioner highlights a potential concern when multiple agencies are working with schools. If the school has a high level of maturity, capability and strong direction and capacity to deliver on this then we merely confirm this and get out of their way.

“There needs to be more clarity around which plan … is driving improvement.  I work with the school on a regular basis to develop and implement a change plan and that change plan might change depending on how it goes in supporting improvement. It is an inquiry-based model with evaluation built into it. But there is a risk that these plans become another layer of administrivia being put in place…It would be better if the evaluation plan lined up with what the school is already doing, rather than adding another layer of complication. We are both supporting the school… It’s a system and process issue, not a criticism of any agency or tools and templates.”

The school profiled below is Kaitaia Intermediate School. The school is involved in a range of change initiatives designed to support teachers and student wellbeing and school performance.

The case profile presented below highlights the importance of tailoring the evaluation process to the school needs and considering the appropriate scope of engagement.

Case Profile: Kaitaia Intermediate School, a case for differentiated evaluation support

School Context

Kaitaia is a community of over 6000 people who live in both urban and rural areas. The main industries in the area are forestry, tourism, and farming. Unemployment is higher than the regional average. Kaitaia Intermediate School is a contributing co-educational state school for year 7 and 8 students, currently serving 240 students.

There are nine classroom teachers, four technology specialist teachers, one Rāranga weaving teacher, and a Resource Teacher Māori who work alongside administration and support staff. 

The school hosts three bilingual classes to provide te reo me ōna tikanga programmes for those who choose high level 4 to beginning level 2 bilingual immersion at years 7 and 8. A level 3-4 te reo programme in mainstream classes is also provided to encourage students and staff to become more confident in identity, language and tikanga practices.[5]

The acting principal and deputy principal pointed to the unique opportunities and challenges of intermediate schools, which only have students for two years of their education, which means that at the start of every school year half the students are new entrants to the school. Students join the school from one of eight feeder schools and in the first several months they must learn to adapt to a different school environment. The school focuses on supporting students to learn how to get on with others, adjust to a more complex school structure, and develop aspirations for the future.

The school has a two-fold vision that exemplifies the focus on the academic, social, cultural and personal outcomes of all students. The first vision is that students will be ‘confident, connected, actively involved, lifelong learners who demonstrate respect, responsibility and form positive relationships.’

Kaitaia Intermediate School values

The second vision is that the school will develop ‘the academic, social, emotional and physical wellbeing of all students through a focus on Ako, Manaakitanga, Whānaungatanga and Moemoea.’

A context of disadvantage

School data indicates that the school faces ongoing challenges with low achievement of students particularly in maths and literacy. Achievement disparities are more prevalent for Māori and male students.

The challenges that students face have been exacerbated over the past two years as a result of Covid-19. Attendance is a key issue for the school.

Sustainability of a partnership approach

The acting principal and deputy principal acknowledged the importance of ERO’s role in supporting the school’s evaluation. They appreciated the shift to a collaborative approach where the evaluation partner works alongside the school. 

The previous approach was episodic and focused on assessment with little apparent appreciation of the school context. The principal explained the implications of the shift towards a more collaborative approach by contrasting the new approach with the previous approach to school evaluation:

“In the past ERO would come in, and they would observe and make judgements based on what they saw and it came down to a judgement of academic achievement. We understand that achievement is important, and that is why we are here. The context is that our students may be entering school at a level that is lower than anticipated. The chance of us getting students up to the expected level of secondary school is a challenge.  A focus on academic achievement does not take into account our context and the challenges facing students and families here. The new approach does take that into account. We are growing students in a range of ways – in reading, writing, maths and we are also focusing on them having aspirations, goals, routines, and being good citizens. We are shifting our emphasis from just achievement to demonstrating progress.”

While the principal and assistant principal valued the collaborative approach, they felt that there had been insufficient level of engagement to date to progress the evaluation plan. The school expressed a preference for more regular contact with the evaluation partner. This was not a criticism of the evaluation partner; the school was aware that the evaluation partner was working across multiple schools, and they also acknowledged the impact of Covid-19 lockdowns, which had prevented in-person evaluation visits. Both senior leaders expressed concern about the integrity of a partnership approach if the evaluation partner was not sufficiently resourced to work in an ongoing and regular way with the school.

Agency engagement with the school

The school is currently being supported by a skilled practitioner serving in the Student Achievement Function (‘The SAF’) and working alongside an experienced educational consultant. The educational consultant is supporting the school with student learning profiles and the broader mission statement for the school. The SAF practitioner is working with the school on a change management process to address staff and student wellbeing within the school. The evaluation partner from ERO is working with the school, the SAF practitioner and the educational consultant in development of the evaluation plan.

The school adopts an inclusive whole school approach to planning, involving all staff, including the caretaker and administration staff in professional learning and development. This communicates an important message that everyone is a leader and has responsibility in being visible in supporting students within the school.

EROs evaluation planning process engages with the agencies that support the school. However, there appears to be a potential risk of duplication of effort as the SAF practitioner and the educational consultant have in-built inquiry learning and evaluation mechanisms to monitor progress against plans. The creation of a separate evaluation plan may potentially duplicate what other agencies are doing in the school and contribute to confusion about purpose and role.

The SAF practitioner emphasised the importance of clear roles and responsibilities of agencies working with schools to avoid confusion.

“If a school is tracking well, they may really value ERO’s role in supporting them to evaluate. If there are others working around improvement where monitoring is already built in, then it might be better for ERO to step back and allow them to do what they do.  Otherwise, clarity gets lost. ERO can then come back into the school and evaluate the outcomes of that work.”

Kaitaia Intermediate School Library

In school contexts where multiple agencies are present and working with the school on improvement, it may be more useful for ERO to maintain an oversight role and plan for a strategic level evaluation of the effectiveness of the change efforts within the school. There is potential for the evaluation partner to establish overarching evaluation mechanisms to monitor the impact of work designed to improve learner achievement, and to evaluate the contribution of partnership ways of working among agencies and schools in progressing improvements.

The approach to school improvement and to evaluation is a team approach. The school works with ERO, the Ministry, and other providers to determine the necessary resources to make improvements. The challenge in inter-agency working is to clarify role expectations among all stakeholders to ensure the process is of value to the school and does not add layers of confusion or redundant paperwork. ERO’s evaluation can then be used by the school and by agencies to understand the benefits of joined-up approaches to school improvement

 

4.6 The Role of the Evaluation Partner

The preceding discussion has highlighted that schools value the collaborative foundation that underpins the new approach to school evaluation. This points to the need to consider the attributes and skills that evaluation partners bring to schools to support an improvement-oriented approach to evaluation.

ERO has identified the knowledge, expertise and personal qualities and dispositions required for high quality education evaluation through the evaluation capabilities framework. 

The evaluation capabilities are intended to ensure professionalism, guide professional learning within the organisation, shape quality assurance mechanisms, and strengthen the credibility and integrity of ERO’s evaluation work.

The core capabilities reflect the requirements of an education evaluation role within Aotearoa New Zealand, that is cultural and contextual understanding and responsiveness, professional leadership and teamwork, evaluation knowledge and practice, and interpersonal and communication skills. The relationship of these capabilities within the context of evaluation for improvement in schools is depicted in the diagram below. 

Evaluation Capabilities

Evaluation is a profession with a specific set of professional skills and competencies. Becoming knowledgeable and skilled in evaluation takes time. The role requires a range of technical, practical, and relational skills. Evaluation partners need to possess:

• knowledge of conditions that need to be in place for school improvement   
• understanding of the New Zealand education system and the policies and frameworks used to support school improvement
• understanding and commitment to the implications of Te Tiriti o Waitangi for equity and excellence
• ability to establish and maintain professional relationships
• capacity to adapt and flexibility in problem solving
• capacity to convey evaluation concepts in a concise and clear way that is appropriate to the audience
• good writing skills to translate discussions into actionable plans
• knowledge of major approaches to evaluation and data collection, and
• strong ethical standards and commitments.

The required skill set is extensive and requires knowledge and skills across a range of domains – education, evaluation, and organisational development.

Experienced evaluators are able to skilfully adapt their skills to fit the needs of schools and school stakeholders. For newer evaluators, the knowledge and skills will develop over time with experience and from learning about the most effective ways to work with schools.

While evaluation partners bring different strengths to their roles and the process is tailored to schools, there is a need for consistency in practice. Templates and tools provide some guidance on process requirements and support ongoing monitoring, but need to be supplemented with opportunities for feedback, continuous learning, and critical reflection. 

Evaluation partners are part of an evaluation team, with supervision, and have access to professional learning opportunities to strengthen their knowledge and skills. ERO had a range of resources available to evaluation partners and access to a range of tools, national research, and evaluation outcomes. ERO will need to identify ways to provide ongoing support and capability building according to the scope of practice as it evolves over time.

4.6 1 Experience of Working with the Evaluation Partner

The evaluation approach emphasises the development and maintenance of a collaborative partnership between the ERO evaluation partner and the school. For five of the six schools their experience of the evaluation partners had been very positive. 

In most schools the evaluation partner was already known to the school having worked with them in prior reviews or, in one case having conducted research with the school via Zoom in 2020 on the impact of Covid-19 on schools. The basis for the collaborative relationship had already been established. In this context, the schools did not notice a shift in the way the evaluation partner worked with them. In their view, the evaluation partner had always been respectful, considerate, professional and collegial. However, the philosophy underpinning the new approach gave this relational way of working more legitimacy and enabled the evaluation process to be more explicitly collaborative. A school principal from one of the schools commented:

“In the past I often got a feeling that some of the people were constrained. They were nice enough, professional, but they were sort of hampered by this set of expectations.  Now, there is a recognition and a focus on context, and on relationship. ERO is now interested in the school context, not just data on paper.”

School representatives from five of the six schools spoke positively about the way their evaluation partner worked with them. They highlighted the collegial, relational way their evaluation partner worked with the school to identify priorities to shape the focus of the evaluation.   

“X (the evaluation partner) has been here 4 or 5 times. We are loving it. I get on well with x and value the conversations we are having.  They are not afraid to challenge me and question me. If I didn’t get on with the evaluation partner, that would certainly make this difficult.”

The relational and interpersonal qualities the evaluation partner brings to work with the school is a core necessity in creating a successful collaborative relationship. Evaluation knowledge, skills and educational experience are also important. 

One of the six schools involved in the case studies held a very negative view of the new approach on the basis of their experience of the development of their evaluation plan. The case profile below highlights how a ‘clash’ of process and expectations with the evaluation partner disrupted the evaluation process.  

The evaluation partner was regarded as competent, enthusiastic and professional by both the principal and the deputy principal. However, the professional relationship became untenable and a decision was made that the evaluation partner would not continue working with the school. The experience of this school of the approach is highlighted in the following case profile.

Case Profile: St Bernard’s College, Unmet Expectations

School Context

St Bernard’s College is a state integrated Catholic school for boys from years 7-15. The school is in Lower Hutt and most students are from the area or nearby Wainuiomata. The roll is 668 students. Twenty-six percent of students identify as Māori and twenty percent of students Pacific.

The wellbeing of students throughout their journey in the school is a core focus for the staff team. The phrase ‘standing on life-giving ground’ that sits below the school banner at the entrance to the school reflects the school’s focus on creating an environment where boys can ‘stand taller and thrive and grow.’ The goal of the school is to inspire success in all students in partnership with whānau and the community.

Students are encouraged to interact and connect through mixed aged group learning opportunities and social activities.

Several buildings are being redeveloped to meet the needs of the growing school community.  In 2018, the gym complex was developed with the addition of a new entrance and foyer, Physical Education classroom and specialist weights room. A new commercial kitchen was completed late in 2021. The Gregor Mendel Science Block was formally opened in 2020. The old main block is being demolished and will make way for a new purpose-built structure to be completed early in 2022.

The college's Marist Way links to the values of manaakitanga, social justice and integrity. The school focuses on:

• student-centred teaching and learning
• celebration of diversity, and
• supporting student growth to their full potential in spiritual, academic, creative, physical and social development.

The school provides for instruction in tikanga Māori and te reo Māori for full time students when requested. Students at all curriculum levels are offered the opportunity to learn te reo Māori.
Māori is compulsory for Year 7 and Year 8 students. Two kapa haka groups are active within the college. New staff and students, and special visitors are welcomed with pōwhiri or mihi whakatau as appropriate.

A group of boys at St Bernard’s College.

Working with ERO- Unmet expectations

The principal understood from ERO communications and initial meetings with the evaluation partner that the new evaluation approach represented a marked change in purpose and process. It was to be based on collaboration and co-construction. 

The principal and deputy principal were hopeful that the new approach would provide an opportunity for ERO to get to know their school context and support them in strengthening their evaluation capability in progressing school priorities for improvement. 

The school had an expectation that the evaluation partner would spend time getting to know the school and its context. School representatives felt the evaluation partner was personable and professional in initial meetings. However, in their view the evaluation partner came in with an agenda of pursuing a ‘project’ that would fit with the new model rather than listening to what the school had been doing and how it wanted to move forward. The principal felt that the evaluation partner did not acknowledge the school’s context or its achievements to date. The principal explained:

“We were really disappointed and offended by the wording and assumptions in the evaluation template (evaluation plan). It feels like they came in with an ERO agenda, but without understanding of our school or the school context. I invited (the evaluation partner) to attend different events. The vision of working alongside schools and collaboration is not adequate if this is resourced for a reviewer to come in for a couple of hours now and again.”

Despite several re-negotiations of the evaluation focus, the relationship between the evaluation partner and the school remained somewhat fractured, and the evaluation process was disrupted by multiple iterations of the evaluation focus, and disagreement on the framing of the evaluation.

Lack of Acknowledgement of school strengths and achievements

The principal and deputy principal expressed frustration and disappointment at the lack of acknowledgement by the evaluation partner of what the school had already achieved, and the school’s existing strengths. The school felt that the evaluation partner as a representative of ERO did not express interest in a number of school-wide and local initiatives that the school was focusing on to improve student engagement, wellbeing and achievement.

The expectation of co-construction and collaboration was not realised.

“In co-construction you come in and listen to the school’s story. You do not bring your own preconceptions in about what achievement and equity looks like. We have been working on a range of improvements in support of students using PB4L, their wellbeing and also on restorative practices. The ERO template was written using terms for objectives like the school will ‘begin building capability.’ It is written as if the school has no internal evaluation capability and capacity.”

School representatives found some of the language the evaluation partner used to define areas of improvement difficult to understand, reflecting ‘ERO speak and jargon’. It was difficult to determine the practical implications for example of phrases such as ‘communication for learning’.  

An example of the challenges experienced in the use of language was shared by the assistant principal, who felt confused by the plan and some of the language used to reference school improvement.

“The plan[6] was full of confusing language. I sat here confused… I have been teaching for a very long time. I did not understand what (the evaluation partner) was talking about… There were times I thought I understood the language and what was meant by particular statements but then I would second guess myself. I would pretend I got it, but deep down from other things that were being said, it was clear that was not the meaning that was intended… we are all confused.”  

The school shared their concerns with the evaluation partner and modifications were made to documents as a result of these discussions. The school agreed to undertake a small-scale evaluation project to progress their improvement efforts. The school expressed keen interest in working on identified priorities and saw a role for ERO in supporting them. 

How can ERO be a support for school improvement?

The principal sees the potential for ERO to support the school with key areas of improvement through sharing practices that have worked in other schools, or by pointing the school to relevant international and national research.

“We are doing a whole review of our junior curriculum. I would like to understand what progress looks like. A key question we have is how do you engage Pacific and Māori families? We have tried a range of things. What is effective? Could ERO be a resource to help us? To help us or guide us to the international and national evidence. Who has given it a go that is similar to us? What are they doing and how are they measuring it?  We also have this huge issue of attendance across the board. We have done quite well, but we still think it is not good enough. We’d like more evidence about what has worked elsewhere.”

What are the learnings from this case?

St Bernard’s College has not had a positive experience with the new ERO evaluation process to date. The school had expectations that the partner would spend time getting to know the school context, attending key meetings and learning about the school culture. This was the message ‘sold’ in ERO communications to the school. However, while the principle of co-construction and collaboration underpins the philosophy of the approach, each evaluation partner has to navigate school expectations along with the need to balance responsibilities of working with multiple schools. 

This case profile points to the importance of the relationship platform. The school and the evaluation partner need to appreciate the opportunities and constraints within the context and identify strategies to progress evaluation that builds on strengths, but maintain a focus on equity and excellence in educational outcomes for all students.

It is clear too that this profile illustrates that the fit between the school and the evaluation partner is key to the development of a trusting relationship and a useful evaluation process. Not all evaluation partners will be a good fit for the schools they are allocated to within their region, and not all schools will be open to a reflective and robust evaluation process[7]. While St Bernard’s Catholic Boys Schools represents an outlier case from the other schools in this report, their experience identifies critical messages for ERO. There are likely to be other schools that experience a disconnect between the philosophy of the new approach and the approach in practice.

Response from ERO:

The relationship between evaluation partner and school is an essential factor. ERO acknowledges that in both the evaluation partner and the school should have been better supported through EROs processes and implementation. The purpose of these case studies is to help ERO identify conditions that facilitate or inhibit implementation of Te Ara Huarau. The value in this feedback enables us to improve the implementation and improvement of Te Ara Huarau.

4.7 Resourcing and Scope of Practice of the Evaluation Partner

Schools valued the opportunity to work on the evaluation plan with the evaluation partner. It was clear that schools see value in learning about evaluation and being able to apply those learnings in other improvement efforts in the school.

The implementation of the approach in the initial group of schools required dedicated time to understand the school context, outline the phases, develop the evaluation plan, and refine evaluation questions.

In the initial implementation of the new approach with the 75 schools, evaluation partners were each allocated five schools. There is an expectation that in future implementation they will work with up to 40 schools. The time allocated to supporting schools in evaluation will therefore be more constrained with a wider number of schools. Evaluation partners will need to balance demand against resources and available time for travel and meetings. They will need to have sufficient flexibility to schedule meetings with schools to align with key phases and school scheduling.

Two schools in the case study raised concerns about the capacity of the evaluation partner to maintain a collaborative evaluation relationship given the number of schools they will be allocated in subsequent iterations of the approach. One principal suggested that this was a risk for ERO in terms of being able to develop an effective relationship with the school and understand the school context: ‘If the evaluation partner has too many schools, where is the rich learning for either party?’

The development of an understanding of the school cannot generally be achieved in one visit and schools appreciated the ongoing engagement with the evaluation partner. With a greater number of schools it will be challenging for the evaluation partner to maintain ongoing contact with schools and initiate regular touchpoint meetings.

“One of my concerns is about what the partnership will look like if the evaluation partner gets allocated too many schools. I cannot imagine they will be able to work closely with the school and provide the same level of support.” (Parklands Principal)

It will not be practical for the evaluation partner to offer the same level of ongoing, regular support to all schools given existing resources. Yet, schools hold an expectation that there will be regular engagement. Schools wanted to ensure that the evaluation partner allocated sufficient time to understand their context. There is a potential disconnect between the philosophy of collaboration that underpins the approach, and available resources.

It is likely that the first time ERO works with a school on evaluation may be more intensive as the school is learning about evaluation, the evaluative processes and the use of evidence. Subsequent evaluation cycles will not require the same levels of intensive support from the evaluation partner as they will have the knowledge and skills to lead the evaluation process. Schools will be able to use the skills they have learned in the first evaluation cycle and adapt it for subsequent evaluations. 

In schools with existing high levels of capability, or where the school can confidently undertake its evaluative work, the ERO evaluation partner may appropriately assume the role of critical friend and validate the quality of the evaluation the school has undertaken, rather than ‘walk alongside’ the school throughout the whole process.

“Part of our thinking here is that now we have this evaluation plan we can replicate it in other areas we want to work on. We can apply the learning to other year levels and also use the evaluation plan to plan other evaluations.”

There needs to be sufficient time for collaborative evaluation planning for it to be meaningful and effective and for the scope and limits of collaboration to be communicated clearly. Without this the credibility of the new approach may be at risk. A key question for ERO is how to manage school expectations within the limits of resourcing available to support school evaluation capability. 

4.8 ERO’s role in supporting learning and accountability

School stakeholders indicated that ERO’s role was important in facilitating learning for improvement in schools and for ensuring schools are accountable to students, their board and the community. These purposes are supported through ERO reviews of schools. Independent reviews of schools promote community confidence in the education system within New Zealand and may help parents and whānau in making decisions about schools for their children. ERO also works with schools to strengthen internal evaluation capability to inform improvement.

For schools participating in this case study, ERO’s prior review model was more focused on external accountability and competition (achieving a 4-5 year return time) rather than developing professional accountability through learning how to do and use evaluation for improvement.

The previous approach to school evaluation was regarded as superficial and narrow by most school stakeholders in case study schools; it was not practically useful to the school in any way.  Most school stakeholders highlighted the increased levels of stress and anxiety among school staff ahead of scheduled ERO reviews. Plans and paperwork were prepared to meet ERO expectations and requirements. The workload and stress on teachers in preparing for the ERO review was not a good use of their time. Schools were relieved when the review was completed, aware they would most likely not be reviewed for at least another two to three years.

Three quotes from internal staff at three different schools are presented below to illustrate experiences of the previous review approach.

“It was a pretty nerve-wracking time, and not of much value to the school. You’ve always got to put your best foot forward and hope the right parents talk to them about how good the school is. The review process was set up for judgement, but without an in-depth knowledge of the school.”

“ERO was focused on data at the expense of relationships and understanding the school.  Everyone would be on their best behaviour, and we used to put together reams of material, but it was of little value to the school.”

“For us ERO’s reviews were a waste of time and money. They didn’t add value to the school. The new approach is trying to support us in evaluation, not just narrowly looking at measurement and what we are doing, what we can show. It’s (the new approach is) an opportunity to be honest and learn, not hide like we did in the past.”

Schools supported ERO’s role as an independent organisation and acknowledged the value of the accountability function. However, accountability was believed to be more likely to be realised from a basis of trust and partnership, than distrust and distance, which characterised the previous approach. School representatives understood that learning and accountability could co-exist; they were not contradictory objectives. Two quotes illustrate the value of both roles:

“Accountability stems from checking in – how is it going and asking good questions. The old model was about checking off, not checking in. The approach now seems to be about working with the school in an ongoing way and checking in about what is and what is not working.”

“ERO… offers an outside professional view of the school and there is integrity inherent in that role.  It is very valuable. There is a huge value in their independence. They have this wonderful base of information from 2500 schools and some very capable and experienced people as well.” (Tauraroa Area School)

The new approach has the potential to be much more useful to the school, in building their evaluation capability and supporting school improvement plans and initiatives. Rather than a ‘one-off’ time limited review, the schools understood that they will be working with an evaluation partner from ERO with knowledge and skills in education and evaluation to support them in developing robust evaluation plans and generating evidence to inform improvement. 

The alignment of the evaluation process with the school’s strategic direction represented an acknowledgement that the evaluation was about the school, and its progress rather than for ERO and its ‘definition of school success’. Aligning the evaluation with the school plan also makes the evaluation process useful for the school.

Schools believe that the evaluation for improvement approach will contribute to improved evaluation and contribute to improvement planning. School representatives in the case studies anticipated that promising or good practices in schools will be shared as part of the evaluation process. Five of the six school principals suggested that improved sharing and dissemination of learnings by ERO will allow them to improve their effectiveness and efficiency in developing actions that will contribute to improved outcomes for students. They referred to the extensive knowledge base that ERO has, both about the challenges schools experience, and also about the most successful strategies to address these challenges. 

The following case profile describes the experience and views of the principal and associate principal at Tauraroa Area School about the new approach. This case profile also highlights the role that evaluation plays in promoting knowledge, learning and accountability. While ERO’s national reports and learnings have been disseminated to schools to inform improvement efforts, there has not been a deliberate strategy to translate this information to shape improvements at the local level. Sharing lessons learning and promising practices through the new approach will enable schools to assess the fit of particular strategies to their context and make direct changes to improve outcomes for learners.

Case Profile: Tauraroa Area School, the contribution of ERO’s approach to Learning and Accountability

School Context

Tauraroa Area School is Year 1-13 school located 22km south-west of Whangarei. There is no settlement around the school itself, just a small number of houses. The school roll sits between 510 and 520 students. 

A few kilometres up the road in Maungakaramea (towards Whangarei) is another school catering for students 0-8 years. Enrolment numbers vary between the schools; as one school’s roll increases, the other school’s declines. The school has a long and proud history in the local community, dating back approximately 100 years as a primary school and offering secondary education since 1958. The school is the largest employer in the local district.

A lot of students travel some distance from home to get to the school from as far as Whangarei, Ruakaka, and One Tree Point. There is even one family attending the school from Mangawhai, 56 kilometres away. Transportation to and from the school by bus can be costly for parents.

From 2020, Year 9 to Year 13 students have been arranged into one of 13 horizontal forms. Each whānau roopu class has students from only one Year level. Tauraroa Area School places a strong focus on supporting pathways to suit students’ interests and aspirations whether that be in farming, trades, university or other study. The school has a careers programme with work experience and Gateway courses to strengthen students’ transitions to work.

ERO’s new approach supports learning and accountability

The principal and the associate principal[8] suggested that the new approach represented a significant shift in ERO’s approach to school evaluation. They recognised the value in an independent review of the school by ERO but felt that previous approaches to school evaluation had not been useful in informing school improvement. The key conditions that prohibited utility were the time-bound or episodic nature of the prior review process, and the lack of opportunity for reviewers to gain a deep appreciation of the school and community context. 

The principal explained his previous experience with ERO review processes.

“In the past when ERO visited not a lot would be achieved. They would come for a few days, and we would be all prepped and ready for the visit. A principal’s career stands and falls on their judgement so it can be a pretty nerve-wracking time. You’ve got to put your best foot forward and hope that the right parents talk to them about how good the school is. It (the review process) was set up for judgement, but perhaps without the in-depth knowledge of the school.”

For this school, as for other case study schools, an understanding of context cannot be achieved in a few short days. The previous review model was predicated on assessments of the school at a particular point in time. Because the reports prepared by ERO focused on opportunities for improvement, they did not always fully acknowledge the strengths of the school or the achievements to date. Many strengths of the school, such as the school’s safe and caring culture, were not easy to quantify. In this sense the evaluations lacked some balance.

The principal and board, however, recognise the value gained from external, objective evaluations of the school, albeit as brief ‘snapshots.’ Review findings could be, and were, fuel for positive change. School reviewers were always open and eager to communicate with senior managers during reviews to ensure there were no surprises in subsequent reports.

“The process really depended on how things were on the day so it could be quite unfair and inconsistent. We hold a view that we are a well-functioning school. Of course, there are always things we can improve, but the ERO reports did not reflect the things we were already doing well. Maybe we didn’t express that properly, maybe we didn’t show them what they wanted to see at the time. The new approach offers opportunities for the school and ERO to observe changes over time, not just at one point in time.”

ERO keeps us on track and accountable

ERO’s new approach, founded on collaborative ways of working was welcomed. The senior leaders saw it as an opportunity to partner on an improvement journey. Accountability is an important element of ERO’s role and reviews over time will mean that the school stays on track with the commitments made to gather and use data to inform improvement. While the previous review approach recognised the importance of accountability, the process encouraged schools to present the most favourable image possible, rather than acknowledge shortcomings and vulnerabilities. An evaluation process over time will ensure that the school maintains its focus on learning, improvement, and accountability. The following quote illustrates the value of regular touchpoints between the school and their evaluation partner.

“ERO (through the new approach) is providing support for us and our journey as a school. It’s great to bounce ideas off ERO and have them work alongside us as we work on it. We are not just being left to our own devices...We all get busy and let things slip behind, or allow some other issue to take priority, but this process with ERO brings it back to the forefront, to make sure it isn’t just one snapshot per year, and we don’t just say, “What can we get together to show them?” We can be honest about where we are at.  Sometimes things do slip a bit visit to visit, but the regularity brings us back on track. Within a month or two we can share what has been going on. If it was three years down the track we would not remember and recall what impact an issue had on the school and how we resolved it.”

Value of sharing effective practice with schools

The principal and associate principal recognised the value and importance of ERO’s role within the New Zealand education system. They anticipate that the new approach will enable learnings to be shared more directly with schools to inform improvement efforts. In this way ERO will be able to show-case examples of promising practice or improvement efforts with schools experiencing similar challenges within similar contexts. The principal highlighted the value of national reports prepared by ERO and sees the new approach strengthening the application of these learnings in schools.

“ERO publishes the national reports, which are goldmines, but reading something and knowing what to do about it are different things. How can this filter down to influence classroom practice? The whole change management thing is so complex. ERO are in a very powerful position, to improve education in New Zealand… The thing I really like about the new model, is the fact that there’s now the opportunity for ERO to show best practice from other schools and things like that, particularly at something that we’d like to work on that somewhere else is doing before. Whereas before we’ve all been working in silos, and unless you’ve known somebody who’s in a school that’s doing such and such, that support wasn’t there. So, whilst we haven’t tapped into that just yet, because it’s too early in our journey, just knowing that that’s there, is a really big help as well.”

Taurora Area School: Pre Apprentice Trades (PAT) class built a sandpit for students at Nga Tau E Toru Te Kohanga Reo, Te Horo School, Pipiwai

Section 5:  Key Questions and Implications

The case studies describe the experiences of six schools of the Schools: Evaluation for Improvement Approach in the first year of implementation. Schools had not yet progressed through an entire evaluation cycle at the time this external evaluation of the approach was conducted. The case studies were designed for a formative purpose, and not to assess the merit and worth of the approach.

However, the case studies are helpful in highlighting experiences and capturing early impressions of school stakeholders of the new approach. It will be important for ERO to continue to monitor school experiences and document outcomes across time to generate a fuller understanding of what worked, what didn’t, and to identify opportunities for further improvement.

The schools in this case study were diverse in school type, size, and context. Across this diversity a number of common themes in experience and perspective were identified. A synthesis of key findings presented in this report highlights a number of questions for ERO.

Each of the following questions is elaborated in the following section, with recommendations identified for consideration by ERO

  1. How will ERO manage school expectation within existing resource constraints?

  2. How can evaluative capability be extended across schools and within classrooms?

  3. How will ERO align its work with other partners who are also working with schools to progress
      improvement?

  4. How will ERO build and maintain its internal capability in evaluation?

  5. How will ERO support schools with internal evaluation and support external accountability?

  6. How will ERO know it made a difference at the school level?

  6. How will ERO know it made a difference at the system level?

Summary of recommendations

It is recommended that:

1. ERO identify criteria for differentiating the level of evaluative support provided to schools. Criteria will ideally be based on school need and context for improvement. Schools and evaluation partners will need to understand the opportunities and limits of time allocated to various aspects of the evaluation process.
2. A process be developed to support monitoring of the spread of evaluation capability across the school in development of evaluation plans, generating evidence from the classroom, and implementation of actions that lead to improvement in student outcomes.
3. ERO identify strategies to support collective, inter-agency ways of working on improvement with schools.
4. ERO identify a suite of strategies to strengthen the capability of evaluation partners (for example, through specialist internal evaluation programmes, targeted mentoring and through learning opportunities within and across evaluation teams in the three regions).
5. ERO establish external reporting requirements that retain a focus on documenting the school context as well as highlighting their performance story over time. The expectations of external reporting should be clearly communicated to schools early in the evaluation process.
6. ERO identify ways to monitor and evaluate the contribution of their work to school evaluation and school improvement at the school, region and national level.
7. ERO make findings and insights from national or regional evidence more accessible and widely available to schools. The evaluation partner may directly draw from this evidence base to engage schools in discussion of improvement actions that are relevant to the school context, for trial and evaluation.

1. How will ERO manage school expectations within existing resource constraints?

It was clear that principals held high expectations of the new approach. They had formed these expectations from regional presentations, materials developed and disseminated by ERO and from conversations with their evaluation partner.

Most schools valued the emphasis on collaborative approaches to school evaluation. They contrasted the new approach with the previous approach where reviewers only came to the school once every three or more years to conduct reviews. Following a few days of data collection, reviewers would then share findings and write a report. For most schools in this case study these episodic reviews generated anxiety amongst staff and were not useful for school improvement. In the new approach they saw their evaluation partner as a critical friend supporting their evaluation and improvement efforts and shared a view that the evaluation partner will have a more regular presence in the school. 

In the initial Research and Development phase the fifteen evaluation partners worked with five schools. As the approach is scaled each evaluation partner will each be required to work with up to 40 schools.

Senior leaders questioned the extent to which evaluation partners will be able to maintain a collaborative and strong professional relationship if the evaluation partner is working with a high number of schools. There is a risk that schools will become increasingly cynical about ERO’s commitment to collaboration and co-design if this is not practically feasible.  

There is a need to consider how ERO can balance the need to be responsive to schools within the limits of resources. It is clear that some schools will require more support than others to build capability in evaluation. This points to the need for identifying criteria for assessing the level of support a school needs. Criteria for differentiated support will ideally be based on an understanding of school context and need, but not necessarily be based on overall assessments of school performance. An example from one of the case studies may be helpful here. Kaitaia Intermediate school was already receiving intensive support from the Ministry and from an educational consultant. This school may not benefit from EROs focus on evaluation capability when this may already be part of the existing support work. The development of an evaluation plan may add another layer of distraction or confusion for the school, particularly if evaluation mechanisms are built into the SAF’s work. In these circumstances it may be preferable to support the work of the SAF and return to the school after completion of this work to evaluate outcomes. 

Recommendation

1. It is recommended that ERO identify criteria for differentiating the level of evaluative support provided to schools. Criteria will ideally be based on school need and context for improvement. Schools and evaluation partners will need to understand the opportunities and limits of time allocated to various aspects of the evaluation process.

 

The external evaluator documented the experience of a limited number of internal school stakeholders in the case study process. The focal perspective of the case studies came from the principal and associate principal of the schools. The external evaluator was only able to speak with other internal stakeholders - a group of parents, students and a Board of Trustees Chair[9] - in three schools. 

It appears that evaluation capability building with the evaluation partner, at this stage, has largely been limited to working with the principal and associate principal. Support for evaluation capability building at the classroom or teacher level was not yet evident. 

Two of the principals communicated a clear plan to cascade learnings to teachers. Their leadership and influence may support uptake by teachers and enhance the skills of staff within the school in using classroom evidence to inform improvements, but this is not guaranteed. 

Recommendation

2. It is recommended that a process be developed to support monitoring of the spread of evaluation capability across the school in development of evaluation plans, generating evidence from the classroom, and implementation of actions that lead to improvement in student outcomes.

3. How will ERO align its work with other partners also work to help schools improve?

The role of the Ministry and the Education Review Office are different, but complementary. The Education Review Office (ERO) independently conducts reviews of schools for learning, school improvement and accountability. ERO is not responsible for managing the performance of schools, but supports schools in improvement through evaluation that contributes to planning, action and outcomes.

The Ministry of Education provides specific guidance to schools, and identifies resources that will support schools in the form of professional development, and through provision of infrastructure and development funds. The Ministry of Education also announced the establishment of Te Mahau (formerly the Education Service Agency) as part of the re-design of the Ministry. Te Mahau incorporates opportunities for partnerships that progress outcomes for schools and students. 

Educational Improvement is a team effort. Evaluation partners are part of teams within ERO, but also may be a part of teams from other organisations or providers working with the school on improvement plans. Improved collaboration among and across teams from the Ministry and ERO and other providers will increase the collective impact of their work with schools, and result in a coordinated effort for improvement.

Recommendation

3. It is recommended that ERO identify strategies to support collective, inter-agency ways of working on improvement with schools.

4. How will ERO build and maintain its internal capability in evaluation?

The findings point to the importance of the technical, practical and relational skills of evaluation partners. Evaluation partners require knowledge about evaluation theory, and its implications for practice. They need strong interpersonal skills to be able to develop professional relationships and manage challenging conversations. They need to focus on practices that will make the most difference to the school in generating and using evidence for improvement.

Evaluation is a profession, and like any profession it takes time to develop knowledge and skills.  Templates and plans are helpful, but they will not guarantee high quality evaluation practice. ERO has a high proportion of new evaluation partners. Understanding the knowledge and skill needs of these partners and identifying development opportunities will be important.

While evaluation partners may attend evaluation conferences and external opportunities for professional learning, not all of these will be relevant to ERO’s context - a context that requires adaptive expertise in balancing the school’s internal evaluation requirements with requirements for external accountability. As a learning organisation ERO needs to also turn the evaluation lens towards itself to monitor its performance in contributing to improved school outcomes.

Five of the six schools indicated they trusted the evaluation partner and welcomed their support in improving their evaluation capability. They felt their partner was knowledgeable and skilled in evaluation. This report does not challenge that perception, but it does raise questions about how ERO monitors and evaluates the knowledge and skills of the evaluation partners.  How will ERO know that the evaluation partner has the required skills to support good evaluation for improvement?[10] 

Recommendation

4. It is recommended that ERO identify a suite of strategies to strengthen the capability of evaluation partners (for example, through specialist internal evaluation programmes, targeted mentoring and through learning opportunities within and across evaluation teams in the three regions).

5. How can ERO support schools in internal evaluation and support external accountability?

Most school leaders represented in this case study of implementation of the new approach referred to short-term outcomes that believe had already occurred as a result of their work with their evaluation partner. Schools in the case studies expressed confidence that the entire evaluation process will be shaped by collaboration and co-design. In their view the collaborative base will contribute to school buy-in to the evaluation process, and build their knowledge and skills along the way.

While building the internal capability of schools in evaluation for improvement, ERO retains the role of ensuring that schools are accountable to their communities and their students. ERO will need to communicate the implications of its external accountability role in relation to public reporting and the limits of co-design in this context.  

Recommendation

5. It is recommended that ERO establish external reporting requirements that retain a focus on documenting the school context as well as highlighting their performance story over time. The expectations of external reporting should be clearly communicated to schools early in the evaluation process.

6. How will ERO know it has made a difference at the school level?

Most school stakeholders suggested that the evaluation partner was already making a difference to the school in terms of learning about evaluation, and co-constructing an evaluation plan that aligns with the school’s strategic direction. 

They shared examples of increased knowledge about evaluation and its role in school improvement, and improved evaluative and critical thinking skills. These outcomes are important in maintaining internal school capability and are also examples of process use of evaluation (Patton, 2008). Engaging in evaluation may trigger changes in thinking and actions that contribute to wider changes within the school. 

Specific comments highlighted that the evaluation partner had spent time building the relational base for the evaluation. While this is important in communicating the parameters of the new approach, there is a risk that in focusing on process issues, there is less focus on intent and outcomes of the evaluation work: equity and excellence for all students. 

Relationships between ERO and the school are important, but not the intended outcome of evaluative work. It will be important to monitor and evaluate the contribution of the evaluation process to school improvement over time.

Recommendation

6. It is recommended that ERO identify ways to monitor and evaluate the contribution of their work to school evaluation and school improvement at the school, region and national level.

7. How will ERO know it has made a difference at the system level?

System-level evaluation provides a structure for assessing and reporting the relative performance of the education system as a whole. It helps determine how effective the system is in delivering education to students, and what difference those educational opportunities are making to students’ lives in terms of improving equity and excellence. Research and evaluation at a system level can highlight strengths and limitations and contribute to a national evidence base about what works in different school and community contexts. This information can then be used to develop policies, frameworks and practices to inform improvement. ERO generates a range of evidence from its work with schools. This cumulation of evidence is a very powerful driver for change.

Most senior leaders indicated that ERO’s system level research and learnings about what works, and in what school contexts and circumstances will be invaluable to them in keeping up with the most effective ways to support student learning and wellbeing outcomes. Schools are working in dynamic environments and while they may review research and evaluation reports, the time to translate the implications of these reports for practice at their school may be limited.

There are opportunities for ERO to share insights from national or system-level reports to inform practical strategies for schools to apply on the ground. Schools can then draw on strategies that have worked in similar contexts to address school priorities.

Evaluation partners may draw on these findings to inform local discussions within the school and to point schools to relevant examples of school improvement.

Recommendation

7. It is recommended that ERO make findings and insights from national or regional evidence more accessible and widely available to schools. The evaluation partner may directly draw from this evidence base to engage schools in discussion of improvement actions that are relevant to the school context, for trial and evaluation.

Conclusion

The school stakeholders involved in this case study of implementation were generous in sharing their perspectives on the new evaluation approach. They indicated a confidence that the collaborative, high-trust foundation will strengthen the relevance and use of evaluation for improvement. The prior approach to evaluation was seen to be focused on judgement and based on narrow review process.

In the new approach evaluation partners work to build and maintain a collaborative, professional relationship to support schools to use data and evidence effectively, assess where they are at, identify the right strategies, and track progress. Principals shared insights that reflect a growing sense of agency in collecting, analysing and using data to inform their own improvement efforts through their work with their evaluation partner. They were not passive recipients of ERO reviews; they were active partners throughout the process. 

A statement by Lee over twenty years ago points to the value of evidence for both internal and external accountability: “Schools that have taken hold of the role of the evaluator - those who really own and use their school data – are best able to tie internal accountability to external accountability. If people in schools can see the relevance of data collection to their own goals and concerns, they will begin to value and use the evaluation process – {and} have an array of information to support demands for external accountability.” (Lee, 1999)

It is the use of evaluation for improvement that matters. An evaluation focus on equity and excellence provides a critical lens to ensure that schools in New Zealand are meeting the learning needs of students. It is still clearly early days for schools and for ERO in implementation of the new approach. The process has begun to assess if the new way of working will result in sustainable shifts in school evaluation mechanisms and school improvement. 

There is already strong evidence that the collection and use of data by schools can be a driver for improvement of schools and student outcomes. Evaluation is not merely conducted by, or for an external group as a requirement, but becomes infused within the culture of the school.  Ownership of the evaluation process and its outcomes allows schools to document the school’s story and context, profile achievements, and strengthen leadership in making changes that benefit students, staff and the wider whānau and community.

 

References

Braun, V., and Clarke, V. (2022). Thematic Analysis: A practical guide. Los Angeles: Sage

Cousins, J.B., & Chouinard, J.A. (2012). Participatory Evaluation up close: An integration of research-based knowledge. Evaluation and Society. North Carolina: Information Age Publishing.

Lee, L. (1999). Building capacity for school improvement through evaluation: Experiences of the Manitoba School Improvement Program Inc.  Canadian Journal of Program Evaluation, 14 (special issue), 155-178

Patton, M.Q (2008).  Utilization-focused evaluation, 4th ed.  Thousand Oaks, CA: Sage

 

---

[1] The case study component was expected to be finalised in September. Due to Covid-19 restrictions and lockdown the contract was extended to December 2021.

[2] The schools are identified in this report with the permission of the principal. Photos are also included to provide a break from text and capture elements of the character of the school.  Approval was gained for photos that include images of students.

[3] All schools were in the early phases of the evaluation process so could not comment on their experience with the entire evaluation cycle/process.

[4] The principal was on sabbatical leave at the time so did not have the opportunity to participate in the construction of the plan. 

[5] The external evaluator attended one joint evaluation meeting on zoom, but due to internet connectivity issues the meeting was limited to the evaluation partner and two support people who work within the school, the SAF and a PLD provider from Cognition Education. Follow up interviews were conducted with these two individuals separately, in addition to interviews with the acting principal and deputy principal.

[6] It is my understanding as the external consultant that the evaluation partner had not yet progressed to formalising an evaluation plan. However, they had written up the areas discussed as potential areas to focus the evaluation.

[7] The perspective of the evaluation partner is missing from this case study. It is likely they have interpretations that can shed light on the disruption to the evaluation process. Each evaluation partner was interviewed before the external evaluator visited each school, in order to gain an understanding of the school context, and initial experiences with the school. Follow up interviews with evaluation partners after school visits were not in scope.

[8] The associate principal has schoolwide responsibility for teaching and learning.  They are supported in this role by the principal and two other school leaders.

[9] These were schools that the external evaluator was able to visit in person.  Information was collected over Zoom during lockdown for the three other schools.

[10] ERO provided a range of professional learning for evaluation partners in 2021.  However, the impact of lockdowns and Covid-19 restrictions meant that many of these opportunities were online and not face to face.  Mentoring new evaluation partners in practice was also limited due to travel restrictions.