External Evaluation of the School Improvement Approach Phase 1 and Phase 2

External Evaluator: Dr Delwyn Goodrick

June, 2021

Purpose

• The purpose of this document is to provide a summary of key findings from the initial phases of the external evaluation of the implementation of the Education Review Office (ERO) Schools: Evaluation for Improvement Approach.
• Findings are based on survey feedback from the first group of Evaluation Partners/Review Officers (December 2020), interviews with internal leaders (April-May, 2021), and observations of a two day hui in Wellington (December, 2020).

Background

• An external evaluation is running alongside the development and implementation of the Schools: Evaluation for improvement Approach. It focuses on the first group of schools involved in the process.
• The external evaluation will be completed in September 2021. This evaluation will inform internal decision making and provide the basis for feedback to the sector. The evaluation is explicitly formative in purpose.

The external evaluation will investigate:

• The extent to which the approach supports education improvement
• The successes and challenges experienced by evaluators in implementing the new approach
• Key stakeholders’ views about the strengths and weaknesses of the approach
• Resources and components needed to support consistency of the approach across schools

• There are three phases to the external evaluation

1. Phase 1- Feedback from Evaluation Partners/Review Officers and external evaluator reflections (completed)
2. Phase 2 – Feedback from internal stakeholders (19) (completed)
3. Phase 3 –Data collection at the school level about the new approach (commencing May - August 2021). This phase involves a survey of the initial group of 75 schools1 and nine case studies of implementation.

Summary

The key findings from data collection activities to date are:

1. There is strong support for the new approach

• ERO’s move toward a more participatory and collaborative approach to school evaluation for improvement was welcomed by evaluation partners and internal stakeholders. The approach has the potential to
  • build school capacity and commitment to evaluation
  • create a shared focus on improvements, and
  • ensure that evaluation is relevant to schools.

2. Implementation of the initial rollout of the approach has been slower than anticipated

• The initial phases of the new approach have taken longer than anticipated. The Evaluation Partners/Review Officers have retained a strong focus on building a relational foundation with the schools.
• Implementation of the approach with the initial group of schools has been affected by CoVid-19 restrictions. This has particularly influenced implementation in the Te Tai Raki region.
• ERO has extended the timeframe for the external evaluation to capture learnings and experiences of schools across phases.

3. Internal Messaging about the new approach

• Evaluation partners and some internal stakeholders pointed to a lack of clarity about the implementation process. Some of the confusion was associated with shifts in language used to frame the approach (for example, prototype, trial, initial pilot) and differences in messaging.
• ERO has a range of existing resources that support schools in internal evaluation. These resources need to align with school needs and school requirements for reporting.

4. The new approach requires a sophisticated range of evaluator skills

• Evaluation partners require education- specific knowledge, evaluation knowledge and skills, relational skills, cultural and contextual responsiveness.
• Internal stakeholders within ERO expressed respect for the Evaluation Partners’/Review Officers’ skills. They acknowledged that the new approach requires a range of expertise in both education and evaluation domains but expressed concern about the preparedness of all Evaluation Partners/Review Officers for the demands of the work.
• As the number of schools expands Evaluation Partners/Review Officers will need to balance the requirements of working with a broader portfolio of schools, with the focus on relational and responsive ways of working.
• The legitimacy of Māori and Pacific cultural values, knowledge, and methodologies need to be explicit in evaluation work with schools. Internal expertise and experience in Kaupapa Maori within Maori medium schools is an invaluable resource for evaluators. It is important that Evaluation Partners/Review Officers understand, appreciate and incorporate these principles into their evaluation practices with schools.
• The Evaluation Partners/Review Officers have valued the professional learning and development (PLD) opportunities made available to them. Some planned sessions were de-railed by CoVid-19 restrictions.
• At the regional level a range of structures are in place (Professional Practice Groups, meetings, book clubs, mentoring) to support professional practice.
• A lot of work is being done internally to develop tools and templates to support Evaluation Partners/Review Officers but resourcing of this work has been inadequate.

5. Practical Concerns about how the approach will work

• Evaluation Partners/Review Officers expressed concern that some of the practical logistics of the approach had not been worked out sufficiently prior to initial implementation. This uncertainty posed challenges for some of the Evaluation Partners/Review Officers in their communications with schools.
• Managers expressed concern about the wellbeing of Evaluation Partners/Review Officers in navigating relationships with schools while maintaining a critical focus on improvement. With a larger number of schools Evaluation Partners/Review Officers may experience even more pressure. Managers acknowledged the importance of providing additional support and mentoring to Evaluation Partners/Review Officers.

6. Demands of multiple layers of change

• There is value in acknowledging the professional and personal shifts required of Evaluation Partners/Review Officers and of schools in implementing the new approach. Developing a methodology with integrity for schools, for ERO and for the system will take time.
• Other internal changes that have occurred alongside the roll out of the approach have exacerbated some of the implementation issues. For example, during the initial roll out ERO has also been implementing a digital platform, which has involved changes to documentation and recording processes. A large proportion of staff within ERO are relatively new to the organisation, and in some cases to evaluation. These staff have required additional support to navigate the role requirements.

7. Demands on leadership and managers have shifted

• The new approach has required Managers Review and Improvement Services (MRIS) to perform an expanded role in support and coaching of evaluation partners. ERO has responded to this by implementing a coaching initiative for all leaders and managers to equip them with the skills to support individuals and teams.

8. Perception that schools welcome the new approach

• Evaluation Partners/Review Officers, Managers and internal leaders indicated that they had received strong support for the approach from the school sector.
• The evaluation capability needs of schools and strategies to address these needs within schools requires additional resourcing.
• Responsibility and resourcing for enhancing the evaluation capability of schools needs to be shared across the system; it is not solely ERO’s responsibility.

9. System-level reporting needs to reflect learning at the school level

• There needs to be a clear alignment between school level and system level reporting to ensure the relevance of the approach to the sector.
• System-level reporting needs to reflect learnings from evaluation at the school level.

10. Leadership of the Change Agenda

• Leaders within ERO need to keep in touch with how things are working on the ground.
• ERO needs to adopt a learning orientation that encourages open sharing of what has worked, and opportunities for improvement to demonstrate that the organization is able to walk the talk of evaluation for improvement.

Recommendations

A small number of recommendations that follow from the first two phases of the external evaluation are presented below.

1. Messaging about the new approach

There will be some evaluation partners/reviewers who are more or less comfortable with the progressive, developmental nature of implementation of the new approach. Some evaluation partners have experienced inconsistencies in the description of the approach, and uncertainty about requirements in terms of working with schools.

It is recommended that the opportunities and challenges of the research and development phase are reinforced in written and verbal communications. The roll out of the new approach is progressive and changes in processes and practices take time.

It is recommended that hui/meetings allow structured time for evaluators to reflect on, and discuss key issues they have experienced in implementing the evaluation approach.

2. Support for the professional learning of Evaluation Partners/Review Officers

The Schools: Evaluation for Improvement approach represents a significant shift in the way Evaluation Partners/Review Officers work. The professional skillset includes strong relational skills, project management and scheduling skills, knowledge of effective education practice and evaluation skills.

It is recommended that tailored professional learning opportunities be made available to Evaluation Partners/Review Officers to strengthen their knowledge in school evaluation and their confidence in their capacity to support schools.

3. Strengthen managers skills in supporting good practice in evaluation

It is recommended that managers are provided with additional support to manage the structural and process requirements of the new approach. Managers may also require support in responding to the professional and interpersonal needs of Evaluation Partners/Review Officers in managing their workload and expectations.

4. Support for Schools

Learnings from initial implementation suggest that many schools need to build or extend evaluation capacity. Opportunities for this need to be considered and ERO will need to listen to feedback from Evaluation Partners/Review Officers and schools.

It is recommended that ERO is deliberate in gathering feedback from Evaluation Partners/Review Officers and the school sector to identify useful approaches to build evaluation capability. This feedback will also generate an understanding of the required resourcing to support schools.

Footnotes

¹ The survey response rate has been exceptional. Eighty-six percent of the schools involved in the initial implementation of the Approach have responded. As the survey remains open till the 18/06/2021 findings from the survey have not yet been formally analysed