Principal Perspectives- Schools: Evaluation for Improvement Approach

Section 1: The External Evaluation

The external evaluation of the Schools: Evaluation for Improvement Approach involves three phases.

Phase 1: Documentation of experiences of Evaluation Partners/Review Officers in implementing the new approach (December 2020-January, 2021)

Phase 2: Feedback from key stakeholders within the Education Review Office (ERO) about implementation of the new approach to school evaluation (February-March, 2021)

Phase 3:  Feedback from the 75 schools involved in the first implementation of the approach. This phase involves a survey of principals in all schools (the focus of this report), and a select number of case studies (April-September 2021).

Phase 1 and 2 have been completed. This summary report provides an analysis of key findings from the survey distributed in April 2021.

 

1.1 About the survey

The survey was developed to elicit perspectives of principals involved in the initial implementation of the Schools: Evaluation for Improvement Approach in 75 schools across New Zealand. The survey items were developed in collaboration with the Education Review Office to ensure questions reflected key domains of interest. The survey incorporated Likert scale items, and open-ended items.

Open-ended questions provided an opportunity for principals to contribute additional detail about their ratings of the scaled survey items, and to add further insights about their experience. Most principals took the opportunity to share additional feedback.

Both the quantitative scaled items and the qualitative open-ended items were analysed.  Descriptive statistics were used for the scaled questions, and thematic analysis for the qualitative questions.

The survey achieved a very high response rate. Eighty-eight percent of principals completed the survey.

1.2 Response Rate

The survey achieved a very high response rate. Eighty-eight percent (66/75) of principals in the initial implementation of the new approach responded to the survey. The response rate indicates that the findings reliably represent the views of the principals involved.

 

1.3 About the Principals who responded

The average number of years respondents had been working as a principal in the school was nearly six and half years with a range of less than one year experience in the role to 23 years of experience.

 

Section 2: Survey Findings

2.1 How long have the schools been working with their Evaluation Partner on the new approach?

Implementation of the new approach with an initial group of 75 schools began in September 2020. 

A survey designed to elicit perspectives of principals involved in the initial group of schools was distributed in April 2021.  It was envisaged that schools would be at least part way through the five phases of the new approach to school evaluation by this time.

Principals were asked when they started working with their Evaluation Partner/Review Officer.  Most respondents indicated that they had started work with their Evaluation Partner/Review Officer in October and November 2020. Ten of the 66 principals who responded indicated they had not formally begun working on the new approach till December or January 2020. 

Due to the starting timeframe and the time Evaluation Partners/Review Officers dedicated to engaging with schools, most schools were still in the early phases of the new approach (that is, Exploring and Focusing, and/or Designing). 

Several principals noted that they were unable to provide a comprehensive assessment of the new approach as they had not yet progressed through a full evaluation cycle.

Implications:  Evaluation Partners/Review Officers spent a considerable amount of time building the relationship with their schools.  Initial stages were focused on building a strong relational base and socialising the new approach with school partners.  As the approach represents a substantive shift in the way ERO undertakes evaluation with schools, the time spent with schools in the Exploring phase seemed important to establishing a collaborative foundation.  

Each of the evaluation partners/review officers were at this initial phase working with up to 15 schools. With the growing number of schools allocated to each Evaluation Partner/Review Officer, there will be a need to rationalise the length of time spent in initial engagement activities.

 

2.2 Who has been involved in the evaluation process?

Principals were asked to provide feedback about which groups in the school and community had, thus far, been involved in their school’s evaluation process.  The key groups who had been consulted or engaged were the senior leadership team, including the principal.  Forty-three of the 66 schools had also involved their Board in deliberations on the evaluation plan.  Whānau groups and the broader community had been engaged in the process in 21 of the 66 schools.

Implications:   A key focus for the Education Review Office is equity and excellence.  It is anticipated and expected that schools will engage with their school community, and with parents and the wider whānau in the community.  This does not appear to have occurred in most schools implementing the new approach.  This finding points to two key questions:  When do schools believe it is most appropriate to engage these groups in the process of identifying priorities for improvement?  How do schools engage with whanau within their school community?  It will be useful to identify successful engagement strategies and better understand how these strategies contribute to school priorities.

 

2.3 Principal Ratings of the Schools: Evaluation for Improvement Approach

The Schools: Evaluation for Improvement approach represents a different way of undertaking school evaluation.  It shifts ERO’s focus to a collaborative and ongoing relationship with the school and ERO working as partners to generate evidence to support school improvement.

Principals were asked to rate a number of elements of the approach on a 1-5 scale, 1 indicating a very low rating, and 5 indicating a very high rating.  Dimensions included:

• opportunity for our school to collaborate with ERO in the new approach
• openness of communication with ERO
• ease of working with the Evaluation Partner/Review Officer
• value of access to evaluation knowledge and expertise
• usefulness of the resources shared with the school
• alignment of the approach with the school’s strategic plan
• value of the approach in identifying opportunities for more effective practice for school, and
• overall usefulness of the process to the school

This report focuses on major patterns in principals’ ratings, and supplementary feedback generated from the open-ended questions.

Forty-eight of the 66 principals rated the overall usefulness of the approach to the school as high or very high.   Principals were generally optimistic about the potential of the approach and valued the opportunity to be engaged in a more collaborative approach to school evaluation. 

Seventy-three percent (48/66) of principals rated the overall usefulness of the approach to the school as high or very high

There was strong support for the collaborative platform. A total of 56 of the 66 principals valued the opportunity to collaborate with ERO during the evaluation process.  Nearly all principals (60 of 66) indicated the openness of communication with ERO was high. 

One principal stated,

“I like (it) and I am supportive of the new approach.  It is far more collaborative and relational than previous approaches.  It also gives ERO the opportunity to share ideas and offer support.” (Respondent 14)

Open -ended comments about the approach clustered around three themes

1. Value of the relationship
2. Value of open discussion and co-construction
3. Alignment of the process with the school’s plans for improvement.

 

1. The value of the relationship formed with the Evaluation Partner/Review Officer.  Principals reported that the Evaluation Partner/Review Officer made a genuine effort to get to know the school context. There was a lot of positive feedback about ERO’s focus on developing trust and building the relational base of the work. Some principals had valued the time dedicated in the initial phases to building the relationship. 

Evaluation Partners/Review Officers were often mentioned by name, with some principals expressing gratitude their school had been allocated such a knowledgeable and skilled partner.  Schools that had a change of Evaluation Partner/Review Officer during the process found this destabilising and changes in personnel had clearly contributed to delays in progressing evaluation plans.

2. The value of open discussion and co-construction.  Principals valued the critical discussion with the Evaluation Partner/Review Officer about the school’s improvement journey.  Discussions provided an opportunity to check-in and critically reflect on the school’s improvement plans.  One principal stated, “What has worked well is us inviting our evaluator/mentor to spend a day with us talking to a variety of staff about how we do things in our school and showing her evidence of this.  Having an outside pair of eyes to support our critical reflection around our self-review of an aspect of our local curriculum.” (Respondent 59, Principal 3 years)

Some principals expressed uncertainty about how ERO was going to balance the co-construction and accountability function - If the evaluation was co-constructed, how would external accountability be managed?

3. The alignment of the evaluation process with the school’s existing plans for improvement. Principals welcomed the focus on aligning evaluation to strategic plans already in place within the school, or that were in the process of development.  In their view this alignment has the potential to reduce duplication of effort and contribute to the usefulness of the evaluation process.

In the next sub-section the focus is on two ratings – alignment of the approach with the strategic direction of the school, and the value of the approach in informing school improvement.

 

2.3.1 Alignment with strategic direction of the school

The approach involves the Evaluation Partner/Review Officer working with the school to understand the school context, and to support the strategic directions of the school. Fifty-eight of the 65 principals indicated that the alignment of the approach with the schools’ strategic direction was strong.  

Eight-nine percent (58/65) of principals considered that the alignment of the approach with the school’s strategic direction was strong

Alignment contributes to the relevance of the evaluative work being undertaken and may also ensure strong school buy-in to evaluation for improvement. 

Comments from two experienced principals illustrate the value to the principals:

“I like the fact that the approach links to our strategic goal that we (BOT, staff, parents/whanau) have already decided are important for our school.  It will be interesting to see how the approach is monitored and I am REALLY interested to see how the community and stakeholders will be informed of processes and outcomes.” (Respondent 51, principal 14 years)

“We have established a strategic plan which is in the third year of implementation.  We have learned a lot along the way and gathered extensive data to help us make evidence-based decisions while pursuing our goals to have all students reach their potential. Our relationship with ERO began at a time when we were analysing the results, setting goals and forming our strategies for 2021…We are hoping that this relationship will prove to be very helpful in supporting us to make changes we have identified in our school.  Our Evaluation Partner has captured where we want to go accurately and we are looking forward to next steps.” (Respondent 13, principal 10 years)

2.3.2 Value of the approach in identifying more effective practice for school improvement

Sixty-nine percent (45/65) of the principals rated the value of the approach highly in generating information that will support school improvement.  Open-ended comments revealed that principals valued the opportunity to critically reflect on school progress in improving outcomes for all students. 

One principal shared that the discussions with the Evaluation Partner/Review Officer had contributed to increased cohesiveness among the senior leadership team about required areas of improvement. She commented

“...We have valued the timeliness from strategic plan discussions to using the ERO approach to improving our alignment to ERO indicators, the richness of the SLT being able to be all on the same page, and line of sight with our plan, and have valued input from others from ERO.” (Respondent 6, principal 3 years)

Fig 1 summarises the data from these two survey items. 

Figure 1: Setting the scene for school Improvement

 

School Capability in Evaluation

Two survey statements sought information about the school’s self-assessment of their evaluative knowledge and skills.  They were:

• Our school has well-established processes in place to support ongoing internal evaluation
• Our school uses a range of evaluation information to prioritise areas for improvement

About half of the principals agreed or strongly agreed that their school has well-established processes in place to support ongoing internal evaluation.  Fifty of the 66 principals agreed or strongly agreed that their school uses a range of evaluation information to prioritise areas for improvement. 

Given the overall positive response to evaluation support from Evaluation Partners/Review Officers, it is clear that principals believe that there are still areas for improvement. 

 

Implications:

How do schools evaluate their capability and capacity?  On what basis and using which criteria? What is ERO’s role in contributing and supporting evaluation capacity?   In the process of co-construction how will differences in assessment of capacity or capability reconciled?

Fifteen of the principals rated the resources shared with them by the Evaluation Partner/Review Officer as low or very low.   Analysis of open-ended comments indicated that this rating may have been associated with an absence of resources possibly attributed to the phase or perceived disconnect between the resource and the school's assessment of their needs.

While most schools feel that their school has existing evaluation capability it is clear that ERO’s role in supporting school capability in evaluation was still considered important.  Forty-six principals of a total of 65 principals indicated that ERO is strengthening evaluation capability through the approach.

Seventy one percent (46/65) of principals indicated that ERO is strengthening evaluation capability within the school

Fig 2 presents the ratings for this statement.

Figure 2: Ratings for ERO’s role in strengthening school capability in evaluation

 

The role of Leadership Partners

The Leadership Partners programme is a strategic initiative developed by ERO and the sector to develop capability in evaluation for improvement.  A number of senior school leaders expressed interest in being part of the initiative.  The group receive professional development in evaluation and work alongside ERO review teams with schools. Leadership Partners make a valuable contribution to the evaluation process and the initiative contributes to the building of a wider network of expertise across the sector and within ERO.

Principals were asked to indicate if their school was working with a Leadership Partner and if so to share perspectives about their value to the evaluation process. Twenty-five of the 66 reporting working with a Leadership Partner and 10 were unsure if they were or not.  The benefits identified of working with a Leadership Partner were their experience in schools and their capacity to quickly relate to their context, and the contribution they made to discussions about school improvement

Survey responses indicated that some principals may have conflated the term ‘Leadership Partner’ with’ Evaluation Partner/Review Officer.’  For example, some indicated the school was working with a Leadership Partner, but included open-ended comments referring to the name of their Evaluation Partner/Review Officer.  The data generated from this question should therefore be reviewed with some caution. It is clear that the language associated with the new approach is still settling. 

Implication:  It is clear that some principals did not differentiate between leadership partner and evaluation partner in responding to this question. 

The conflation of the titles makes it difficult to ascertain principals’ perspectives on the Leadership Partner programme.  The external evaluator understands that there is a separate evaluation project focusing on the programme.

 

The Approach- What is working well and what isn’t?

4.1 What has been working well so far?

Several principals indicated that it was too soon to judge what was working well about the new approach.  Most mentioned the quality of the engagement and the genuine interest the Evaluation Partner/Review Officer had expressed about getting to know the school and its context as a key benefit to date. Key themes elicited from an analysis of the open-ended questions are summarised below.

• Value of the collaborative approach – Principals valued the open communication and a sense of a partnership that underlies the new approach.  They appreciate the focus on co-construction over time, rather than a compliance focus over a shorter period.   An interesting, and alternate view of the allocation of time over a longer period was offered by one principal.  In his view a lengthier process of engagement was contributing to a decrease in staff momentum for change. This may be associated with the staff perceiving less urgency for change.
• A genuine interest in listening to the school’s story – “The ability to share our story as a school and as a community.”  Some principals noted the value of thought-provoking conversations, and indicated that the approach formalised opportunities for critical reflection in a busy school environment.  The focus on understanding by the Evaluation Partner/Review Officer appears to have contributed to a level of trust and confidence in sharing the school’s improvement journey. A relatively inexperienced principal commented, “We are always evaluating, but I can’t say that we always get it right.  Being honest in our review, evaluation has been positive in the sense of not being judged negatively. Open sharing and discussion of practice has been great.” (Respondent 6)
• Alignment with strategic and annual goals –The focus on aligning reviews to the school’s strategic plan was valued. The underlying message in many open-ended comments was that this strengthened the usefulness of the evaluation process, and reduced duplication of effort.  One principal commented that the evaluation process was working well in “refining our evaluation processes, questions and evidence sources to support specific evaluation of our progress and successes for learners and learning. (It fits with) where the school sees itself and what is important to the school.”  (Respondent 15)

 

Example Quotes: What is working well so far?

“Alignment with Strategic and Annual Goals - our everyday work. Refining our evaluation processes, questions and evidence sources to support specific evaluation of our progress and successes for learners and learning.” (Respondent 15)

“Sharing. Bouncing ideas. Being challenged. Having someone to share thoughts/ideas/wonderings with.” (Respondent 47)

“The collaboration and positive professional relationship developed… Her knowledge and ability to capture our discussions and focus our thinking. The opportunity to dig deep into what we do well and what our challenges might be.” (Respondent 41)

“A positive partnership that is being set up to exist over a longer time frame. Having this expertise available for us to work with.” (Respondent 46)

4.2 What has not worked so well?

Principals were also asked to share their perspectives about what wasn’t working so well thus far in the new approach.  Less positive comments appeared to be related to the perceived lack of clarity about how the new approach would work, delays in moving through the phases, and a tendency for ERO to fall back on the traditional ‘outsider’ approach to school evaluation.

Principals acknowledged that the new approach was accompanied by a level of uncertainty.  They were aware that phases and expectations were being developed as they were being implemented.  For some this was expected and appropriate, but for others the lack of certainty brought with it some confusion.  One principal commented,

“We are in uncharted waters (and that) means it lacks some shape and this can feel uneasy for staff who have a concrete mindset.  Not knowing what comes next, a lack of clarity about the phases.” (Respondent 3). 

A related concern was about consistency of messaging.  There appeared to be still some confusion which was epitomised by perceptions that there were different understandings of the approach. This appeared to be particularly the case if principals had heard different or conflicting messages from different sources, for example, other principals or others within ERO. For example, one principal stated, “… some schools appear to know more than others.  (I am) not sure where to look for information in regards to the new approach.” (Respondent 19)

While principals valued the collaborative platform and time taken by the Evaluation Partner/Review Officer to understand the school context, some felt these initial phases had taken too much time, and delayed progressing through the evaluation.

Other principals noted their concerns about their level of access to their Evaluation Partner/Review Officer and the likely frequency of visits given the number of schools the partner will be required to support. A small number of the principals identified that they were aware that their Evaluation Partner/Review officer’s time was stretched and questioned how the collaborative nature of the evaluation process could be sustained over time.

Several principals expressed concern about reporting and reporting requirements  At the time the survey was distributed most schools had not reached the reporting phase.  While recognising the value of continuous improvement, some principals were concerned about what the ‘end’ stage would look like.  It was clear that principals were aware that while ERO is supporting the school evidence base to inform improvement, some judgements about the school will inevitably be made.  

Example quotes: What is not working so well so far?

“I still don’t really understand what the end result will look like, how we will be judged. (we are still) making sense of the parts of the process.” (Respondent 5)

“(Challenges in) developing a clear understanding of the expectations of how the compliance process part will work.”  (Respondent 8)

“While our evaluation partner is friendly, professional, and has abilities in her field of assessment and evaluation, I do not feel the involvement of ERO in projects we already have a clear direction on will be inherently value-adding.” (Respondent 48)

“The evaluation partner has been very open, honest and appreciative with our kura.  However, my staff team still feels they are being reviewed as opposed to having someone walk alongside them.  Learning how to partner takes time.” (Respondent 22)

What can ERO do to better support the school in evaluation?

The comments shared by Principals about opportunities for improvement were particularly instructive.  Not all principals responded to this question.  Some principals expressed the view that as it was still ‘early days’ in the process, it was difficult to make recommendations for improvement of the process.

 

This section summarises key messages from the survey findings and identifies some implications for improvement.

Clarify expectations of reporting

Principals understood that they were involved in the implementation of a new way for ERO of working with schools to build evaluation capability and capacity.  They recognised that any new approach takes time to stabilize, and that not everything can be prescribed in advance.    However, it is clear from the survey findings that greater clarity about the phases and expectations is needed.

While they recognized that the approach is designed to be collaborative and that the evaluation plan represented a co-construction between the school and the Evaluation Partner/Review Officer, they were unsure how this would actually work, particularly with reporting that may be external.

Further clarity about expectations would also contribute to a more efficient process for ERO and for the school.  For example, one principal commented, “A bit more insight into what each step entails would help us to be better prepared and know what to do, and achieve between visits.  This would ensure that the time we have with our evaluation partners can be maximised, rather than doing things that we could have done ourselves first, then shared with her for feedback”. (Respondent 36)

Most schools in the first group of schools had not yet reached the reporting phase but expressed interest in understanding what reports would not look like given the emphasis on co-construction. This raises two issues for consideration:

1. How will the balance between improvement and accountability be determined and reported?
2. If the reports are to be co-constructed, how will this practically work?   Will schools have the opportunity to respond about the appropriateness of any external judgements?

 

Schedule meetings and associated accountabilities

Evaluation Partners/Review Officers are required to balance the demands and challenges of working with multiple schools to support them in the evaluation process. Evaluation Partners/Review Officers need to accommodate schools’ timeframes and timelines, while also supporting the momentum of the evaluation process.

Principals recommended that a clear schedule be agreed and documented so that the school could prepare well for the Evaluation Partner/Review Officer’s visit.  A clear pathway through the evaluation would also ensure schools and ERO are clear about their accountabilities.

 

Offer opportunities for professional learning in internal evaluation

Some principals noted the value of ERO reports and documentation about the principles and practices of school evaluation. However, on their own such publications are not likely to be effective in supporting school in improving their evaluative capability or capacity. The role of the Evaluation Partner/Review Officer in prompting use of these documents will support the translation of principles into practice. The school may also benefit from learning more about the resources that are available to them to assist them. Better support structures are required to promote school self-evaluation.

Two comments illustrate the value of professional learning programs:

“Are there training workshops on internal school evaluation?  There is an assumption schools know all about this, partly this has been created by ERO distributing booklets and assuming these are read and understood. In reality this is not the case. Training school management, face to face in how to internally review their own schools would set the process up for even greater success.” (Respondent 38)

“Continue the publications, continue with this model of walking beside schools.  Encourage schools to look at their process as a growth exercise and not an evaluative judgement exercise, therefore actually looking at what needs to happen to grow. (Respondent 29)

The two principals who shared these comments about the potential value of offering further opportunites for professional learning had an average of 4.5 years experience as a principal.  It may be important for ERO to target activities according to experiential base of the principal and the school. On-going capacity building activities will also be necessary to ensure the sustainability of evaluative expertise, and to extend that expertise.  

 

Showcase promising practices

Evaluation Partners/Review Officers work with a range of schools.  They gain an understanding of successful evaluation and improvement strategies that have been adopted by schools.  Over time the knowledge base accumulates about what works in particular school contexts.

Principals want to hear about good practice, in part to save ‘reinventing the wheel’ and also to consider trialling strategies that have worked elsewhere. There was a strong interest in hearing about best or promising practices being implemented in other schools.  Principals that made this suggestion indicated that information could be shared about good practice without identifying the school. Evaluation partners could also share these insights and link schools working on similar issues. The comments presented below provide practical guidance as to how this might work.

• “Share more models of effective evaluation that ERO deems suitable.  That is better than playing second guess what is in my head.” (Respondent 30)
• “Bring knowledge of evaluation and that lens of working across a number of schools and location.” (Respondent 45)
• “Give some real examples of good practice to go and view, speak to that relates to our goal.” (Respondent 17)

 

Conclusion

The impressive response rate to the survey indicates a level of engagement and interest in providing feedback to ERO about the new approach.  Principals are generally very supportive of the intent of collaboration and co-construction, but are still uncertain, and in some cases wary about how it will work. 

It is clear that the approach has great potential and that the focus on building a strong relational basis has generally been working well to date.  Ongoing consultation with schools will enable ERO to continue to be responsive to school needs.  Improvement of evaluation mechanisms within ERO and across schools will ensure that there is a strong evidence base to contribute to actions that will improve equity and excellence for all students across New Zealand. 

Appendix 1: Raw Data - Survey Dimensions

Dimension	VERY LOW	2	3	4	VERY HIGH	TOTAL
Opportunity for our school to collaborate With ERO in the evaluation process	1.52% 1	4.55% 3	9.09% 6	31.82% 21	53.03% 35	66
Openness of communication with ERO	1.52% 1	0.00% 0	7.58% 5	27.27% 18	63.64% 42	66
Ease of working with our evaluation partner	1.52% 1	3.03% 2	6.06% 4	33.33% 22	56.06% 37	66
Value of access to evaluation knowledge and expertise	1.54% 1	7.69% 5	21.54% 14	44.62% 29	24.62% 16	65
Usefulness of the resources shared with our school	9.23% 6	13.85% 9	27.69% 18	33.85% 22	15.38% 10	65