Evaluation of the Turnaround Schools (TAS) Pilot Program

Key findings from the external evaluation

Overall Implementation Processes

• Stakeholders consulted in this evaluation noted a range of differences in the underpinning philosophy and activities of the TAS pilot from ERO 1-2 reviews.  The major points of demarcation included 
• the pilot’s emphasis on the development of a collaborative working relationship with schools, with the Ministry and NZSTA,
• the focus on using monitoring and evaluation for school improvement rather than primarily for accountability, and
• use of the TAS reports produced each term for advocacy for additional support or resourcing
• A collaborative platform was established from the design and inception of the pilot, but the way that communication occurred between agencies was sometimes patchy. The evaluation found evidence that the levels of collaboration between ERO and the Ministry improved over time.
• Ministry of Education representatives expressed support for the work of the TAS team in the schools. The term by term reviews mobilised action by the schools to address performance issues.
• ERO Directors from the three regions acknowledged the importance of working closely with the schools.  However, one of the directors suggested that the pilot was overly-ambitious and questioned the focus on school evaluation as the key mechanism for school improvement. 
• Directors indicated that strategic communication about the progress of the pilot did not always occur within ERO, which made it difficult to assess how the pilot or the schools were progressing.  

Implementation at the School level

• The evaluation found evidence that the team were effective in working in partnership with the schools. The relational approach, and the continuity of the team anchor in each school appeared to be important in supporting critical reflection.
• Regular contact with the schools enabled reviewers to gain a deeper knowledge of the school context over time. Demonstrated understanding of context appeared to positively influence the level of professional trust between reviewers and the schools.
• It was evident that the match of particular reviewer skillsets to school needs was an important consideration.  The skills of the reviewers and their fit to the school enhanced professional trust and supported respectful practice. A clear example of this was drawing on the knowledge, leadership and skills of Māori colleagues to facilitate connections and conversations within schools.  Two school stakeholders also spoke of the translational skills of Māori reviewers in the TAS team.  These reviewers assisted them in understanding the issues identified in TAS reports. Pākehā reviewers in the team valued the opportunity to work with Māori colleagues.  They believed that co-working positively influenced their own cultural understandings and were a rich source of professional learning.
• Review reports validated the perspectives of school stakeholders, and generally were considered accurate reflections of the issues that needed to be addressed. However, there was a view there was a mis-alignment between verbal and written reports on some occasions. Where this had occurred principals were able to voice their concerns, and modifications were made -if appropriate - to the written report.
• School stakeholders perceived a tension between the team’s role as evaluators vis a vis provision of support and advice. Most of the stakeholders interviewed suggested it was a lost opportunity for influence with schools if ERO maintained an exclusive focus on monitoring and evaluation. 
• Over the three years of the pilot the TAS team developed a range of resources to support schools to sequence actions for improvement. The School Evaluation Indicators (ERO, 2016) were elaborated to ‘unpack’ progressions for schools classified at the lower end of the rubric.  Qualitative radars were developed to map progress on a range of dimensions from review observations and data collection.  These tools may be useful as progress markers for other evaluation work undertaken by ERO with schools.

Outcomes

• Evaluation evidence indicated that the TAS pilot was effective in accelerating improvements within five of the six schools. Organisational dynamics, including the commitment and readiness of leaders within each school influenced the rate and pace of change. It is likely that the context of the school and its relationship with the wider community will also influence the sustainability of improvements. 
• Identified improvements within the schools were not solely attributed to the pilot.  A range of support initiatives had been put in place within these schools during the pilot timeframe. The cumulative value of the initiatives by ERO, the Ministry and NZSTA had created the conditions necessary for improvements to occur. 
• School stakeholders considered the pilot was of value to the school. The average rating of the value of the pilot to school improvement was 4 out of a possible 5.
• School stakeholders (principals and board chairs) indicated that the review and reports each term had created an urgency to respond. While in most cases the reports were seen as accurate reflections of the school, school stakeholders expressed feeling overwhelmed by the number of issues they were required to address in the termly reports.
• There is evidence that the TAS pilot has strengthened knowledge about ways schools can use data to assess progress. While most school stakeholders viewed this positively, two of the principals reported that the ‘relentless’ focus on particular kinds of data was a distraction from tailored activities they were developing and/or trialling to improve student engagement and learning.
• Ministry stakeholders highlighted the value of the TAS reports in identifying additional resourcing and support needs within schools.  The reports provided a robust evidence base. In most cases it appears that the Ministry was able to use the reports to expedite support to schools.

An overview of quantitative and qualitative changes in schools from baseline to 2019 is presented in Table 1.  This report focuses on stakeholder feedback about the role of the pilot in contributing to school improvement.

Table 1: Summary of Pilot school improvements

School	School-level changes – Quantitative outcome data since baseline (2016-2019)	Improvements - Qualitative
School 1	- Improvement in NCEA Level 3. - NCEA Level 2 is solid with those students improving from Level 1. This is where the school has put their focus. - Growth in community connections and leadership	- Whānau structures developed to support learning -Teacher capability has increased in the school - Implementation of curriculum modules to promote engagement and literacy, and enhance vocational/academic aspirations
School 2	-No formal quantitative data available. Note: In 2020 it was recommended that a Commissioner be appointed to this school as the school had not demonstrated sufficient improvement	-Local curriculum focus on local Schooltanga and community tanga in learning design -Schoolwide focus on student wellbeing
School 3	-Years 7 to 8 and Years 9 to 10 classes are receiving more appropriate teaching programmes focused at their level of learning.   - Teacher capability in math and evidence of improvements in mathematics	- Distributed leadership model promoted consistency in strategic communication within the school, and may support sustainability - School and board interest in using data and enhancing evaluative thinking - Whole school numeracy focus
School 4	- Improvement NCEA Level 1, 2 and 3 - Marked improvement in NCEA Level 1 2017 to 2018. (At least 100 students at Years 11, 12 and 13) - Reduced disparity to national decile band data for each level - School roll is steadily increasing	- Strengthened leadership capability of Senior Leadership team and Heads of Faculty - Improved school systems, particularly in data monitoring and tracking. - Re-invigorated board with growing capability in governance - Rigorous curriculum Review-Improved range of pathways for students
School 5	- Improvement NCEA Level 1 and 2. Stable Level 3.  The school is now close to decile average. - By the end of 2019 more students were completing external NCEA standard - Leavers with at least NCEA Level 1 has steadily increased (however, still below national and decile 1) - School roll is growing rapidly	- Significant shift in school culture and community engagement - Whānaungatanga - positive changes in school climate and evidence of strong student voice within the school - The newly appointed board chair has a strong focus on performance measurement
School 6	-Improved NCEA Level 1 Literacy/Numeracy	-Local curriculum development is progressing -Staff capabilities in te reo Māori, maths and literacy -Whānaungatanga/positive school climate

Opportunities for Improvement

Stakeholders identified opportunities for improvement of the pilot approach that – from their perspective – would have made the pilot more effective.  Five key actions were identified from a thematic analysis of feedback. They were:

• Clarify roles and responsibilities and provide ongoing communication mechanisms to ensure all stakeholders are clear about purpose, scope and role differentiation.
• Improve the alignment between verbal and written reports.
• Build on the collaborative platform with schools and include opportunities for co-construction of reports.
• Actively share promising practices about what works for school improvement with the schools to increase their knowledge and learning, and reduce duplication of effort, and
• Sharpen criteria for evaluation of progress and exit of schools from intensive monitoring and review.

Recommendations

The following recommendations are intended for consideration by ERO. They relate to the strategic and structural elements of school evaluation approaches for schools experiencing persistent performance challenges.

1. It is recommended that ERO maintain a dedicated group of experienced evaluators to focus on schools with persistent performance challenges.  The extension of the pilot to a greater number of schools through the High Priority Schools (HPS) approach in 2020 is formal recognition by ERO of the importance of this work. However, the continuation of the pilot level of resourcing (time and scope of work with schools) is likely to be unsustainable as work is extended to a greater number of schools. ERO will need to identify strategies to balance internal resource constraints with their capacity to influence change.
2. It is recommended that the TAS team’s role be extended to provide additional support and/or PLD in monitoring, evaluation and using data for improvement purposes.  The team has developed a sound knowledge base about what works with Turnaround schools within the NZ context, and this information will continue to be useful for ERO, the Ministry and for schools.
3. It is recommended that the learnings from the TAS pilot be used by ERO and the Ministry to inform the work undertaken to shift performance in other underperforming schools. 
4. It is recommended that schools – principals and potentially board chairs - be provided with an opportunity to share lessons learned, and to highlight promising practices that support school improvement.  The success case profiles in this report highlight some topics that potentially could be explored and elaborated. Principals in the six schools indicated they would welcome an opportunity to share experiences.  A joint forum with ERO and the Ministry would also provide a further opportunity for learning about useful strategies that have worked in similar school contexts. 
5. It is recommended that the tools and resources developed over the past three years by the TAS team be shared more widely across ERO and the Ministry. Additional work may be required to provide guidance on the use of each tool, and the purpose and process of use with schools to ensure they are used appropriately and with fidelity. Some of these tools (e.g., the school radars) may be particularly useful in bringing together qualitative assessments and judgements of the review team, with quantitative school outcome data.
6. It is recommended that evaluation mechanisms are built into school improvement approaches to allow for progressive formative feedback.  The explicit inclusion of process evaluation within any school improvement approach also has the benefit of demonstrating that ERO and the Ministry ‘walk the talk’ of evaluation for improvement and learning.
7. It is recommended that a simple map of the improvement phases be developed to increase school and board understanding of progress markers for withdrawal or ‘dial-down’ of the intensity of engagement with ERO.

 

Structure of the Report

The report is divided into five sections.

Section 1 provides a brief overview of the policy context and rationale for the Turnaround Schools Pilot Approach.

Section 2 summarises the purpose and methodological approach adopted in this external evaluation. An evaluation data matrix presents the sources of evidence and evaluation methods. Additional detail about the evaluation methods and analytic processes is included in the accompanying appendix.

Section 3 provides an overview of the objectives of the pilot and describes the major components. This section also presents a programme logic map developed during the evaluation, which describes the ‘pilot on a page.’  It summarises policy drivers, key phases and pilot activities in schools and aligns these to intended outcomes. This section also provides a brief description of each of the schools in the pilot, the implementation activities, and stakeholders’ initial responses.

Section 4 is the main section of the report.  It presents detailed findings about stakeholders’ views of the pilot and the way it worked, and key elements of effectiveness.  The three core outcome domains that were examined in the external evaluation were:

1. Schools have an enhanced capacity to gather and use data for planning,
2. Schools find the TAS reports useful for planning for improvement, and
3. Schools have necessary supports in place to progress planned improvements

Promising practices identified by school stakeholders or ERO reviewers working with the schools are highlighted in this section of the report.   The presentation is explicitly strengths-based. The section also outlines opportunities for improvement.

Section 5 summarises the lessons learned, key implications and recommendations emerging from a synthesis of evaluation findings.

 

Section 1: Introduction and background

All young people within New Zealand deserve access to a quality educational experience. A key role for schools is to provide the environment and conditions to support students to develop skills that will prepare them for further study or work.  Schools also play an important role in supporting the social and emotional wellbeing of students.

However, a small percentage of schools within New Zealand are failing to provide students with equitable access to high quality learning experiences and students are not achieving expected outcomes. These schools, invariably are located in Decile 1-3 areas and have a high proportion of students who identify as Māori or with Pacific ancestry[1].   The continued poor performance of these schools widens the gap in achievement and threatens equitable outcomes for all New Zealanders.

In 2011, ERO developed a differentiated review methodology that focused reviews to school context, performance and evaluation capacity.  While most schools were subject to reviews every three to five years, some schools were identified as requiring a more regular review every one to two years. 

The schools classified as Repeatedly Poorly Performing (RPPS) often demonstrated poor performance in five areas (which correspond to the 2016 School evaluation indicators).  The areas were:

1. Equity and excellence of student achievement outcomes
2. Other valued student outcomes including student engagement and wellbeing
3. The quality of teaching and curriculum design and opportunities to learn
4. Professional leadership
5. Stewardship – notably the cycle of planning reporting and resourcing, and/or health and safety

The Ministry and ERO work together to evaluate the quality of education from a common framework to enhance student outcomes.

The Education Review Office (ERO) independently conducts reviews of schools for learning, school improvement and accountability. Poorly performing schools are primarily identified through ERO’s evaluations. Schools that are experiencing performance challenges and low student achievement outcomes are placed on a longitudinal review process. While ERO conducts a review of these schools every one to two years, ERO is not responsible for managing the performance of schools.

The Ministry of Education also has mechanisms to identify schools that are poorly performing or at risk of poor performance. Regional advisors within the Ministry monitor school performance through annual reports provided by the school on academic achievement, school absence, enrolment and operational issues.  The Ministry of Education provides specific guidance to schools, and identifies resources that will support schools in the form of professional development, and through provision of infrastructure and development funds.

1.2 Policy drivers for a new approach

A number of policy drivers provided a mandate for a new approach to ‘circuit break’ the decline of RPPS and RDS schools and support improvement.

The revision of the Education Act in 2017 included reference to the need for ‘more graduated range of interventions’ to ensure swift support was available to schools.  There was an acknowledgement that while the support needs to be targeted to have most impact, it cannot be short-term or piecemeal. The Act also supported alignment in processes and practices between the Ministry and ERO.  As noted above both agencies have complementary, but different roles in school evaluation and school improvement.  As a result of the update to the Education Act, both agencies have been working on new protocols, systems and processes for working together.

Signal Loss, a NZ initiative publication pointed to the need for the education system to urgently address the ongoing underperformance of schools. While the report acknowledged the contextual conditions that influence student outcomes, it made a strong case for intentional use of data and evidence to inform improvement.

The report identified that "some schools, despite intervention, perform poorly for as long as, and in some cases, longer than, the entire school career of their students - with possibly serious implications for the students in them and the state of our nation".

A report by Tomorrow’s Schools Independent Taskforce (2019) identified a range of systemic changes to support improved outcomes for students. There was a call for a shift to a high trust education system to spread of effective practices, and supports ongoing improvement.  The report called for ERO, the Ministry and other agencies to work in coordinated and relational ways with schools to support improvement.

These policy drivers provided the authority for a  new way of supporting schools that were experiencing ongoing performance challenges.

1.3 Rationale for The Turnaround Schools Pilot

The Turnaround Schools (TAS) pilot was implemented from 2017-2020.  The strategic purpose of the pilot was to develop and trial an appropriate approach to identify the root causes of poor performance, and collaboratively work with schools and the Ministry to support school improvement. The stated purpose of the TAS pilot was to ‘intervene, disrupt the decline, highlight the systems that are required to shift, build internal capability and capacity, increase the momentum for change and support the school to enter a recovery phase.’ (School Turnaround Evaluation Methodology). 

An assessment of pilot documentation for this external evaluation indicates three different, but interrelated objectives of the pilot. 

The first objective was to develop and implement an approach to interrupt the decline of poorly performing schools. Monitoring and evaluation by the team was identified as the key lever or catalyst for improvement.   The pilot was designed to contribute to a school’s evidence base about organisational, practical and academic issues that were influencing student outcomes, so that schools could plan and implement strategies for improvement.  

The second objective was to develop and implement an approach to draw on the system and sector supports required for these high priority schools.  There was a recognition that a single agency approach would not be sufficient.  ERO could not ‘turn schools around’ with monitoring and evaluation mechanisms without the support of the Ministry.  The Ministry needed evidence of school performance to identify support requirements. The collaborative platform between the two agencies and NZSTA was identified as key to success. 

The third objective of the pilot approach was to enhance evaluative capability within the schools.  One indicator of improved evaluative capacity may be the school’s use or intention to use data to inform improvement. It was envisaged that through the pilot the six schools would have the opportunity to learn about the value of data, how data can be used for planning and improvement through term by term reviews, and implement changes that will create and sustain improvements.

1.4 Timeline for Development

The timeline and key activities of the TAS pilot from initiation to implementation with the six schools are presented in Fig 1.

Fig 1:  Development Timeframe

 

Section 2:  Evaluation of the TAS Pilot

The External Evaluation of the pilot

An external evaluation of the pilot approach was commissioned in March 2020.  The pilot evolved into the High Priority Schools (HPS) in mid 2020 and the team has extended their reach to a wider number of schools.  The focus of the external evaluation is on the TAS Pilot 2017-2020, not the HPS project. 

The evaluation fieldwork for the TAS external evaluation formally commenced in July 2020[2]. The external evaluation was explicitly framed as an evaluation of the TAS approach, not of the schools.

This report presents the findings of an external evaluation.  It describes perspectives from key stakeholders about TAS components, their effectiveness, strengths and weaknesses and opportunities for improvement. 

Interviews were conducted with 26 stakeholders, including Principals and Board Chairs of the six schools, ERO Regional Directors and individual members of the TAS team, Ministry representatives from each region (managers and some advisors) and LSMs (where present in the school).

The report presents key findings from the evaluation. It is intended that the findings and recommendations in this report will be used to shape decisions about the merit and worth of the pilot.  It is envisaged that these learnings will also inform the continued work of the HPS team, and contribute to the knowledge base within ERO and the Ministry about what works to support improvement in schools.  

The key evaluation questions orienting the evaluation were:

1. How was the TAS pilot intended to work to support school turnaround?
2. How effective was the Turnaround Schools (TAS) Pilot in supporting school improvement?
3. In what ways could the approach be improved to better reach schools with performance challenges?
4. What are the lessons learned about effective strategies for school improvement within persistently low performing schools?

2.1 Framing the Evaluation

The evaluation questions require knowledge about the work undertaken by the TAS team with the schools, and the outcomes that were achieved through pilot activities.

This external evaluation focuses on the contribution[3] of the TAS pilot to changes from the perspectives of a range of stakeholders. Secondary data obtained from TAS review reports, and student outcome data provided a further source of evidence. 

2.2 Evaluation Methods and Sources

The methods and sources of information generated through the evaluation included:

• Interviews with each member of the TAS team, about their role in the team, and schools they were appointed to as the TAS anchor.
• Visits to four of the six schools[4].  A key part of the visit was to gain an understanding of the context in which the school was located.
• Interviews with all six principals and Board chairs.  In two schools with relatively new Board chairs, the previous Board chair was also interviewed.
• Semi-structured interviews with the Manager of Education, and Directors in all regions.  These interviews were primarily conducted primarily by Zoom, and in some cases managers, and advisors from the regions attended to provide their perspectives.
• A group interview with the three regional directors within ERO
• Secondary document analysis and review of ERO reports and relevant, support documentation.
• Three evaluation forums with ERO and Ministry stakeholders, and with the TAS team to review findings and recommendations.

The schools that participated in the TAS pilot are not identified in this report by name.  The decision was made to de-identify the schools to protect their privacy.

A summary of the evaluation methods and sources according to key evaluation question is presented in Table 2.

Table 2: Evaluation Questions and Data Matrix

Key Evaluation Question	Information Requirements	Data Sources	Method of data collection or retrieval
1. How was the TAS pilot intended to work to support school turnaround?	-Description of the TAS model/approach and assumptions. How was the TAS approach different from other ERO reviews? How was the TAS pilot intended to work to support school turnaround?	- The TAS Team (6) - Regional Directors (3) - Pilot documentation	-Zoom individual interview.  -Zoom small group interview -File review
2. How effective was the Turnaround Schools (TAS) pilot in supporting school improvement the six schools?	-Assessment of shifts in student outcomes from baseline -Analysis of contribution: Views of the school representatives (Principal and Board) involved. – Perspectives of Ministry partners and regional directors within ERO	-Quantitative data available on school outcomes from baseline (2016). Including NZEA level 1 and Level 2, Attendance data	-Retrieval from TAS Files and secondary analysis -Interview x 2 with TAS team member with responsibility for data support)
3. In what ways could the pilot have been improved?	-Identification of conditions associated with success -Stakeholders’ views of strengths and weaknesses -ERO reports/partnerships	- Regional Directors (ERO x 3) -Regional Directors (5), Manager (4) Advisors (2) - Ministry of Education Manager -Principals (6), LSM (2), Board Chairs (5) plus immediate past Board chair (2 schools) -Feedback forums (x3)	-Zoom small group interview - Zoom interview and one face to face interview -Face to face interview (1) -Face to face (11) and Zoom interviews (4)
4. What are the lessons learned in terms of what works to support schools with ongoing performance challenges?	-What were the changes that occurred in schools that are attributed to the pilot? -Learnings from the pilot -What is the likely sustainability of these changes given objectives, scope and resourcing?	- Regional Directors (ERO x 3) -Regional Directors (5), Manager (4) Advisors (2) - Ministry of Education Manager -Principals (6), LSM (2), Board Chairs (5) plus immediate past Board chair (2 schools) -Feedback forums (x3)	-Zoom small group interview - Zoom interview and one face to face interview -Face to face interview (1) -Face to face (11) and Zoom interviews (4)

2.3 Scope of the Evaluation

Data collection activities for the external evaluation occurred over a four-month period – from late June to September, 2020.  All primary data collection and analysis was undertaken during this time.   For most of the TAS schools the start of 2020 was their third year of involvement with the TAS pilot.  

Principals of the six schools were interviewed as well as the current board chair, and immediate past board chair (if appropriate). Interviews with boards as a governance group, and interviews with Whānau or community members were outside of scope of this external evaluation. 

As a key partner, Ministry representatives were also interviewed at the National leadership level, and at the regional level. Group zoom interviews were conducted with regional directors, managers and advisors.  A small group interview was also held with regional directors of the three ERO regions.

The evaluator had access to a range of other pilot documentation, including project files, progress notes, email communications, and reports over time, and school level information about each of the schools involved. In addition, secondary data, that is quantitative data on achievement in NCEA, school leaver data and attendance data was retrieved from ERO reports to supplement the extensive primary qualitative data gathered during the evaluation.

This report focuses on the effectiveness of the TAS pilot from the perspective of key stakeholders. 

2.4. Audience and Key Stakeholders

The key audiences for this evaluation are Nick Pole (CEO) and Jane Lee (Deputy Chief Executive, Review and Improvement) of the Education Review Office, and Jann Marshall in her role within the Ministry of Education as the Acting Associate Deputy Secretary, Sector Enablement and Support.  The report is intended to inform decisions about the merit and worth of the TAS pilot, and to identify lessons learned that may inform further joint work with schools experiencing performance challenges.

2.5 Limitations of the Evaluation

The evaluation relied on a range of methods and sources to generate claims about the value of the program.  However, there are a number of limitations that need to be acknowledged.  

No observation of the TAS team in practice: The evaluator gathered information from the schools about their experience with the TAS pilot, and their perspectives on the way the TAS team worked with schools, and with the Ministry.  Although the key activities of the TAS team were described by the team, and by stakeholders, no observations of school reviews were conducted. 

No consultation with community members: A school is part of its community.  From the perspective of many stakeholders, the school cannot be considered in isolation from the wider community. The scope of the current evaluation did not allow an opportunity for consultation with members of the wider community.

No opportunity to consult with NZSTA personnel working in the schools:  The evaluator recognises that the education system represents a partnership among a range of agencies and personnel.  In the TAS pilot this includes the schools, and communities in which the school is embedded, the Ministry, ERO and NZSTA. There was no scope in this evaluation to conduct interviews with NZSTA personnel who had supported any of the six schools.  Similarly, there was no opportunity to consult with any other individuals who supported school improvement (e.g., SAFs, curriculum advisors).

Reliance on either self-report data or accounts of change: The qualitative data gathered during the evaluation painted a rich picture of how the TAS pilot worked with schools. Interviews with managers and school stakeholders were helpful in understanding the context of each school, and perspectives on the school’s improvement journey. However, these perspectives were not substantively validated by any other source.

While quantitative data was retrieved for key quantitative outcome measures over several years to allow some comparative analysis, the outcomes are at the whole school level, and do not necessarily show the granularity of change within the school, nor do they demonstrate the contribution of the pilot to school outcomes. Data is not included in this report to ensure that schools are not identifiable.  Only global patterns are presented.

2.6 A note about attribution and contribution

The TAS pilot was a key mechanism for gathering evidence about the issues of concern within the school, and for tracking progress. The external evaluation gathered information to describe and document the implementation, and assess the effectiveness of TAS in improving school performance in the six schools. The focus of the external evaluation was on stakeholders’ views of the value of the TAS pilot in progressing change within the schools.   Stakeholders were encouraged to share their observations about the contribution of the TAS to these changes. Secondary data on student outcomes from baseline to 2019 were also drawn on to align perspectives with actual shifts over the past three years.

In public policy evaluation it can be challenging to disentangle the influence of an intervention or project from other influences in the context, such as other educational support provided during the pilot timeframe,  the progressive maturity of leaders and teachers, support provided by the Ministry and other agencies, and contextual changes emerging over time[5]. 

Initiatives external to the TAS pilot will also have contributed to observed school improvements.  For example, the engagement of an LSM (in three of the six schools), the appointment of a new Board, or implementation of curriculum support specialists are other key supports of change. While it cannot be claimed that any improvements are a direct result of the TAS pilot, triangulation of data from schools, the TAS team and Ministry representatives builds a credible evidence base for claims made about effectiveness in this report.

Section 3:  What is the TAS Pilot and how does it work?

3.1 A programme logic map for the TAS pilot

A programme logic map (or logic map) was developed during the evaluation process to provide a structure to data collection and to summarise the pilot on a page.

A logic map is a one-page depiction of the relationship between drivers, inputs, activities and intended outcomes.  In other words, it shows the relationship between what is done and what happens as a result.  These maps are useful in understanding an initiative's intent, but also for developing tailored performance measures and focusing evaluation data collection options. The logic map informed the development of propositions about the relationship between TAS activities and outcomes that could then be subsequently tested during the evaluation.

As the external evaluation evolved, the logic map was elaborated. The map is presented in Figure 2.  The diagram should be read from left to right. 

The external evaluation focused attention on the relationship between the activities undertaken by the TAS team and short-term outcomes.

Fig 2:  Logic Map

3.2 What were the key drivers for the new approach?

There are a range of issues that cumulatively pose barriers to improvement in schools classified as repeatedly poorly performing schools (RPPS) or those in rapid decline (RDS).

Many of challenges identified in RPPS and RDS relate to one of the following performance dimensions:

• Health and Safety
• Stewardship and Governance
• Leadership
• Teacher motivation and performance
• Quality of teaching and responsive curriculum
• Positive learning culture and learning environment

Conventional review strategies have not generally been effective in turning around performance in these schools.  Reviews focus attention on indicators, and inform schools of barriers that are inhibiting performance, but reviewers may not have sufficient time to identify the root causes of performance challenges. Standard 1-2 year reviews may also not be sufficiently intensive to mobilise action, or build the school’s capacity for improvement.

The architects of the TAS pilot recognised that the conditions that influence student outcomes are often found in the school – its structure, processes and practices. Attention must be paid to growing the capacity of the school if sustainable changes are to be made.

A set of propositions[6] underpin the pilot and were considered as part of the evaluation.  The propositions were:

• That an intensive review (each term) will be an effective strategy to ‘turn schools around’
• That a collaborative platform will increase the school’s ownership of required improvements
• That the dose of the pilot (termly reviews) will be sufficient to mobilise school stakeholders to progress improvements 
• That an experienced group of reviewers is required to work with schools experiencing performance challenges.  The proposition to be considered is that  the specialist skills of the team will be a good fit for the school and its wider context
• That a dedicated ‘anchor’ for each school will facilitate trust and continuity
• That a phased approach to evaluation between key agencies - the school, ERO and the Ministry is required to build a solid evidence base for improvement, and support school capacity in monitoring and evaluation
• That the outcomes achieved are sufficiently valued by stakeholders to warrant the level of investment.

3.3 How was the TAS Approach different from 1-2 year reviews?

ERO monitors, evaluates and report on educational outcomes for children and young people.  Evidence gathered from reviews can be used as a ‘catalyst for change’ for improving schools and the wider education system within New Zealand. The historical focus within ERO tended to privilege the accountability function of evaluation, which was appropriate in the context of the organisation’s role. The TAS pilot was different.

A major difference of the TAS approach from 1-2 year reviews was its intensity. Reviewers from the team returned to the school each term to collect and analyse data.  They were able to initiate a staged approach to improvement with the schools.

The ongoing interaction with the school required a different approach to the professional relationship.  Reviewers focused initially on building a collaborative platform, between the reviewer and the school, and with the Ministry. 

Each review was scheduled with the school, and the process discussed with the school and regional managers. At least two reviewers participated in each review. One reviewer was appointed as the anchor for the school to ensure continuity.

Reviewers modelled evaluative skills and encouraged the schools to use data for school planning and for tracking improvement.[7] The team was able to triangulate their observations and interpretation of key issues to sharpen judgements. The feedback conversations were not always easy- for the reviewers or for the schools.  Reviewers were able to present issues clearly and assertively with the confidence that other members of the team would back them up. The reviewers made a concerted effort to focus on a small number of issues so as not to overwhelm the school.  There was a tension here on occasion where there were several issues that needed to be addressed for accountability reasons (for example, health and safety issues within the school and other issues posing potential risk). 

While ERO has committed itself as an organisation to be culturally responsive, in TAS this focus was made very explicit. The team acknowledged the importance of Kaupapa Māori and initiated ways of working to operationalise the principles into practice within culturally diverse settings.  This work has been aligned with He Taura Here Tangata Strategy[8].

A report was completed as soon as possible following the review. Often, reviewers sent emails with a summary of the review.  The majority of principals in the six schools had experienced an ERO review before in previous work in other schools. 

The three quotes presented below were shared by school principals involved in the pilot. They powerfully highlight perspectives about the difference between a ‘standard’ ERO review and the TAS review process

Working together towards positive outcomes

‘…The ERO approach before this was that they come into the school, write some report and then they walk away. So, over the whole period of the (TAS) project, there has been a very strong level of support in the sense that we have always been working together towards positive outcomes for improvement in our school. And you can’t do that by just simply evaluating. You can’t just do that by measuring. It won’t work.’ (School 3 - Principal)

A very different experience

‘The model of ERO going into a school, creating havoc, walking away leaving the school to deal with the aftermath and all the blood on the floor, was one that the team discarded.  I guess supporting school improvement hasn’t been part of an [ERO] model, it’s always just been a bit “We’re here to review. We’re going to tell you what you’re not doing very well”…This pilot was a very different experience.’ (School 4 - Principal)

Warts and all conversations

‘(The difference for me was that) they (The TAS team) had a genuine desire to work with us, which I really liked…In all the visits, we were able to critique and challenge any of the things that they had found or that we didn’t agree with. We developed a kind of trust where we were not sort of frightened of having a ‘warts and all’ kind of discussion…We’re working on this together. We are all wanting the same thing’ (School 5 - Principal)

3.4 Underpinning Principles and Approach

The school turnaround methodology was based on principles of Developmental evaluation (Patton, 2011), Rapid review approaches to monitoring and evaluation (Kumar, 1993) and Kaupapa Māori.[9]

Developmental evaluation (DE) has much in common with an educational action research cycle, but is more explicitly focused on collaboratively identifying opportunities for innovation. DE allows for emergence and adaptation as an initiative unfolds. Following a recipe for improvement is not appropriate in this approach to evaluation. In an educational context the evaluator works with schools to identify, co-design and co-construct data to inform improvement. Creative ideas are celebrated, trialed and evaluated progressively.

The use of rapid review approaches complements Developmental evaluation in that these approaches support the ongoing process of improvement and organisational learning. Rapid approaches are particularly relevant in contexts where timely feedback on progress is required to support the next improvement step.  In the TAS pilot the team recognised that strategies were required to generate and share data effectively and efficiently each term with the schools.

It was acknowledged by the team that the TAS methodology and implementation of the pilot would necessarily evolve over time.  As the team became more experienced in undertaking specific reviews, there was increased attention paid to the cultural responsiveness of the team’s approach[10].  The original Kaupapa (approach) that guided the TAS pilot explicitly identified the importance of cultural responsiveness and relational ways of working.  The TAS pilot aimed to be inclusive of Kawa-Tikanga (ways of doing and being), Whakaaro Hōhonutanga (Deep thinking), Whakawhanaungatanga (relationships and partnerships), Hitoria-Whakapapa Tangata (History, genealogy and Connections, and Kia Māia, Kia Mataara (Courage and Leadership).  

These principles influenced how the team engaged with the school, how they built relationships within the schools and with their boards, and how they undertook data gathering, analysis and synthesis activities. The team recognised that they needed to be bold, and step up to challenging conversations.  But, they realised that without a platform of trust, respect, authenticity and transparency among all partners, sustainable change would not be likely.    

‘You’ve got a cultural dynamic, which cannot be ignored – the relationship between the Iwi and the school…Any approach needs to be mindful of  the wider context of the area...Whatever you do in that area from a cultural perspective you have to also consider the wider implications around the wider tribal boundaries.’ (Regional Director, Ministry of Education)

The TAS Team summarised principles and implementation stages of their approach in a range of documents produced through the pilot.  These resources were shared internally within the team to support review work.  Others have been shared within and across other teams within ERO. 

The team produced a summary diagram that define three elements that differentiate the TAS approach from the longitudinal review. The three elements are:

1. Critical underpinning principles – an acknowledgement of complexity and the need for clarity to drive actions within schools
2. Evaluative philosophy –The evaluation approach, informed by Developmental Evaluation and rapid cycle approaches, recognises that evaluators must be adaptive and responsive to context.  The reviewer’s role in part is to promote capacity within schools to identify workable solutions.  Different strategies may be required for different school contexts.
3. Improvement Actioning – A key role for the TAS team is to provide schools with evidence that supports action.  The evidence needs to be stepped and sequenced so that change plans are doable.  This provides a stabilising environment for improvement activities to occur.

The manager of the TAS team provided the evaluator with the following diagram to represent these key elements.

Fig 3: TAS Team Philosophy and Approach

3.5 How was TAS intended to work in practice?

As noted in an earlier section of this report, the pilot involved an intensive school improvement approach using the lever of monitoring and evaluation as a catalyst for change. The reports -based on data collection at the school each term -generated evidence of issues and challenges within the school that were inhibiting student outcomes. It was envisaged that schools, with the support of the Ministry, would then use this evidence to develop an action plan.  Term by term reviews provided an opportunity to map progress against the plan.

The pilot incorporated ten phases. Schools progress through the phases, developing local solutions to challenging performance issues.  While the ten phases appear to be linear, the practice was not. The team recognised that schools were at different places, and had differing needs. There needed to be a level of adaptation, and flexibility in implementation according to context.

Research on school improvement has identified that improvement efforts often fail to probe deeply into why schools have experienced decline in the first place (Duke & Hochbein, 2008; Meyer & Zucker, 1989; Stringfield, 1998). Without this knowledge solutions may not be well targeted or relevant to the school context.  Different school contexts may require different change mechanisms.

A key part of the initial phase was diagnostic with an explicit focus on root cause analysis.  The school evaluation indicators were elaborated within the context of poor performing schools. Sub-stages and actions were identified to support schools in setting priorities, in an appropriate sequence to guide their own assessment of progress.  This diagnostic phase provided an evidence base for the school to inform the school’s development of an action plan.  

The team was required to work swiftly during term reviews with the schools.  At the end of the face to face review, a verbal summary of key findings was presented to the principal and often to the board.  A report on progress, challenges and required actions was also produced from the review. 

Subsequent phases built from the diagnostic, and allowed the team to review progress against the plan and the previous review, and to facilitate capacity in the school’s use of data for planning and improvement.

A range of resources and tools were developed by the TAS Team over the course of the pilot. The use of particular tools was informed by an assessment of the relevance to the context and issues faced by the school.  One of the tools, the ‘qualitative radars’ for school improvement is a creative, visual way to depict observed shifts in key organisational conditions within schools over time.  The conditions are aligned with established markers of school performance in the School Evaluation Indicators including safety, wellbeing, stewardship, leadership, evaluation capacity, connections curriculum, climate, culture, and outcomes.  The radars have not yet been shared with the schools, but have informed briefings within ERO and with the Ministry.

3.5.1 The schools selected in the pilot

Six schools were nominated to participate in the pilot. The schools were all Tier one schools, which means they have not improved their performance over four years through at least two successive ERO one or two year evaluation cycles.  The schools included two large urban secondary schools, a medium-sized high school, a composite school offering years 1-13, and a very small rural primary school.

They were classified as repeatedly poor performing schools and/or schools in a rapid spiral of decline. The schools were located across New Zealand, five schools in the North Island and one in South Island.

A brief summary of the characteristics of the pilot schools is presented in Table 2. 

Table 3: School Location and Key Characteristics

The schools are not identified in this report for privacy reasons.  The presentation of the key characteristics of the schools (composition and school size) is therefore presented in general terms.

Table 3: Key characteristics of schools in the pilot

Schools in the TAS Pilot	Characteristics –
School 1	A medium-sized school in the north island.  The majority of students identify as Māori.  There are approximately 250 enrolled students.
School 2	Very small rural school catering for children in years 1-8.  There are under 50 enrolled students in this school who all identify as Māori.
School 3	Small rural school in the South Island catering to students in years 1-13.  The school population is approximately 150.
School 4	A very large urban secondary school of over 1000 students that offers years 9-13. The majority of the student population identifies as Māori or as from a Pacific heritage.
School 5	A large high school catering for students in Years 9-13. Most students identify as Māori. Nearly a third of the students identify as Samoan, Tongan or as Cook Island Māori.  The school has an enrolled population of approximately 500 students.
School 6	Small rural composite school catering for approximately 100 students across years 1-15.  The majority of the students enrolled in the school identify as Māori.

3.5.2 The Specialist Improvement Team of Review Officers (SIRO)

The intensity of focus, the challenges of working in a dynamic and complex school environment with diverse stakeholders, and the collaborative sense-making process that underpinned TAS required a skilled review team.  They needed to be experienced in school review contexts, possess strong relational skills, have the capacity to be adaptive (rather than reactive), and be able to bridge the role of evaluator and guide.

A team of five experienced review officers were seconded to work in the TAS team under the direction of a national manager.  For the first year of the pilot there were only 2 EFT personnel in the team with the other team members seconded from various regions across the country for TAS reviews as required. During this time these individuals were also responsible for completing existing reviews in their region. This contributed to some challenges in scheduling reviews, and added an additional workload on top of the work associated with the TAS pilot.  

A full-time national team was put in place in October, 2019. As of September, 2020 there were five full time equivalent reviewers, under the direction of the national project manager.

Attention was paid to aligning the skill set and review approach believed to be appropriate for each school.  Eight other ERO reviewers have been drawn into work on specialist reviews or specific issues with the core TAS team. For example, a Samoan reviewer led the special review of a Samoan bilingual programme in an English medium setting at one of the schools.

The TAS Team Anchor

One member of the TAS team was appointed as anchor for each school.  This ensured some level of continuity between reviews, and a point of contact for the school and the Ministry.

This individual was the first point of contact for the school, and that- where possible- this team member was available for each termly review to the school to promote continuity.  As the team was small in size, most schools had the opportunity to work with a generally consistent team across terms.

The core members of the team met together once a term to discuss progress in the six schools, consolidate observations, and plan next steps. These team meetings also provided the opportunity for reviewers to debrief about challenges and share strategies that had been effective in working with the schools. Informally, the team maintained regular contact via phone calls and email. 

A pilot generally evolves during implementation.  Initial plans may change as context may demand a different approach.  Strategies the team adopted with schools shifted as the team learned about what worked and what didn’t work.

There was a strong commitment to culturally responsive and inclusive practice from the beginning of the pilot, but how this was enacted developed over the timeframe.  In two schools, the TAS team lead was an experienced Māori reviewer, and initiated and conducted the reviews with a Kaupapa Māori lens based on lived experience and knowledge.  The Māori reviewer facilitated respectful conversations between Kaiako (educators), tamariki (children), and whānau (families) to inform school improvement. On specialist reviews involving Māori or Pasifika students, specialist Māori or Pasifika reviewers led parts of the review.

Consider who engages

‘You need to actually spend time to unpack what the context looks like and what the approach should be so that is responsive in a cultural way. That often means not just considering how to engage but (thinking about) who does the engaging.’ (Ministry of Education representative)

3.5.3 What did the team do?

The composition of the team that went to each school each term was carefully considered in terms of required knowledge, relational style and ‘fit’ with the school context. At least two team members attended each review to the school.  Team members were rotated across schools according to availability for scheduled reviews, and balance with other commitments.  The team, therefore, had some familiarity with all six TAS schools, and a deeper knowledge of one or two schools as its anchor person.  

Conducting the reviews in pairs or in small groups ensured that team members had the opportunity to triangulate observations with a colleague, and facilitated the team’s capacity to manage - sometimes challenging - feedback processes with school stakeholders.

During the one to two days of the review each term the reviewers consulted with the principal and the board, undertook observation and interviews with teachers and students, and documented observations about a range of dimensions, including teaching and learning, curriculum, health and safety, stewardship. Issues emerging from the review were shared and discussed with the regional manager or advisor within the Ministry.

There were 24 outputs required each year for the pilot. These equated to reports of reviews – either internal to the school reports or external public reports. In some cases lengthy email summaries of the major findings, progress of the school and issues for attention were generated and sent to the principal, board chair, LSM and Ministry representatives if they had attended the initial feedback meeting with the school.

3.6 What were the intended outcomes of the pilot?

The pilot was designed to develop and implement an approach that would progress outcomes within the six schools.  It was anticipated that through collaborative ways of working with the school that in the short term that the school stakeholders would demonstrate commitment and enthusiasm for change.  The TAS reports were intended to focus on the key issues within the school that were most urgent so that the school could progressively focus on actions for improvement.

An assumption was that the trusting professional relationship created through term by term reviews, and the provision of progress data would improve capacity for reflective practice and the development of targeted action plans.

The TAS team envisaged that the regularity of the reviews would lead school stakeholders (particularly the leadership team and the Board) to an appreciation of the value of qualitative and quantitative data in assessing school outcomes and improving practice.

In the intermediate term the working assumption was that the collaborative work with schools, and their access to additional support and resources will improve understanding of the attributes of quality teaching and learning practices. In time the schools will exhibit more confidence in applying these principles with students in the classroom.

In the longer term, outside of the initial pilot timeframe, achievement of these short and intermediate term outcomes was associated with the implementation of actions that will contribute to positive and sustainable changes in the school environment and shifts in student outcomes.  Schools will have developed the skills and the resilience necessary to navigate structural and contextual challenges in the future.

3.7 ­­­­ Context Matters – Turnaround Schools in Context

It is reasonable to expect that senior leaders, teachers and Boards want the best outcomes for their students.  They recognise that education can escalate young people out of social disadvantage.  But, they also recognise that the circumstances many of their students face at home, in the community, and in the school may inhibit full engagement with learning.

Principals of all the schools highlighted the importance of context to the success of any improvement strategy.  Lifting student outcomes in a secondary school setting is complex in circumstances when the students come to the school with lower than anticipated skills in reading, writing and numeracy.  School attendance may also be sporadic because of family commitments, care-taking responsibilities, or because of long-term disengagement with education.  

Young people in schools in these communities may experience social exclusion arising from a combination of social issues such as, family or intergenerational unemployment,  low family income, poor quality or overcrowded housing, high crime, poor health and family breakdown. These issues are often mutually reinforcing.  Compounding the problems is that many of the lowest-performing high schools are at a severe disadvantage because their feeder schools may also experience challenges. Students arrive in high school with major learning challenges, and this further threatens their level of engagement with formal education. While the academic problems in these schools are challenging, ‘the pervasive social problems of poverty — discouragement, distrust and low expectations — are often even bigger barriers to success.’[11] (pg 1)

Three examples shared with the evaluator by the principals portray the kinds of challenges that schools face in supporting student achievement.  These examples are not meant to sensationalise the issues and are not offered as excuses for poor performance.  Rather, they point to the need to understand the context in which educators work, and the potential challenges they experience in ‘turning their school’ around.  A common theme from these examples was the need to get the school culture ‘right’ and raise student expectations before implementing formal strategies to shift student achievement.

A neglected school

School 1:  I didn’t think it was possible to be shocked by the condition of a school.  I have quite a bit of experience and thought whatever I was walking into I would be able to manage.  But, I was shocked when I came here. The physical condition of the buildings was poor, and in the past several years there had been little or no money spent on the library or on refurbishment of buildings or equipment. In one meeting I remember seeing a rat running across the floor…  It felt like the school had been neglected for years. Attendance and engagement in classrooms was really low. Student behaviour was challenging. The Police came onto the premises at least daily to address concerns or issues with fighting, violence or damage to property. It was an expectation they would be here, not an exception. Our focus here has been to work to get students engaged, to welcome them when they come to school, and make learning interesting to them.

Lack of services and support in the community

School 5: This community experiences a lot of social and economic disadvantage. When I came here the roll was dropping, and there just didn’t seem to be any confidence in the school. The area has high levels of unemployment, poverty, crime and youth crime.  Students were coming to school unable to read and write at the expected level.  They may have had patchy attendance at the primary school and they may not be that engaged in education.  The first thing I had to do was to work on the culture of the school. If you don’t get that right you don’t get anything right.  We have made a concerted effort to change the culture of the school and the language that we use to describe how we operate as a school in every area in terms of our management of and care of our students.

The Principals’ office as a triage room

School 6: When I first came here my office was like a triage room.  The table was covered in incident reports – mostly involving students, but some involving staff as well.  There was very little order in the school. For years there had been inconsistent leadership and it took time just to settle the students. I would speak to students about behaviour issues and they seemed surprised that I was following things up with them.  They didn’t realise there were consequences. There are not many of the services we need in this community. A number of the families in this community experience drug and alcohol issues, but the services available in the community do not address the level of need. On one occasion I took a student in my car to get him the support he needed. The service was over an hour away…I would like to do some creative things…but a lot of my work in the first two years has just been getting the students settled.

Section 4: Evaluation Findings

The major findings of the external evaluation are presented in the following sections of this report.  For each evaluation question, key findings are presented as bullet points.  This will be helpful for readers who are interested in obtaining a quick overview of key messages.  Each point is then elaborated in the narrative section with supportive evidence and interpretations.  Direct quotations from key stakeholders that are relevant to the presentation are interspersed throughout the findings section.

4.1 What were stakeholders’ Impressions of the Pilot?

Impressions of the Pilot - overall

• Overall, the TAS pilot was highly regarded by school stakeholders. It was considered valuable as a mechanism for school improvement.
• Ministry of Education representatives were aware that the TAS pilot was a ‘different approach’ to school monitoring and evaluation. They were supportive of the explicit mandate for collaboration between the agencies.
• Ministry representatives in the regions indicated that initially there was a lack of clarity about the pilot, and how it would work. Regional Directors indicated there were ‘teething issues’ concerning communication about TAS processes, and progress made in schools. These issues were associated with a  breakdown in communication, particularly in terms of sharing what was occurring in the pilot schools within the regions.  Communication issues may also have been  exacerbated by layers of reporting at the national and at the regional level and also a result of some turnover in staff. Communication between ERO and the Ministry appeared to improve over the course of pilot implementation.    

ERO and the Ministry -on the same page

‘The ERO team and the Ministry worked really well together, very closely.  And we have lots of conversations about what we are seeing and what is happening.  We see things the same way, but bring different perspectives to the table.  We are making a concerted effort to be on the same page about things because if the school hear a different message then one group can get played off against the other.’ (School 2, Ministry representative).

• Regional Directors within ERO were supportive of the core principles that underpinned the pilot approach. They identified the need for a stronger evidence base about how the model was intended to work in practice, and the evidence base for the approach to school evaluation.   The model was a resource-intensive approach to school evaluation, so clear communication of the underpinning evidence for the approach to key internal and external agencies was required.   One Director questioned the evidence base for focusing primarily on schools as agents of change, rather than the wider community.  From this perspective the TAS model may have been overly optimistic in its objective to support school improvement primarily through evaluation mechanisms. 
• There was a view that opportunities for internal collaboration between the TAS team and the regions within ERO did not always work well. While the TAS team shared information about progress within the TAS schools, the nature of the communication was often descriptive (focusing on specific issues within each school) rather than pitched at a strategic level. 

• Principals generally welcomed being part of the pilot.  They saw it as an opportunity to get additional support and/or resources for the school.  One principal viewed the pilot as a way to get ‘direct’ support from the team with implementation of school improvements.  However, direct support or advice was not the intention of the pilot. This expectation appeared to affect the school’s engagement with the TAS team, generated confusion about purpose, and negatively influenced perceptions of the value of pilot activities.
• Some board members did not appear to understand the difference between the pilot and other support offered by the Ministry of Education or LSMs (in the schools where they existed).   While most board chairs were positive about the pilot, some were not clear about the differences in roles between representatives of the Ministry and ERO. It was evident that some Board Chairs had the impression that ERO reviews were predominantly a ‘tick and flick’ approach to ensure school accountability.
• In the three schools with an LSM, there appeared to be confusion about the roles and responsibilities of the TAS team, the LSM and the Board.  An LSM in one school attempted to resolve the confusion by integrating ERO recommendations into the school’s action plan to clarify the relationship between reports and agreed actions.
• Principals were interested in ERO and the Ministry creating an opportunity to share learnings with other schools involved in the pilot.   The quotation below illustrates the value of progressive sharing of findings through a pilot process.  Other principals noted the value of a forum with all the schools involved to highlight learnings and share promising practices.

Value of progressive reflection

‘Every now and again I think it would be beneficial to call a ‘time out’.  Bring the principals together and ask them how it is going.  Are there things that we could be doing differently?  Are there things the schools could do to support the project more? It would also support us (principals) in our learning through discussion.  It would be a massive learning opportunity.  “Why are you valuing this over that?  I am interested in your thinking on that”.’ (School 4, Principal)

Perceptions of the Process

• Principals and board chairs appreciated working closely with the team anchor. This individual ensured some continuity between reviews, and became a trusted partner. 
• The continuity of the team across school reviews was valued by school stakeholders.  Working with the same team over time enhanced levels of trust, and facilitated ‘frank conversations’ about school improvement. However, three of the six principals noted that when other TAS team members attended the schools for reviews, there tended to be additional issues raised about school issues that had not been raised before, which they found frustrating. 

Continuity of the Team

‘One of the things that was valuable was the continuity in the TAS team with schools because there was a growing understanding of the painpoints, and points of traction. A balance then needs to be struck between identifying the range of issues within a school that may be inhibiting improvement, and being realistic about what improvements can be made over a given timeframe.  A balance also needs to be struck between providing evidence of gaps, and acknowledging progress or small wins.’

• Termly reviews created momentum for change.  School stakeholders indicated they prepared for the ‘visits’, and in between developed action plans to address concerns outlined in ERO report. While this does not mean great gains were always made between reviews each term, the regularity of ERO contact with the schools appeared to mobilise reflection and action.
• The relational skills of the TAS team were highly valued.  It was clear that TAS team members were viewed positively by school stakeholders.  As the relationship between the reviewers and the schools developed, school stakeholders looked forward to the termly ‘visit’. 

The two quotations presented below point to the level of professional respect and relational trust that was developed between school stakeholders and TAS team members:

‘Over the course of this project we’ve had lots of conversations…I don’t want to say we were all part of the same team, but I felt that I could share whatever was on my mind with the team safely.’ (School 3 - principal)

‘The support too, I must say that it was very humbling and I was very grateful. We were all very grateful that we could actually be real and say to the team, “Look, I don’t know what you’re talking about.”  And, then someone would explain it.’  (School 2 - board chair)

4.2 How effective was the pilot in achieving intended outcomes?

The overall key evaluation question orienting this evaluation was focused on effectiveness, that is 'How effective was the TAS pilot in supporting school improvement?'  Related questions focused on strengths and weaknesses of the pilot, opportunities for improvement, and lessons learned about promising practices in working with persistently poor performing schools.

Claims about the effectiveness of the pilot were based on a synthesis of all evaluation evidence, including shifts in student outcomes from baseline to 2019, perspectives of agency stakeholders (ERO and the Ministry), and interviews with school principals and board chairs.  The rubric that informed these judgements is included in Appendix 1.

• Fifteen school stakeholders (that is, principals, Board Chairs and LSMs[12]) were also asked to rate the overall value of the TAS Pilot to the school. The rating scale was 1-5 where 1 was low or poor and 5 was high or excellent. The overall value of the TAS pilot to the school to the principals involved was 4 out a possible 5.  Two principals acknowledged that because the TAS pilot was a trial, there was room for improvement, but indicated the work of the TAS team had been very beneficial to their school improvement journey.

The average rating for the four other dimensions was 3.7 out of a possible 5.  This indicates that most stakeholders saw value in the TAS pilot.  Two of the school board chairs rated the pilot approach very low on a number of dimensions.  In both cases these members were relatively new appointments to the role within their respective schools.  Their perspectives need to be acknowledged and the reasons for their views further explored. It may be that they had not been involved sufficiently long enough to see evidence of the work undertaken through the pilot.  Another explanation may be that these individuals held expectations of the pilot that – in their view - had not been realised.  Interviews with the previous chairs in both these schools indicated markedly more positive views of the pilot.

• The cumulative value of collaborative work by the Ministry, ERO and other agencies appeared to be the most effective change strategy in the pilot.

Schools identified a range of improvements that they believed that involvement in the TAS pilot had progressed.  Evaluation evidence suggested that the pilot accelerated school improvement.  While this claim indicates a positive change, it cannot be claimed that the TAS  pilot was solely responsible for the changes observed.  Several stakeholders noted that other change efforts were in place across the six schools at the same time as the pilot, including supports provided by the Ministry of Education for professional learning and development, and specific appointments such as an LSM or curriculum advisors. 

• The TAS pilot was valued by Ministry stakeholders as reports provided a high level indicator of progress within the schools.  One Ministry representative noted that the TAS team ‘connected in’ with the school through reviews on a much more regular basis than their own regional team was able to, which was valuable for the school.

Outcome Domains

ERO situates the child at the centre of school improvement.  However, it does not directly influence students’ learning and outcomes.  It works with schools to determine what school level or teacher level improvements will have the most impact on student learning.  The review process is designed to build the capacity of the school for improvement through knowledge about what is required to progress change, providing access to meaningful data on school progress in particular areas,  and advocating for the school - if need be - for additional resources.

This evaluation focuses on determining the contribution of the TAS pilot to three short term outcomes (or lead indicators) of school improvement:

1. Schools have an enhanced capacity to gather and use data,
2. Schools have necessary supports in place to progress planned improvements, and
3. Schools find the TAS reports useful for planning for improvement.

School stakeholders found the pilot effective in progressing these three short term outcomes. In particular, school stakeholders highlighted the contribution of pilot activities to the school in its use of data for planning and improvement, and the capacity of the pilot to broker additional support for the school.   

There were mixed views about the usefulness of the TAS written reports.  Some principals and Board Chairs found the reports overwhelming in the number of issues or areas for improvement required.  Others pointed to an occasional mismatch between the verbal reports and the written report received by the school after the review.

These three outcome domains are depicted in fig 2 on the following page. The subsequent section outlines the perspectives of stakeholders about the contribution of the pilot to these outcomes.

Figure 4 :  Key Activities and intended outcomes of the TAS pilot

4.2.1 Outcome 1: Enhanced capacity to gather and use data:

The TAS pilot’s role in supporting evaluative thinking

Key Messages

• Most stakeholders indicated that the TAS team were skilled in monitoring and evaluation
• The intensive focus on data collection, analysis and reporting encouraged school stakeholders to consider strategies to collect and review both qualitative and quantitative data.  The schools began progressively sharing additional data that had emerged from review activities with their boards.
• Principals highlighted the personal benefits of becoming more critically reflective about school progress
• Principals acknowledged the value of data, and were keen to continue to develop knowledge and skills to strengthen local level data collection and analysis.

Nearly all stakeholders interviewed in the external evaluation pointed to the team’s strengths in the collection, analysis and synthesis of data.  The skill set of the team was acknowledged and trusted. 

The TAS team aimed to enhance the evaluative capacity of schools to support improvement. The term evaluative capacity is often used interchangeably with skills in evaluative thinking. However, both terms are often muddy and imprecise.  Buckley, Archibald, Hargraves, Trochim (2015) define evaluative thinking as

… critical thinking applied in the context of evaluation, motivated by an attitude of inquisitiveness and a belief in the value of evidence, that involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking, and informing decisions in preparation for action (pg 378).

Evaluative thinking is clearly a skill set that is useful for principals seeking to improve school processes and outcomes. Increasing evaluation skills among school leaders may have positive flow on effects to other staff, and enhance ongoing improvement efforts.

One of the key roles of the TAS team was to generate data to highlight performance concerns, and then monitor this data over time to assess school progress.  As the team was working with schools each term the persistent collection, review and presentation of data became a familiar activity.  It appears that in most schools these activities generated interest in progressing actions that would improve student outcomes.  There was evidence that the process and emphasis on collecting qualitative and quantitative data on a regular basis had focused the attention of principals and the boards of the schools on their improvement pathway. 

Asking the right questions

‘The support that they gave us was transformative and the way that our Board members were able to kind of really discern the information and the data that we were getting.  So, we were able to ask the right questions, look for the right things, understand the connection between strategic goals.  There was a huge flow-on effect in our roles because of the support and learning that we had from the team.’(School 3 - board chair)

Value of critical review

‘The project (TAS) has asked the challenging questions - or the provocative questions - that has made us critically review our own practice in what’s worked, what’s working and what’s not working. And I think that as a principal I have appreciated that because we can think we’re doing well or whatever, but then the review team come in and they say “Well show us!” ‘ (School 5 - principal)

Support to be honest

‘I think that as the principal, being really honest to yourself is really important, but at times you can lose your sense of objectivity. Because you can think ‘Oh things are really marvellous’ and some poor teacher in their classroom is having absolute hell with their class or students are not making sufficient gains. I think that the process and the project has helped to keep us honest about our progress.’ (School 4 - principal)

Building an evidence base

‘Schools sometimes are not very good at capturing what they do well, so we’ve been having that discussion too with the team. The question we are working on is how do we present this or record it in a robust way and in a format and where it’s really obvious that this has been something that’s worked well and something that hasn’t worked well.’ (School 3 - principal)

Some stakeholders indicated that the presence of the TAS team had raised questions within the school about ways to improve the capture of information about progress at the school level.  Most of the principals acknowledged that schools often do not adequately collect evidence about their programmes, or rely only on anecdotal information shared by teachers or students.  The TAS team had opened up opportunities to consider more robust ways to document progress.

The TAS team raised awareness of the type of data that could be gathered or retrieved to inform school planning. However, there was broader acknowledgement that the use of data remained a ‘work in progress’. A principal explained, “It's still is a weakness for us to be honest, you know. I've come into this year thinking we've just got to be better. We've got to be data sharp and we've got to really grow our capacity and understand what's going on with our kids in all – right across – not just English, maths, and reading.”  (School 6, Principal)

A key learning from the pilot is that the school needs to see that the collection of evidence serves a purpose.  Data for data’s sake is not helpful.  For one of the principals the ‘relentless’ focus on quantitative outcome data (for example, NCEA levels or attendance data) had meant that other, more local or individual markers of progress were dismissed.  While it is the evaluator’s understanding that the TAS team encouraged local data to be collected by the school, it was the emphasis on reviewing achievement data that was of concern to this principal.  The view was that there were non-numeric signs of progress or indicators which would be more helpful in tracking school performance and student achievement.

Be clear about the purpose and use of data

‘I think that the team’s emphasis on data can have a negative effect too. People go into panic mode and start collecting everything, but they are not really sure why they’re collecting it, or what they’re trying to achieve at the end of it. So, I think it’s fine about saying ‘collect data’, but it is more important to be really clear around the purpose and how you collect the data and how you use that data, too.’  (School 3 - principal)

4.2.2 Outcome 2:  Schools have necessary supports in place to support improvement

The TAS team reports as a catalyst for brokering support

Key Messages

• TAS reports provided the Ministry with a robust evidence base to inform decisions about required resources and support to progress improvements in the school. 
• Most principals were very clear about ERO’s role in monitoring and evaluation, and the Ministry’s role in providing other support and resources.  However, in one school the  perception of the roles of the two agencies appeared muddled, and it was evident the principal and the newly appointed Board Chair were disappointed that ERO did not provide more direct support to the school.
• There was a general view that identification of external support or resources identified through the TAS reports were activated ‘quickly’ by the Ministry.
• The context of the school (the issues identified, readiness for change, and the level of need) required different responses and potentially different layers of support. 

One of the Ministry’s roles is to provide targeted  support and/or further resourcing to schools to drive improvement. TAS reports identified the challenges experienced by poorly performing schools, pinpointed the areas that needed to be addressed, and recommended options for improvement to the school.  The issues facing the school were discussed with the Ministry.

The collaborative working relationship between the Ministry and ERO enabled stronger support for the school where and when they needed it.   ERO reports were important to the Ministry stakeholders, but equally the capacity of the Ministry to broker the right support and resources was critical in supporting school improvement.

Brokering support

‘Our team (the Ministry in the region) did a good job in brokering people in… Again, it’s about knowing people and having the right people in the PLD to make sure you pull the right people in. So, it is all about a relationship-based environment that picks up on who the best person is in this space and then identifies how we get them there.’ (School 3 - Ministry representative)

There is evidence from this evaluation that the TAS reports were valued by Ministry stakeholders in assisting identification of support requirements. The reports were often used as a basis for advocacy. They provided a solid empirical basis for Ministry decisions and were taken seriously.   A regional director from the Ministry explained, “Without the TAS pilot there wouldn’t have been that capacity for that very intensive support. I mean the Ministry was doing a lot of things and were bringing in Professional learning and Development (PLD) people that were suited for the role and suited for the context, but without that intense support of the school and that independent voice from ERO, some of these changes may not have occurred.”

A regional director within the Ministry questioned whether being selected as a TAS pilot school ‘automatically’ created an expectation in the school that they would be able to access further support and resourcing. In her view schools were often inappropriately ‘sold’ this message.   In one school both the principal and board chair expressed disappointment in the TAS team for the lack of support to the school.  This view indicates a misunderstanding or potential mis-representation of the TAS pilot’s role.

Most school stakeholders indicated that the collaborative identification of issues had contributed to clarification of key resource requirements and expedited support. For example, a board chair shared:  “After the review we had people the next day ringing us, and making contact. In some cases we had been waiting for support for maybe a year, 18 months for a return or a reply or engagement. But with this. it was super quick” (School 2 - board chair)

Swift action to address support needs

‘…I remember sitting round in this office with them (the TAS team)  and we'd sort of brainstorm up on the wall what were issues and what would help. And we just wrote up these lists and, which I think x (the Board Chair) was here as well and he said straight away, “extra staffing would really help with some of the pressure spots in the school." And we managed to get that.  It did help, and it still does. And how we use that staffing has changed a little bit over each year.’ (School 6 - principal)

4.2.3 Outcome 3: Usefulness of TAS Team reports

The value of reports for school planning

Key Messages

• Most principals indicated that the ERO reports were useful to them in planning for school improvement and tracking progress over time.  
• In general, principals, board chairs and Ministry representatives agreed that the reports were accurate reflections of the issues within the school.  Reports often validated assessments of issues within the school, and provided additional ‘momentum’ for action. School stakeholders reported feeling overwhelmed by the number of issues, which appeared to contribute to a loss of focus.
• The reports provided to schools were often lengthy and the priority issues the school needed to work on was not clear.  The reports made it difficult for those outside the TAS team to identify a clear pathway for improvement.
• TAS reports frequently highlighted leadership issues within the school that were inhibiting progress.  Some principals used this feedback to initiate structural changes to their senior leadership team, and/or develop distributive leadership structures. Other principals appeared to take the feedback personally, which may have influenced levels of engagement with ERO.
• Some of the principals and board chairs pointed to a disconnect between verbal reports (presented by the team during the review) and the written report. 
• Schools valued the opportunity to provide feedback on TAS team reports. In two instances principals and board chairs challenged the judgements in the ERO report, because of a belief that the school had been misrepresented. The judgements were considered particularly alarming if the report was designated as an external facing (public) publication. Provision of additional evidence by the school, and discussion resulted in some modifications to the final report. 

Most school stakeholders expressed a view that the reports were accurate and useful for school planning. While there were two instances where conclusions were challenged, the discussion with the TAS team had lead to modifications.  It was evident that the capacity to challenge interpretations was facilitated by the relational trust that had developed between the TAS team members and school stakeholders.

Involvement from an external agency like ERO in school evaluation was seen to be helpful in ‘seeing things from a different perspective.’  The principal’s views were either validated or challenged with evidence. Two of the six principals reported than an external review of the school supported them in communicating the importance of changes to the staff team.  One of the principal’s explained how the reports gave him additional credibility to propose changes, “A report from ERO gives the authority to act and creates some momentum to get changes to happen.  We’ve used the intervention (TAS) as another way of getting people involved in change, so it’s not just me.  You do hear a few war stories of teachers, players, senior players, senior teachers, that sometimes just don’t like that principal or another member of the senior leadership team.  If I say it is one thing.  If it comes from an external source like ERO it has more traction.” (School 4, Principal)

The value of the reports in communicating the big issues that needed to be addressed was inhibited when reports were overly detailed. Reports that included multiple issues or weaknesses were experienced as ‘overwhelming’ or ‘deflating’ by their key audience- the school.  School stakeholders would have preferred a targeted focus on a few issues rather than a lengthy list of actions. Two quotations from school stakeholders illustrate the potential negative impact of the reports on the school:

‘I recall one report – I think one interim report I think it was – I counted there were 40-something points that we needed to work on. I counted them all up out of the report. And, we went through a lot of them in the next review, and I acknowledged that the school had not been able to get to them all…I missed some of those issues because there was too many of them…But sometimes the important things get lost in the sheer number of issues being raised.’ (School 6 - principal)

‘They (the TAS team) have seen all these things you didn’t see.  And, they list them and tell you what you need to cover by when. There's good and bad in that because it does create pressure and stress. And that's okay, but it sometimes got out of hand and we couldn’t keep up. It was very stressful.  And you felt bad about it. And other people felt bad about it in the school.’ (School 5 – board chair)

Internal stakeholders within ERO also questioned the value of some of the reports.  The length and detail often made it difficult for an individual outside the team to map the school’s change trajectory.  One director explained, “ It was hard to see what the journey looked like, where they were and what the priorities were.  Everything was in there.” 

The most effective reports appeared to be those that included feedback to the school about their progress, balanced with a limited number of areas for improvement with specific ‘do-able’ recommendations.

‘We knew we didn’t have to read between lines, we didn’t have to second guess, the information we were receiving was very clear (including) the recommendations that were being made.  So, we didn’t kind of get, “This is failing, do something about it.”  It came with recommendations and safe recommendations because they knew the context and all of our community and our school.  So, it wasn’t like, as a board, we weren't second-guessing…It wasn’t like that. It was really clear.  It was very clear and concise.’  (School 2- board chair)

4.3 School profiles- Promising Practices

The TAS approach was piloted in six schools.  The evaluation findings focused on feedback from a range of stakeholders about the pilot.  As this is an evaluation of the pilot this is appropriate.[13]  It became apparent during a review of ERO reports that each school had developed particular strategies to address issues raised in ERO reports. The actions often related to feedback about challenges in a specific area, such as curriculum or teacher capability. But, what is notable about these actions is that they were developed to fit the context of the school and the skills of the educators involved.  

School profiles are presented to highlight promising practices used by each school to promote improvements.  These are illustrative only.  No external judgement has been made of school effectiveness.  

For each school a promising practice has been selected.  The process for selecting a promising improvement practice was based on three principles:

1. Each school stakeholder was asked to comment on a significant or important change that they believed had occurred within the school and to comment on the contribution of TAS to that improvement.
2. The identified practice (or practices) was then cross-checked with written review reports, both internal and external facing (public) reports (where available). 
3. The TAS anchor for each school and/or the TAS Manager was asked to validate selection of promising practices.

The practices described by stakeholders were taken at face value.  There was, however, some triangulation with the secondary data from the reviewers’ reports, and from discussions with the TAS team.  Many of these activities were accelerated by the intensive support from ERO, but no claim is made that these improvements are directly attributable to TAS; the schools were supported in a range of ways by the Ministry and other agencies. 

The presentation of each school’s promising practices in this report is explicitly an affirmation of the school’s journey to improvement.  The purpose of presenting these practices is to showcase improvements that may be potential enablers for ongoing change, but the impact of these strategies may not yet show up in current student outcome data. 

A focus on one of the nominated promising practices is presented below for each school.  A summary of all promising practices is provided on page 54.

School 1: School community as learners and Leaders

This medium-sized secondary school caters for approximately 250 students.  The majority of students identify as Maori.  The experienced principal has strong leadership skills, and has initiated a joint focus on student engagement and building the professional capability of staff working within the school. 

The principal and board chair see the school as a learning community.  It is not just the students that are seen as learners.  The students are learners and leaders and the staff are learners and leaders.  Curriculum offerings in the school are student-centred.  Student interests are canvassed as a way to inform the development of curriculum options that are engaging to them.  Whānau structures are used to promote learning and to promote leadership opportunities within the school. The school staff team work closely with whānau to strengthen community support of the school and to involve parents and carers as much as possible in the school.

The school has a specialist literacy leadership role and additional resources have been added to the library to build on students’ interests. They have built from these supports to progress maths and reading capability across the school community. Students in this school are encouraged to actively engage with learning, and to develop as leaders within their community. One of the initiatives the school has adopted to support learning and leadership involves a strong focus on writing to support students to identify their career aspirations.  Students undertake research about that career to gain an understanding of the pathway to particular careers.  

‘In our school it is about building a learning culture… Students here are very loyal to this school. The students will tell you that this is the school for leaders.’ (principal) 

School 2:  Localising the curriculum to enhance cultural connectedness

This small rural primary school has emphasised the development and implementation of a localised curriculum in English medium and also Māori medium in a dual pathway. Both mediums include a strong connection with the local marae, and with whānau and hapu in the immediate and local environment. The ‘kura is part of the community’ and ‘the community is part of the kura’ is a key underpinning kaupapa. The school's partnership with kaumatua, kuia and whānau supports the localised curriculum. Community consultation processes identified important local history, which are then embedded into curriculum planning.  Students are connecting with their whakapapa and hapu histories. The localised curriculum was seen as a way to engage students in learning that affirms the learners’ language, culture and identity alongside meeting whānau aspirations. There has also been an increased use of te reo in classrooms. 

‘Having the Turnaround Schools pilot come in at that time was timely for us because we really wanted to move as a kura. We also wanting to support our principal…Then together we started on the same page and we just wanted to keep journey(ing) forward in terms of what our parents and our community want for our children.’ (board chair)

School 3: Student wellbeing and distributed leadership

This small rural school was in danger of being closed when it entered the pilot. The school’s roll has steadily increased over the course of the pilot, and the school has renewed support from the local community.

This school has focused on improving student wellbeing and engagement. There is a strong pastoral support, and student surveys indicate that students feel positive about their school. The school emphasises positive behaviour based on shared school values.

A distributed leadership model spreads the authority for change, ensures consistency in messaging across the school, and is a mechanism to promote sustainability.  Leaders have been appointed in three key areas – Primary, Year 9’s and 10’s and years 11 plus. Teachers in these leadership positions share information about what is occurring in student achievement within their area to inform the school’s insights about issues and opportunities for improvement.  Regular consultations with students and staff inform decisions made by the senior leadership team and the board.

Over the course of the pilot the school has increased the range of subjects offered to students.  An initial focus on maths has provided a template for new approaches to teaching and learning across other subject areas.

‘…We used ERO in a positive way - when they were here - to keep strategically having chats with individual staff members about, “How are you going to get better?  How are you going with your maths?  How’s it looking in the class?” And, rather than going and just reviewing I would encourage them (the TAS team) to talk with staff and discuss what they were seeing. We were in this together.’ (principal)

School 4: Data-driven decision making

This school is a large secondary school with a high proportion of students who identify as Māori or locate their cultural identity with one or more Pacific nations.   A key focus for this school during the TAS pilot has been to strengthen the school’s capacity to use data.   The school has demonstrated an interest and appreciation of data in tracking school progress and student outcomes.    The school is making greater use of Ministry data, data from NZQA and other externally produced data.  Information from these data sources is being used more substantively to inform school decisions. The appointment of a data manager demonstrates the investment of the school in using data effectively.  As a result of this initiative progress data is able to be shared more widely across the school community. The data manager shares information with the school  board, senior and middle leaders in relation to what data matters and how it is best presented. The board now receives regular ‘dashboard reporting’ that includes indicators of improvement – for example, attendance and NCEA progress. The board has demonstrated interest in this data and is beginning to identify what additional data would be helpful to inform decisions.   There is now a greater expectation on middle leaders to use Year 9 to 13 data to report on faculty area effectiveness.

‘The endorsement and validation from the team (TAS Team) about what we were doing to improve the school was very important.  They would ask, “Wow, what’s this?  What’s going on here?”  And, that inquiring question was really useful in getting us to reflect and think about our progress and how we use data.’ (principal)

School 5: Community, culture and connection

This school is a medium-sized school in the North Island.  A large proportion of students identify as Māori or are from Pacific nations. The school roll had been declining for a number of years before the principal joined the school.  Early consultations indicated to the principal that the community had lost confidence in the school. 

The principal intentionally focused on shifting student and community perception of the school. This was done by building on students’ cultural strengths and identities and by  strengthening connections. 

Students, local iwi and community were consulted to design a school motto, logo and other key values.  The promotion of school values connected to culturally responsive teaching practices were made a priority and formed the basis for improving teaching effectiveness.   Teaching initiatives reinforced student engagement and data over time has indicated improved attendance and positive engagement in learning. Senior students were called to ‘step up’ as leaders and more teachers from the local community joined the staff. Students’ progress and achievement in NCEA has improved over time and the school roll is steadily increasing.   The school has demonstrated Improvements in NCEA Level 1 and 2.  There is a stability in Level 3.  The school is now close to decile average.

‘Our schooling improvement was through a very, very unashamed cultural - ‘lens’ isn’t the right word - ‘vehicle’. The final culture we developed was the vehicle by which we not just sort of changed the outward look of who we are but it was about the systemic changes to the way that we operate in the school to signal to our community that we were serious about change…(With the TAS team) we had the opportunity to talk about what our latest successes were, or not, or what things had been a challenge for us. I felt that each time we had the team visit, it gave us an opportunity to present what we had done since their last visit’(principal)

School 6:  Building student engagement and wellbeing

This small rural school has a student population of just over 100 students.   Most of the students identify as Māori. The principal returned to the area and joined the school in 2017 after several years working in another part of New Zealand.   He became aware very early on in his tenure that student behaviour was a real issue within the school.  He has focused his attention on settling behavioural issues within the school and strengthening local curriculum offerings. 

The school care team monitors student wellbeing and learning to ensure that students get the support they need. The school has been supported with additional staffing to support curriculum improvements in te reo, maths and literacy.  External partners are increasingly valued to support and provide better learning outcomes for students. A school wide focus on attendance has improved attendance in some year levels across the school.

‘(The TAS pilot) took some pressure off us in a way because there was some extra resourcing that went with that and extra supports.  We identified that some students need extra support and it is hard in this area to get that support. You have to get that right first. So staffing conversations were helpful, and staffing changes were helpful. (It meant we were able to) identify those students that needed extra support with their behavioural issues and/or learning. And, our capacity to develop processes around referral and support has been very important.’ (principal)

Fig 5: Schools and promising practices

4.3.1 What about shifts in student outcome data?

The key objective of the TAS pilot was to develop and implement an approach to school evaluation that would support schools in improving outcomes for students.   The education system measures student progress with a range of indicators. There is a range of data available on schools, including attendance data, NCEA Level 1, 2 and 3, attendance data and school leaver data[14].

The schools involved in the pilot required support that would shift thinking, expectations and practices of the leadership team and the Board, of teachers and of students.  Prior research indicate that changes in mind-set and practice may take at least 3-5 years to be implemented and to realise benefits (Berends, Bodilly & Kirby, 2002). 

The TAS team considered it more helpful to examine trajectories over time of cohorts and to look at patterns of change. Discussions of attendance rates, NCEA levels, and levels of literacy and numeracy provide different prisms to assess the extent of school improvement.  Each measure tells us something about student engagement or achievement. But, they do not tell us the complete story of student achievement, nor do they tell us about the role of the pilot in supporting student outcomes.

Quantitative data on key indicators that relate to student outcomes has been progressively collected from baseline in 2019. The quantitative data does not yet demonstrate strong improvements in student outcome data.   Broad patterns of change in the quantitative data is included in the appendix summary tables of each school.

The TAS team suggested that leavers data is often the ‘last’ data set to shift over time and 2020 marks the point that the TAS team expect to see the impact of improvement efforts on school  leaver data.  Some students are accelerated into pathways from school without a positive impact on the leaver data. In the six pilot schools, leavers’ data may not shift because they students are staying longer and/or because they are moving to a range of other pathways.  Retention is also a key factor affecting the quality of leaver data outcomes.

Stakeholders from the Ministry, schools and the TAS team identified a range of qualitative improvements that have been highlighted in this report. These improvements may be lead indicators that will support ongoing improvement in student outcomes over time if sustainable changes are made within schools.  Continual tracking of these progress markers will be important to gauge the sustainability of school improvement efforts for student outcomes.

4.3.2  Would school improvements have been made without the TAS pilot?

A key question in this evaluation that needs to be considered is whether any reported changes over the past three years would have occurred if the intervention had not been in place.  In evaluation terms this is called the counterfactual.

All stakeholders who were interviewed for this evaluation were asked to comment on the likelihood that change would have occurred without the initiatives undertaken during the pilot.  The dominant view was that the pilot had accelerated improvements, rather than being the direct impetus for improvement.  This was particularly evident in schools that had a range of other interventions or change initiatives underway. 

However, three of the six principals were unequivocal in their view that the improvements within the school would not have occurred without the TAS pilot. The three other principals identified the value of the pilot in bringing data to bear on the challenges the school was confronting, but suggested that the role of the pilot had been  to speed up changes, rather than create them.

The TAS pilot accelerated improvement

‘I think we would have gotten where we are without TAS, but it would have taken longer.  I think we would also have been missing a review tool which was very helpful to us. As a principal, being really honest to yourself is really important but at times you can lose your sense of objectivity... It hasn’t been in itself the lever that created the improvement in the school, because I think that we did that, but it certainly helped.’ (School 5 - principal)

4.4. How could the TAS approach be improved?

Stakeholders were asked to identify limitations of the pilot, and/ or to comment on aspects of the pilot that did not work as well as anticipated.  In this section these comments are re-framed as opportunities for improvement.

The five identified areas for improvement were:

1. Provide ongoing messaging about purpose and role
2. Share promising practices about what works for school improvement with schools
3. Encourage processes to co-construct reports
4. Align verbal and written reports
5. Provide schools with clear criteria for progress and exit

These recommendations for improvement will be important considerations for the HPS team in their work with an expanded group of RPPS schools in 2021.

1. Provide ongoing messaging about purpose and role

The success of a collaborative change process involving multiple agencies and stakeholders is highly contingent on clarity of role, clarity of purpose, and clarity of expectations.  While role clarity was acknowledged as important by the TAS team and the Ministry stakeholders, it was clear that it took some time to settle into respective roles and to understand the most effective ways to communicate progress in the pilot schools.

At the beginning of the pilot it is fair to say that - even within ERO - there was a not clear sense of how TAS was intended to work.  TAS was a national project involving reviewers from multiple regions.  Reviewers were responsible for briefing their regional managers about schools that were in their region.  Regional managers would then share information with their Ministry counterparts about progress in schools. 

Over time aspects of the approach became clearer through discussions with the TAS team, through PLD opportunities, and through internal communications.  However, for some Ministry and ERO stakeholders there was a breakdown in communication at the regional level about progress within the pilot schools.

A regional director from the Ministry explained,

“You had two organisations working together.  You had ERO running a national project from the centre to do things and they were pulling resources from across the country to help in a particular school.  And, then from the Ministry perspective, you had the regional directors being responsible for the particular school and the area.  So, sometimes it seemed like a little bit of a mismatch.” 

In May, 2020 a Zoom forum was convened by ERO with the Ministry of Education to share school progress.  Several regional directors (Ministry) indicated this forum was very useful in mapping progress within the TAS schools in their region.

A demarcation of boundaries of influence is appropriate within and between organisations.  Clarity about these boundaries may create efficiencies and reduce duplication of effort.  However, demarcation can also limit collaborative work, particularly if the nature of the initiative limits opportunities for feedback or critique.

A regional director within the Ministry pointed to the initial lack of alignment between the Ministry and ERO in working with a school that had contributed to confusion within the school, and potentially delayed action.  “There was an initial tension between ERO and the Ministry.  I think setting the intent and being clear about roles is really critical. We should have all been on the same page about what the intent was, but when it came down to whose role will do what, it was quite muddled.  This is not a criticism of ERO.  It reflects too on us in communicating that clearly.” (Director - Region 2)

Three of the board chairs recounted their initial confusion about the difference in focus and support by ERO and the Ministry. In two schools with new board members, there was a view that there had been little or no attempt to explain the TAS pilot approach.

Where an LSM was involved, the potential for confusion about roles by the Board was exacerbated.  There was confusion about the TAS team reports and their relationship to the school’s action plan developed by the LSM.  The LSM had to work hard to integrate the recommended actions into a single school plan.

Confusion about purpose

‘The board kept saying  “there are so many reports and plans.  We don’t know which is which.”  And, I think there is something there on both sides- the Board’s capability around planning, and also the potential confusion about the role in developing plans, and having different plans or reports.  But, at the end of the day the confusion cannot be an excuse…We need to have a relentless focus and be ready to have difficult conversations about the core issues.’ (School 3 - Manager)

A Ministry representative indicated that the approach adopted through TAS will ideally be ‘business as usual’ for both agencies, not a special or dedicated approach by a single agency.  From this perspective co-designing a transparent joint approach would be more sustainable and limit confusion over roles and responsibilities.  It may be useful to explore the implications of this suggestion.  On the one hand  it may lead to better alignment in processes and in messaging.  Costs associated with a dedicated team would also be minimised.  On the other hand without a dedicated focus on poorly performing schools opportunities for influence may be lost.

2. Share promising practices about what works for school improvement with schools

TAS team members made a clear distinction between the roles of the Ministry and their responsibilities within ERO as reviewers.  The role of the Ministry is sector enablement, the role of ERO is monitoring and evaluation. 

Several school stakeholders expressed disappointment that the TAS team did not readily share good practices they had observed in other TAS schools, or suggest specific options for them to consider in improving particular aspects of the school. Some principals associated that this boundary between review and advice was a residual practice from ‘standard’ ERO reviews[15] that emphasise public accountability. 

The perceived legacy of this traditional split was a lost opportunity for influence.  An LSM from one of the pilot schools also expressed disappointment that learnings from their experience in other schools could not be shared.

‘It’s been my point of contention for years, before Turnaround Schools came along.  ERO holds so much knowledge – they carry institutional knowledge and professional knowledge.  They also know how things work in the system.  They know the senior people.  We want them to tell us where the good practice is, where we can go to get this.  I think if they are going into this school recovery area rather than a strict review model, they’ve got to learn how to share the stuff.’ (School 4 LSM)

It was evident from the evaluation interviews that some schools were seeking more direction from the TAS team.  In one school there was an interest in how data could be used more effectively within the school as this was identified as a weakness in the school. The principal sought support from the team to identify potential strategies that other schools had used successfully.  Initially this information was not shared, and was only discussed after repeated requests.

As this example indicates it appears that on occasion the TAS team offered some reflections to the principals and senior leadership team, and suggested options for the schools to consider.  This more interactive role was highly valued by the schools.  These conversations had validated what the schools were doing, and also encouraged schools to be more critically reflective about their strategies.

Ministry representatives also suggested that ERO’s knowledge about school improvement in Turnaround schools would be valuable to share with schools and more broadly with the wider sector. Most principals suggested it would be useful for them to share their experiences and learnings from the pilot in a face to face forum.  They were not concerned about being identifiable as a TAS school. They expressed interest in learning from others in similar types of schools. The proceedings from the forum may  also document ‘what works and what doesn’t work’ to turn schools around in New Zealand.  

Share knowledge about what works

‘I think ERO are in a good position to reflect with schools about what they have learned about how to turn schools around.  They have that expert knowledge now.  Maybe they could say, ‘here’s what the school up the road or across the country has done to address these same kind of issues.  It may not work here, but maybe we can think it through.’  It’s not about giving them the answers, but perhaps giving them more of a roadmap. Schools that are struggling need more direction sometimes.’ (School 2 - Ministry representative)

3. Encourage processes to co-construct reports

For principals the process of the review was generally collaborative, but the process for developing the report was not.  Principals and board chairs noted that the reports usually highlighted the issues, and did not sufficiently acknowledge improvements or the strengths of the school. 

The TAS team made a concerted effort to focus on key areas for improvement to lessen the burden on the leadership team and to mobilise action. For the TAS team there was an attempt to keep a balance in the report.  However, this was challenging in situations where the team became aware of multiple issues that posed risks to the health and safety of students or teachers, and negatively affected student outcomes.

A number of the principals suggested that reading the TAS reports was ‘overwhelming’ and the content was disheartening. One stakeholder indicated that while the team included references to progress between reviews, points of progress ‘always seemed to be followed by a “yes, but”. There was an acknowledgement that some progress had been made, but the reports emphasis was on all the other improvement work still to be done. In some cases the number of issues raised in the reports appeared to have paralysed the school from positive action.  A regional director from the Ministry explained:

‘The ERO report was great in that it identified all these issues in an attempt to look at the school holistically.  But, we got lost in the trees and we couldn’t see the forest. There was too much in the school’s plan that they needed to do, which actually enabled them to do nothing.  The result of this was that there were a range of things that gave the school an opportunity to make excuses. “I haven’t got to that.  I don’t know what I should be doing.”  Perhaps we would have seen some progress if there were smaller things that the school could work on, or we could have made other decisions earlier about what was the priority.’ (School 2 - Ministry representative)

A more inclusive process of co-construction was recommended as a way to build ownership of the school and the board in implementing actions.  In their view such a process would give the school more agency in its improvement journey.   Co-construction was not seen as a threat to accountability, but rather as an acknowledgement that the agencies share a single purpose – to improve schools and outcomes for students.  An opportunity for co-construction may also assist the school to focus on agreed priorities and identify creative ideas for resolving issues.

Value of Co-Construction

‘I think the reports need to be a more co-constructed effort. If the report is going to be useful, there’s got to be a shared ownership and buy in from all the parties where the challenging questions can be asked. It’s not to say that it’s going to be rosy and that it’s going to gloss over things, but I think that the process of writing the report needs to be - I think ERO needs to take some risks about the ownership of the report.  If it’s going to be useful for improvement it should be co-constructed.’ (School 5 - principal)

4. Align verbal and written reports

TAS team review reports were generally seen as accurate reflections of the issues in the school. Most principals and board chairs considered that the reports validated their own views of the issues in the school. However, some stakeholders interviewed during this evaluation observed a mis-match between the verbal reports, and the written reports constructed after the review. The misalignment was also noted by two regional directors.  They indicated that schools were particularly concerned if it was a publicly available report.

‘We often don’t like being the bearer of bad news.  I think sometimes some of those ERO visits and the subsequent reports were not aligned in some way.  The issues were the same, but the framing of the issues was different.’ (School 5 – LSM)

For one school the difference in tone and scope of required improvements between the verbal report and the written product contributed to an initial loss of the trust that had formed between the school and the TAS team.  This was particularly challenging for the school and the Board as the report was to be released publicly.

The principal explained,  “When we read the draft report which was, you know, a big thing for us, we asked ourselves a question “So what school has this report been written about? It’s not our school.” We felt that we had this trusting and amazing relationship with the team and we had produced a lot of documentation about the progress that our school had made…And our board felt betrayed. Absolutely betrayed. We had trusted these people. We thought we were on the same page. We thought we were all working together. We’d had a lot of very positive feedback from the team right through this time, but the written report came as a real shock.” (School 5 - principal)

The principal and board representatives of this school contacted the TAS team, and expressed their concern about the report.  Following a face to face meeting, and the presentation of additional evidence, some statements made in the report were modified.  While the quotation presented above indicates that the school had lost confidence in the team’s judgements, it was felt that the team’s openness to challenge, and genuine listening to the school’s concerns restored trust.

Summary emails prepared by the TAS team following a review appeared to be an effective way of capturing the key messages, and reinforcing observations.  Another advantage of email feedback was that it offered quicker real-time feedback to schools.  It bypasses traditional peer review and quality assurance processes in the standard ERO procedures. 

5. Sharpen criteria for evaluation of progress and exit from TAS

A premise of the TAS approach is that intensive, multi-layered support is required to enable schools to turn around.  However, this also assumes that there will be a point where the new practices become embedded and focused external support becomes unnecessary.  While external support is often useful, such efforts are time-consuming for schools and resource-intensive for ERO.

The developmental nature of the pilot meant that sometimes schools did not understand ‘where schools were at’ in the TAS phases.  Schools, boards,  ERO and Ministry stakeholders need clarity about school progress markers.  A key question is, when will the school be at a point where intensive support is no longer necessary?  The school’s progress along the trajectory of school evaluation indicators from ‘needs developing’ to developing is one clear marker.  However, the phases of improvement may need to be made more explicit.

‘I am not sure that we were really clear about the sort of criteria or process around how TAS schools might move out or move on from being a test (pilot) school. They needed have some sort of benchmark or something… Some tools have been shared with us, but they weren’t communicated to the schools in the beginning so they don’t know where they are in the process. You know, if you’re going into a difficult situation it’s much easier if you know what’s this going to look like, and what the process is. And you know, knowing the process helps you feel more comfortable in terms of what you’re moving into.’ (School 3 - Ministry representative)

Ministry representatives understood that when the TAS approach was initiated, there was an urgency to act. Delaying intervention to get ‘all our ducks in a row’ advance would not address the pressing need to address inequity in student outcomes.  The approach has now been seeded, and it may be appropriate to focus on mapping the improvement process, and collaborate with schools to develop localised criteria for progression.

Section 5: Lessons Learned from the TAS Pilot – Implications and Recommendations

The TAS pilot was an ambitious change initiative for the Education Review Office and the Ministry.  The external evaluation of this pilot offers learnings for both organisations, particularly about useful mechanisms to support school improvement in those schools experiencing ongoing performance challenges.  ERO’s mandate includes school and system accountability, however these schools need intensive monitoring, evaluation and support for improvement.  These schools may not have always been well served by traditional approaches to reviews. 

The Ministry and the TAS team were committed to work together with the school on improvements. A lot has been learned about the conditions for effective collaboration from implementation of the TAS pilot. The evaluation found evidence that the strength of the collaborative platform between ERO and the Ministry improved over the course of the pilot. 

The TAS pilot was intensive and targeted. Reviews with the six schools were conducted each term. Rapid cycle monitoring and evaluation approaches were used to catalyse school improvement. The majority of school stakeholders interviewed in this evaluation rated the TAS pilot highly. They noted the contribution of the intensive approach to improvements in the school’s critical reflection on school performance and use of data. External viewpoints on school improvement help to provide an outside account to validate the findings of internal evaluation and to provide challenge.

While there are a range of school improvements that have been made, it would not be appropriate to claim that the six schools do not require further support.  The trajectory of change appears to be positive in all but one of the schools. At least three of the schools still experience barriers that are likely to inhibit the pace and sustainability of change including leadership, governance and situational barriers.

It is clear from interviews conducted in this external evaluation that the TAS approach has offered a very different way of working with schools. From the perspective of some stakeholders there were noticeable differences in the TAS review approach.  The respectful collaboration with schools over time which fostered a sense of camaraderie and joint problem solving instead of a focus on collection of data for judgement and external accountability. 

Team members with specialist skills or knowledge were matched to schools with specific evaluation requirements.  The TAS team worked with schools and with other agency stakeholders to understand the school and community context and promote learning alongside accountability.  While there appeared to be some tension between the two purposes in some instances, most schools and Board Chairs valued the move from a compliance function to a collaborative approach to improvement. ERO’s traditional focus on accountability may have constrained opportunities for a genuine partnership with schools, and may have limited the school’s sense of agency in self-evaluation.

The skills of the TAS team have not been systematically assessed in this evaluation.  Reference has been made to the Team’s views about the necessary skillset and feedback from school stakeholders on the relational and evaluative skills of team members has  been presented.   The skill set of those who work with schools experiencing ongoing performance issues may be a critical consideration. Matching reviewers appropriately to the needs of the school and community context is important for engagement, and is potentially even more critical in this improvement context.

No approach is without weaknesses and stakeholders identified a range of ways to improve the reach and effectiveness of the approach.  The pilot has offered an opportunity to develop and evolve an approach that was appropriate for very different school contexts. The implementation of the pilot, and this evaluation offer a number of learnings from the TAS pilot that may be relevant for ERO and the Ministry in their work with high priority schools. 

The evaluator’s reflections and synthesis of learnings and potential implications for school improvement approaches are presented below. A limited number of recommendations is then provided for consideration by ERO and the Ministry.

Designing a school evaluation approach for schools with ongoing performance challenges –

Learnings from the TAS external evaluation	Implications for designing school improvement approaches
Clearly define roles and responsibilities among stakeholders	-Collaboratively define the purpose and roles of all stakeholders to avoid duplication of effort and facilitate clear messaging to school stakeholders.
Describe the improvement process (stages and exit points) clearly and concisely	-Agreements about the improvement process should be negotiated and agreed with agency partners and with schools.  -Schools need to be engaged in mapping where they are. Readiness (not just expressed commitment to the process) appears to be an important precursor for improvement -Clear, simple guides and reports are more likely to support engagement than lengthy documents. -Establish criteria for transition points so that schools are clear where they are in the process.  where they are going, and how judgements are the end point of engagement with the improvement team will be reached
Formative feedback on improvement in schools is important, as is formative feedback on the improvement project	-Build in structured opportunities for agency and school stakeholders to provide formative feedback about improvement initiatives.  Early ‘teething issues’ may then be resolved swiftly and responsiveness to feedback is likely to facilitate improved engagement.  -The improvement programme should ideally be reviewed progressively by partners, not only at the end in a formal evaluation.
Foster professional relationships that are strengths-based with accountability (on both sides)	-Undertake a systematic diagnostic activity to identify root causes rather than just symptoms.   Focus on the most important changes first. -Ensure a targeted focus in reports on the key issues. Schools need to see their pathway to improvement, including progress and achievements.
A ‘recipe’ for improvement will not work.  The pilot approach has identified the importance of adapting strategies to suit the context of the school.	-Customise improvement strategies to the school and its context.  While the approach needs a strong structure to support fidelity across schools, a ‘recipe’ that is following prescriptively is unlikely to work across school contexts.
Resources and tools for improvement are useful to reviewers, and also to schools and partner agencies	-Provide a clear map of the resources with clear guidance and how and when the tool may be used, and in what circumstances it is appropriate.
Local data is important in identifying issues and mapping progress     Provide opportunities for development of skills in local level monitoring within the school to enhance confidence and engagement in tracking progress	-There may be value in providing support to schools to design, collect and analyse local level data (from teachers, students and community) that connects to specific initiatives schools are implementing or trialling.  -Global outcome measures of student performance are not necessarily sufficiently sensitive to pick up the impact of curriculum or practice changes in the short term.  Schools may benefit from support to collect formative, interim and summative data on student learning and outcomes.
Continuity among review teams contributes to a growth in understanding of context and may foster trusting professional relationships	-There is a need to consider the strengths and limits of a consistent team (or individual) working with schools.  Reviewers working in pairs have the opportunity to test observations with each other and with school stakeholders.
Improvement teams need specialist skills to work with schools with ongoing performance challenges	-Mapping competencies required for working with schools experiencing performance challenges may be valuable.  Identifying required competencies may help identify strengths and identify opportunities for further development.
Share promising practices about what works in particular school contexts	Schools may welcome more explicit guidance from ERO for addressing issues identified in reviews.  This is not characterised as formal advice, but acknowledges that the reviewers have a rich knowledge base about what works in differing contexts to support school improvement.   Some learnings may be transferable between schools, which may also reduce duplication of effort.

Recommendations

1. It is recommended that ERO maintain a dedicated group of experienced evaluators to focus on schools with persistent performance challenges.  The extension of the pilot to a greater number of schools through the High Priority Schools (HPS) approach in 2020 is formal recognition by ERO of the importance of this work. However, the continuation of the pilot level of resourcing (time and scope of work with schools) is likely to be unsustainable as work is extended to a greater number of schools. ERO will need to identify strategies to balance internal resource constraints with their capacity to influence change.
2. It is recommended that the TAS team’s role be extended to provide additional support and/or PLD in monitoring, evaluation and using data for improvement purposes.  The team has developed a sound knowledge base about what works with Turnaround schools within the NZ context, and this information will be useful for ERO, the Ministry and for schools.
3. It is recommended that the learnings from the TAS pilot be used by ERO and the Ministry to inform the work undertaken to shift performance in other underperforming schools.  Learnings could usefully be extended to inform improvement practices within early childhood services as well.
4. It is recommended that schools – principals and potentially Board Chairs - be provided with an opportunity to share lessons learned, and to highlight promising practices in support for school improvement.  The success case profiles in this report highlight some topics that could be explored. Principals would welcome an opportunity to share experiences.  A forum would provide a further opportunity for them to learn about potentially useful strategies that have worked in similar school contexts.  Persistently low-performing schools may be better served learning from schools similar to themselves, rather than ‘best practice’ cases.
5. It is recommended that the tools and resources developed over the past three years by the TAS team be shared more widely across ERO and the Ministry. Additional work may be required to provide guidance on the use of each tool, and the purpose and process of use with schools to ensure they are used appropriately and with fidelity. Some of these tools (e.g., the school radars) could be particularly useful for bringing together qualitative assessments and judgements of the review team with quantitative school outcome data.
6. It is recommended that evaluation mechanisms are built into school improvement approaches to allow for progressive formative feedback.  The explicit inclusion of process evaluation within any school improvement approach also has the benefit of demonstrating that ERO and the Ministry ‘walks the talk’ of evaluation for improvement.
7. It is recommended that a simple map of the improvement phases be developed to increase school and board understanding of progress markers for withdrawal or dial-down of the intensive improvement initiative.

References

Education Review Office (2019). He Taura Here Tangata Strategy

The New Zealand Initiative (2016).  Signal loss: What we know about school performance. Written by Martine Udahemuka

Kumar, K. (1993). Rapid appraisal methods.  The World Bank.

Morgan, D. (2020). Iterative thematic analysis: a new method for analyzing qualitative data.  The International Journal of Qualitative Methods.  Vol 19:1-11

Patton, MQ. (2011) Developmental Evaluation: Applying complexity concepts to enhance innovation and use. The Guilford Press

Appendix 1:  Technical Appendix

A range of stakeholders were interviewed as part of this evaluation. Perspectives from twenty six school and agency stakeholders form the evidentiary basis of claims made in this report. Interviews were completed of between one to two hours in length[16].

In the initial stages an interview was conducted with each member of the TAS team.  These interviews focused on their approach with schools and their overall TAS approach.   As each member of the team was designated as an anchor for a TAS school the interview also represented an opportunity to understand the context of each TAS school, and the key issues emerging in ERO reports.

Interviews were conducted with a range of stakeholders involved in the implementation of the TAS pilot.  Thirteen school stakeholders (Principals, Board Chairs, and LSM – where applicable) participated in individual interviewers. A one hour interview with the directors of the three ERO regions was also conducted. The interviews were either face to face or via zoom and each interview was at least an hour in length.

Interviews with Ministry representatives (Directors, Managers and Advisors) in each region were scheduled over Zoom.  Directors generally invited managers and advisors to these meetings as the direct contact with the schools.  A semi-structured interview was also conducted with Jann Marshall as Manager of School Improvement within the Ministry.  As schools have been de-identified in this report names of principals, board chairs, Ministry and ERO representatives who were interviewed are not listed.

Four schools were visited to conduct interviews with Principals and Board Chairs, and LSM where they were in place.  These visits helped the evaluator to develop a sense of the school in context. Unfortunately, due to Covid-19 restrictions the remaining two schools were not possible to visit.

Readers of this evaluation report should keep in mind the evolving nature of the pilot from inception through implementation, the readiness of the school to make change, and the differing school and community contexts in which the pilot was implemented.  These conditions necessarily influence what was shared, and also the role of the pilot in supporting improvement.

Interview Questions:

Interview questions were developed for each key stakeholder group.  While there were some common questions across all interviews to facilitate comparison, the stakeholder role determined the focus of the questions. The key topics covered in each interview were based on the key evaluation questions, an evolving understanding of the theory of change, and on review of program documentation.

Key topics included:

Perspectives about what works to turn struggling schools around

Views of the TAS Approach:

• The contribution of TAS to school evaluation and improvement
• Strengths and Limitations of TAS
• Level of collaboration and clarity of communication
• Capacity of TAS as a broker for additional resources

School context and needs:

• Changes within the school over the time TAS has been involved
• Contribution of TAS to these improvements (if any)
• Potential sustainability of changes

Opportunities for improvement:

• Suggestions for how TAS approach could be improved (with reflection on implementation)
• Lessons learned

School stakeholders were asked to rate five outcome dimensions at the end of the semi-structured interview.  These related to key outcomes identified on the logic map (see page 21) including the clarity of communication about the TAS pilot, the value of TAS in enhancing the school’s use of data for improvement, the usefulness of TAS reports, the value of TAS in brokering additional support and/or resources for the school, and overall usefulness.  Each individual was asked to rate the TAS pilot on each dimension on a scale of 1 (not valuable/useful to 5 (extremely valuable/extremely useful.

Analysis of the Interviews

Interviews were conducted between 5 July to 12 October, 2020 by the external evaluator.  The interviews represent the stakeholders’ perspective at the time of the interview, and it is important to be aware that these perspectives are time-bound.  Perspectives may have shifted or may potentially shift in the future.   The interviews provided an opportunity for the interviewee to identify their perspective on the value of the TAS approach overall.

Transcripts of all[17] interviews were prepared to ensure transparency, and a sound basis for claims.  They were particularly useful as a supplement and elaboration of interview notes as the evaluator has limited experience within the NZ education system.   

Most of the formal analysis of the interviews was conducted after all the interviews were completed.  However, immediately after each interview the interview was summarised and then the summary checked against and elaborated following transcription.

NVivo (software that supports the management of qualitative data) was used to collate, classify, and cluster interview material by stakeholder role, by connection with a specific TAS school, and by key evaluation question.

Iterative thematic analysis (Morgan, 2020) was adopted as the core analytic framework.  In public policy contexts and evaluation with tight timelines, the traditional inductive approach to thematic analysis was not warranted.  The analysis was therefore guided by outcome domains and key evaluation questions. 

The interviews were summarised individually, then clustered with other interviews pertaining to the school. A comparative analysis was also undertaken to determine:

- Perspectives across stakeholder roles.  This involved examining the similarities and differences in perspectives of Ministry of Education representatives, ERO representatives, and principals, LSMs, and board chairs.

- Perspectives pertaining to each school.  All interviews pertaining to a particular school were collated and analysed as a set. 

Two ERO feedback forums were held to provide an opportunity to test out the findings and collaboratively generate recommendations.   The first was an internal forum to discuss the draft report.  The second forum included senior Ministry representatives. 

Interviews conducted as part of the External Evaluation

Key Informant Interviews

Role within ERO	Interviews conducted
The TAS team (6)	Six interviews – Members of the team were also asked to participate in verification checks in draft and final reports
Di Anderson (Deputy Chief Executive Review Improvement.    Note:  Di Anderson left this role in May.  The position is now held by Jane Lee	Informal interview – background to the pilot and policy drivers
Regional Directors (3)	1 Zoom online interview

Interviews with School and regional Ministry Stakeholders

TAS School	Regional directors, Managers and advisors[18]	Principal	Board Chair	LSM
School 1	Regional director, manager and school advisor (3)	1	1	Not applicable
School 2	Director and manager (2)	1	1 and 1 previous board chair	LSM was a new appointment (not interviewed)
School 3	Director, manager and school advisor (3)	1	1	Not Applicable
School 4	Director, manager and school advisor (3)	Yes	1	1
School 5	Director, manager and school advisor (3)	Yes	1 and 1 previous board chair	1
School 6	Director	Yes	No (family bereavement)	Not applicable

Acting Associate Deputy Secretary Jann Marshall

 

Evaluator Assumptions

An evaluator needs to be explicit about the processes and outcomes the evaluation is focused on, and the criteria for assessing effectiveness. The three points below clarify my working assumptions of evaluation that have shaped my understanding of TAS and the conduct of the evaluation. 

 

Context Matters

The schools were diverse in size, geographical location, cultural history, communities, and stability of key leaders.  An understanding of context appeared to be important in developing the TAS approach.  Initiatives need to tailored and targeted to the needs of the school in terms of geographical location, leadership and history.  A one-size-fits-all approach will not be workable.

I bring to this evaluation a perspective that similar resources and strategies sometimes generate very different results in different contexts. This necessitates that the external evaluator has some understanding of the differing school contexts.  I acknowledge that a single school visit offers a very limited understanding of context.

 

Sustainable change is a long-term process, not a short-term event

Any targeted intervention has the potential to lift quality temporarily.   However, the sustainability of quick gains is questionable when other support conditions are not in place. Sustainability of improvement is more likely when the conditions within the school environment are present and maintained. While pockets of good practice may be identified, schools cannot maintain high quality without some foundational conditions and capacity being in place.

Timely and focused feedback each term is likely to assist the school’s understanding and capacity to change. Conversations about educational practices and reflection have the potential to reinforce educators' confidence and capacity to implement improvements.

 

Contribution not attribution

The relationship between the work done as part of pilot and the results or signs of progress in schools is complex and non-linear.  While outputs (number of review reports, review visits within schools) are reasonably proximate and traceable to the pilot, outcomes are interdependent and influenced by a range of contextual factors. It is unlikely that the TAS initiative or any change initiative on its own will be the silver bullet.  A more reasonable focus is on assessing contribution rather than attribution.

 

Judgements about effectiveness

A key question orienting this evaluation is concerned with effectiveness. To address questions of effectiveness information was retrieved about the patterns in school level data from baseline (2016), and from exploring stakeholder perspectives of the TAs pilot.  

Qualitative data: perceptions of key stakeholders was used to generate a story of how the pilot worked in schools.

Quantitative data: The evaluator did not collect or independently analyse the quantitative data.  Rather, data was provided to the evaluator by the TAS team from baseline to the most recent available data collection period (2019).  School data of this nature gives a more global indication that student outcomes are improving. The data included NCEA external achievement standards, School leaver data, attendance data.  NCEA standards data was considered most useful as indicative of school improvement.

A generic rubric provided the basis for evaluating progress according to each of the outcome areas. Each data source was converted into a rating. The following table briefly outlines the process used to make those conversions.

Generic rubric and synthesis process

Rating	Quantitative data	Qualitative Data
Very effective	Improvement in key quantitative data associated with identified changes	All Stakeholders (school and board/Ministry) identify the pilot as effective/valuable in supporting improvement
Effective	Improvement in some of the quantitative data at the school level across schools	-Most stakeholders identify the Pilot was effective and/or valuable to the school -Ministry of Education indicate ways that the pilot has contributed to school improvement
No clear evidence on effectiveness	Little or no change in any of the quantitative measures	-Stakeholders do not associate any of the observed school level or student outcomes to the pilot
Ineffective	No shift in any of the quantitative measures	-Stakeholders express clear doubt that the TAS team did anything of value in the school, and/or indicate the project made no difference in the school

Appendix 2: School Stakeholders - Raw Ratings Data

School 1

Stakeholder group	Overall value of TAS to school improvement	The TAS reports were useful to the school	School has enhanced capacity to use data for planning and improvement	Capacity of TAS to broker additional support for the school	Clarity of Communication
Principal	3	3.5	4	4	4 (18.5/25)
Board Chair	4	4	3	4	4 (19/25)

School 2

Stakeholder group	Overall value of TAS to school improvement	The TAS reports were useful to the school	School has enhanced capacity to use data for planning and improvement	Capacity of TAS to broker additional support for the school	Clarity of Communication
Principal	4	4	3	3	2 (16/25)
Board Chair	3	3	2	3	3 (14/25)
Previous Board Chair	5	4.5	5	5	4.5 (24/25)

School 3

Stakeholder group	Overall value of TAS to school improvement	The TAS reports were useful to the school	School has enhanced capacity to use data for planning and improvement	Capacity of TAS to broker additional support for the school	Clarity of Communication
Principal	4	5	5	4	4 (22/25)
Board Chair	5	4	3	4	4 (20/25)

School 4

Stakeholder group	Overall value of TAS to school improvement	The TAS reports were useful to the school	School has enhanced capacity to use data for planning and improvement	Capacity of TAS to broker additional support for the school	Clarity of Communication
Principal	4	3	4	3	3 (17/25)
Board Chair	3.5	3	3	Not able to rate this	4 (13.5/25)*
LSM	4	4	Not able to rate this	3	4 (15/25)*

School 5

Stakeholder group	Overall value of TAS to school improvement	The TAS reports were useful to the school	School has enhanced capacity to use data for planning and improvement	Capacity of TAS to broker additional support for the school	Clarity of Communication
Principal	5	3	4.5	4	4.5 (21/25)
New Board Chair	2	3.5	1	Not able to rate this	1 (6.5/25)*
Previous Board Chair	5	3	4	5	4 (21/25)
LSM	5	3	5	4	4.5 (21.5/25)

 

School 6

Stakeholder group	Overall value of TAS to school improvement	The TAS reports were useful to the school	School has enhanced capacity to use data for planning and improvement	Capacity of TAS to broker additional support for the school	Clarity of Communication
Principal	4	2	4	4	4 (18/25)

Averages	Overall value of TAS to school improvement	The TAS reports were useful to the school	School has enhanced capacity to use data for planning and improvement	Capacity of TAS to broker additional support for the school	Clarity of Communication
Average	4/5	3.5/5	3.5/5	3.8/5	3.6/5
Range	2-5	2-5	1-5	3-5	1-5
Mode	4	3	4	4	4

 

Appendix 3: School Summary Tables

School Summary Table

School	Experience and Achievements	Themes	Ministry Support
School 1   Medium-sized school   Majority of students identify as Māori   Overall Judgment:  Needs development	Positive view about the pilot – with caveats re perceived limits in acknowledgement of the school context   Achievements: Whānau structures for learning   I am module to promote literacy and engagement   Progress in NCEA level 2   Community connections and leadership   The school has demonstrated improved leadership within the community and enhanced teacher capability within the school	-Nominated into the pilot by the local regional office of the Ministry -Source of tension for the school in ERO’s ‘relentless’  focus on data with what appeared to be a limited understanding of the context of the school, and the community - Some frustration expressed by the principal of the ‘long list of issues’ that needed to be addressed in ERO reports. -View that pre-conditions for learning should be considered before engaging in formal assessment (and production of global data)	-PRA -Professional learning and development in curriculum leadership -Specialist literacy leadership role -Resources for the library to promote school wide literacy
School 2 Small school Year 1-8   All students identify as Māori   Overall judgement: needs development	Mixed views about the value of the Pilot   Note: This school has struggled to progress required improvements. ERO and the Ministry recommended a Commissioner be  appointed.  The appointment was made in 2020   Achievements: No comparable quantitative outcome data   Development of a positive behaviour for learning approaches to build a more positive school culture   Curriculum focus local cultural knowledges and history   Schoolwide focus on student wellbeing	- School welcomed the TAS Pilot as an opportunity to get ‘direct support’ - Board is pro-active in engaging with the community - Board appeared to be confused about role differences between ERO and the Ministry	-PRA -LSM -Principal was released from teaching to re-develop operating systems and processes, including assessment   -PLD opportunities for teachers and the Principal -SAF appointed
School 3   Small rural school Years 1-13   Overall Judgement:  Developing	Extremely positive view of the TAS Pilot.   Before the current principal was appointed in 2017, the school had seen 12 Principals in 10 years.  Community support for the school was low.     Key achievements:   Community support for the school is increasing   Distributed leadership approach to promote consistency and sustainability   Teacher capability year 1-8 maths.   Stable Board   Individual tracking of students   Whānau partnerships strengthened   Years 7 to 8 and Years 9 to 10 classes are receiving more appropriate teaching programmes focused at their level of learning.   Years 1 to 10 are making progress in reading, writing and Mathematics (positive trajectory, but still relatively low)   School values that support more consistent and positive behaviour for learning   Property development and renovation	-Welcomed support for improvement -Strong collaborative platform between agency stakeholders (ERO and the Ministry) and the school   - A strong expectation that the TAS Pilot would offer an approach different from ‘standard’ ERO reviews   -Continuity of the TAS anchor, and relational and evaluation skills of the team enhanced trust in the pilot process   -Creation of distributed leadership model promoted consistency in messaging across the school, and may contribute to sustainability -Strong Interest in using data and enhancing evaluative thinking	-PRA -Student Achievement Function (SAF) practitioner -Assistance from a senior advisor and learning support advisor -Teacher professional learning and development (PLD) in mathematics and some specific support in literacy -Resource Teachers of Learning and Behaviour and Literacy (RTLB and RTLit).
School 4   -Large school -Year 9-13   Classification: Developing	Extremely positive view of the TAS pilot   Key Achievements   Significant improvement in NCEA Level 1 2017 to 2018.   Building of leadership capability of senior leadership team and Heads of Faculty. Improved school systems, particularly in data monitoring and tracking.   Ongoing data tracking (data manager) Improving Outcomes NCEA L1, L2   Re-invigorated board with growing capability in governance   Community consultation by the Ministry informed ongoing curriculum developments.   Inclusion of culturally responsive programmes within the school	-Welcomed the TAS approach and support for improvement -Principal found it ‘professionally very rewarding’ and valuable for the school to be involved -Sometimes identified a disconnect between verbal and written reports -Reports were usually ‘spot on’ in terms of the issues raised/resonated with the Principal and the Board Chair -A focus on improvement rather than just ‘accountability’	-LSM -NZSTA (mixed views about effectiveness)
School 5   -Large school -Years 9-13   Majority of students identify as Māori or from Pacific nations	Positive view about the pilot   Before the current principal was engaged the school had poor school-community relationships   NCEA levels 1-3 close to decile band average   Principal was pivotal in shifting the school culture and connecting with local community   School is steadily growing   School climate and student voice is strong   More students completing external NCEA standard end of 2019	-Strong relational leader who focused on getting the condition right for learning (cultural and community engagement) - Board appeared to be confused about the difference between the LSM responsibilities and objectives and the TAS Pilot -Collaborative platform resulted in frank, robust discussions.  An openness on both sides to challenge -Continuity of the team built a trusting, open relationship and made it easier to have difficult conversations -Identified a disconnect before verbal and written reports -The new board chair has a strong focus on performance measurement	-PRA -LSM -NZSTA   Seconded DP for leadership of curriculum
School 6   Small rural school Composite school Year 1-15   The majority of students are Māori   Overall judgement: needs development	Mixed views about the pilot.    Improved NCEA Level 1 and 2 Literacy/Numeracy   Local curriculum underway   Staff capabilities in te reo Māori, maths and literacy   Whānaungatanga/positive school climate	-Isolated rural community -‘Mana’ of the Principal. - Principal focused attention on settling behavioural issues in the school for first year of his appointment - Challenged some of the interpretations of ERO.  Preferred continuity in team as a relationship had been established with key members - Attention to ERO requirements sometimes got in the way of innovation and trying new things to turn the school around.	-Two FTTE positions one dedicated as a SENCO -Property team remediation in 2018, and property team support and design of master plan/template Student Achievement Function personnel (SAF) -Additional resourcing for students at risk

---

[1] In this report reference is made to students who identify with Pacific ancestry, or from Pacific nations.  The term ‘Pasifika’ student is commonly used to refer to students who are from the Pacific Islands or identify with the Pacific Islands in terms of ancestry or heritage. I acknowledge that the term does not fully capture the diversity within particular communities.

[2] COVID-19 restrictions and school closures delayed the initial timeframe for beginning fieldwork.

[3] This external evaluation focuses on contribution rather than attribution.

[4] The evaluator scheduled visits to all six schools.  However, during the planned fieldwork, a second outbreak of COVID-19 occurred, and these visits were cancelled.  Zoom interviews were conducted with the principals and board chairs in these two schools.

[5] In early 2020 the emergence of Covid-19 presented another event that necessarily will have implications for schools, and disrupt patterns in school outcomes for some time.

[6] In the logic map on the previous page the key propositions are referred to as working assumptions.  These were examined as part of the evaluation.

[7] Multiple studies show that schools that use data in a comprehensive and systematic way have improved teaching practices, and better student outcomes (Dobbie & Fryer. 2013; NSW Centre for Education Statistics and Evaluation)

[8] These claims were derived from interviews with members of the TAS team.  They have not been formally verified, but the evaluator has seen planning tools that align working practices with the Strategy.

[9] For elaboration of the way Kaupapa Māori informs evaluation see the work of Linda Tuhiwai Smith, Nan Weipeihana and Fiona Cram.

[10] While the HPS pilot is not the focus of this evaluation, the team used the learnings from the pilot to frame their Developmental Evaluation approach.  The approach was aligned with the He Taura Here Tangata Strategy (launched in Late July, 2020)

[11] Turnaround High School Principals: Recruit, Prepare and Empower Leaders of Change.  Southern region education Boardm Georgia, USA

[12] Two of the three LSMs were interviewed.  The LSM from the third school had only recently been appointed so it was decided that an interview may not be useful in the context of the information requirements of the external evaluation.

[13] The TAS team review reports provide a sufficient evidence base about what worked and what didn’t work within each school.  Judgements were based on observation, interviews and data collection across several terms.   For further information about these assessments please refer to specific ERO reports.

[14] Quantitative summary data from each of the schools was compiled by a TAS team member with responsibility for data collation.

[15] One TAS team member referred to the historical emphasis of ERO as a ‘Scriven-driven’ approach to evaluation.  Michael Scriven is a key figure in the evaluation field, who has clear views on the weaknesses of internal evaluation and pseudo-collaborative approaches to evaluation.

[16] Note:  Some interviews with Ministry and ERO stakeholders were conducted as small group interviews, not individually.

[17] One interview was not transcribed as it was not recorded due to a failure of equipment.  Full notes were taken to inform analysis

[18] Counted as 1 interview as conducted in a zoom group interview