In this paper, Robinson discusses the inherent tension between evaluation for accountability and evaluation for improvement. As a means of managing this tension, the links between the two are often kept loose. The downside of this is that the school and external reviewers often end up evaluating different things instead of discussing why their judgments are at variance.
As a strategy for achieving both accountability and improvement purposes, the author proposes that a common set of indicators (possibly supplemented by additional, optional indicators) be used by both schools and external reviewers. If this is done, school leaders and external reviewers can bring their own provisional evaluations to the table and discuss the evidence and reasoning behind them, with the aim of reaching a consensus.
Over time, this will enable the development of shared understandings about what the indicators mean and what meeting them to a high standard looks like. Importantly, the process will also build the evaluative capability of all those involved. A further benefit will be a much clearer system-level picture of the strengths and weaknesses of our schools.
A problem with the existing indicators is that they are often too vague to be useful. Robinson argues that the new indicators should be strongly normative, communicating in the wording what counts as high quality.
The author was tasked specifically to provide critical commentary on the current Dimension 3 (Leading and managing the school) indicators. She argues for a broad conception of leadership that encompasses both formal and informal leadership, and for effective leadership to be understood as leadership that has a significant effect on student achievement and well-being. She then goes on to briefly discuss and make recommendations on each of Dimension 3’s ten sub-dimensions and indicators.
Introduction
The purpose of this paper is to suggest ways in which the ERO reviews and indicators, especially those relevant to Dimension 3: Leading and managing the school, can be revised to achieve important system goals. Arguably, the most critical challenges facing the New Zealand education system are persistent large disparities in educational achievement between different social groups and a declining level of overall excellence (May, Cowles, & Lamy, 2013). In mathematics, the situation is particularly serious, with New Zealand showing a negative trajectory of change in performance on PISA assessments between 2003 and 2012 (OECD, 2014).
The cycle of reviews conducted by ERO provides a potentially powerful lever to challenge schools and support them as they address these achievement challenges. Since there are very few such levers in our self-managing system, I am suggesting that ERO be explicitly positioned as an agency that promotes school improvement through tightly linked processes of self- and external review. The current review framework is already intended to promote continuous improvement through school self-review, which is seen as complementary to ERO’s external reviews (Education Review Office, 2011). In the following section I outline a theory of improvement that shows how tightly linked self- and external reviews could promote improvement across the schooling system.
A theory of improvement
A theory of improvement communicates the logic that links purpose, strategies and intended outcomes.
In the case of ERO reviews, school accountability and improvement are both important purposes. The accountability purpose is met by providing a public report on the extent to which a school has established the conditions required to meet the goals of the New Zealand Curriculum (Education Review Office, 2011, p. 3; New Zealand Ministry of Education, 2007) and met all its regulatory responsibilities. The improvement purpose is pursued by designing an external review process that encourages continuous self-review processes in schools. One output of these self-review processes are the judgments made by school leaders about the extent to which their school has met the indicators used by the external reviewers.
There are challenges in designing evaluation processes that achieve both accountability and improvement purposes. One common strategy for reducing the tension between them is to make the links loose rather than tight. School leaders may be given considerable choice about the focus of the evaluation and the indicators to use.
The problem with loose alignment between internal and external processes is that it rules out reciprocal feedback about the quality of the judgments made by the school and the external reviewers. The focus of the feedback is on different aspects of school performance rather than on the validity of the evaluative judgments made by each review team. If, however, internal and external reviewers make evaluative judgments based on a common core set of indicators, then the initial judgments of both parties can be treated as formative. The summative evaluation, which is publicly reported, should be the outcome of a rich dialogue about the validity of the judgments that they have made. Evaluative capability is built in both parties as they discuss the evidence and reasoning that led to their various judgments.
So the suggested strategy for achieving these two purposes is the use by both school leaders and ERO reviewers of a common set of indicators. This alignment is essential for building a common understanding of what the indicators mean and what meeting them to a high standard looks like. There are also system-wide benefits in the use of a common set of indicators in that policy makers will be able to discover the indicators on which schools perform more and less well – information that can then be used to target appropriate support.
A prescribed set of common indicators could be supplemented with additional optional indicators so that schools can exercise some choice about the focus of a review.
Greater system-level coherence could be achieved by using a common set of core indicators for ERO reviews and for other evaluative processes, such as planning and reporting, that are required by the Ministry of Education.
The effectiveness of this theory of improvement could be judged over time by the extent to which it has contributed to: greater public knowledge about what constitutes a high performing school, increased numbers of schools who are judged to be high performing, and more equitable and higher student outcomes.
The indicators of effectiveness
“An indicator can be defined as an item of information collected at regular intervals to track the performance of a system” (Fitz-Gibbon & Tymms, 2002, p. 2).
Indicators of school effectiveness should communicate clear signals about what is required to achieve the goals expressed in the New Zealand Curriculum and focus attention on the things that matter most. While it is desirable that the number of indicators be restricted for practical reasons, it is equally important that the indicators in the core set capture a range of valued outcomes and the processes that are critical for their achievement. The use of a range of indicators prevents gaming of the system and an undesirably narrow concept of what counts as school effectiveness.
I argue for a set of core indicators that are aspirational, directly linked to review purposes and prescribed. While the current indicators are intended to “provide a clear description of the kinds of outcomes, behaviours and practices ERO would expect to see in a high performing school” (Education Review Office, 2011, p. 7), many of them are so vague that they could be interpreted as reflecting low, medium or high performance. For example, a reviewer might conclude that the indicators relating to leadership opportunities were being addressed on the grounds that management units had been allocated to staff. The problem with such descriptive indicators is that they do not communicate the qualities required to translate (in the example) leadership opportunities into educational benefits for teachers or students. To do this, the indicators need to be strongly normative, i.e. they need to communicate what counts as a high quality leadership opportunity.
A core group of indicators should be prescribed for use in every review (not exemplars which may or may not be used). These core indicators should be based on the best evidence we have about the conditions and practices required to achieve the overarching goals of the New Zealand Curriculum. The hospitals analogy is instructive: while hospitals vary greatly in size, type, specialties, etc., there are some qualities and practices we would expect to find in every hospital. These relate to the core business of making patients better and meeting health and safety standards. Other indicators may be specific to the type of hospital or reflective of local conditions. Even though arguments can be made for additional context- specific indicators, this work should not overshadow the primary purpose of developing a prescribed set of core indicators. A common set of core indicators is vital for maintaining a strong national public schooling system.
Given a clear set of aspirational indicators we can begin to develop shared understandings about what it means to be high, medium and low on each of them and engage in rich discussions analogous to those that teachers have when moderating students’ assessments. When self- and external review processes are tightly aligned through the use of the same set of indicators, internal and external reviewers are mutually accountable for the judgments they make. This means both parties are seeking to answer the question, “What does high performance look like on this indicator, in this context?”
Critical commentary on Dimension 3: Leading and managing the school
The first evaluative question, “How effective is professional leadership throughout the school”, signals two important concepts. The first is effectiveness, which is explained as leadership that has “a significant effect on student achievement and well-being”. This is a student-centred concept of leadership effectiveness as it judges quality by impact on students rather than adults (Robinson, 2011). Given the need for a relentless focus on equity, a revised definition could refer to both excellence and equity of student achievement and well-being.
IMPLICATION
Effective leadership is by definition “leadership that has a significant effect on the excellence and equity of student achievement and well-being.”
The second important concept is found in the words “throughout the school.” This signals that the focus of the review is on school-wide leadership, not just the principal. Given that school-wide or distributed leadership is often mistakenly understood to include only those with formal leadership roles, it would be helpful to have a statement that is more explicit about the meaning of school-wide leadership.
IMPLICATION
Leadership should be understood to embrace both formal and informal leadership so that teachers feel a collective responsibility to influence each other in the direction of improvement (R. D. Goddard, Hoy, & Hoy, 2003; Y. L. Goddard, Goddard, & Tschannen- Moran, 2007).
The second evaluative question, “How well does school leadership use evidence to make decisions?” and those that follow focus on specific dimensions of leadership. They do not match the ten sub-dimensions that are elaborated in the subsequent sets of indicators.
IMPLICATION
There should be either one evaluative question, “How effective is professional leadership throughout the school?” or one question for each of the subsequent sub- dimensions.
The fourth evaluative question asks, “How well do the school’s procedures and practices align with policies and directions?” Alignment and coordination are crucial for school improvement. The research literature is replete with examples of externally or internally driven initiatives and strategies that are not coherent and that overlap, producing overload, stress and burnout.
IMPLICATION
The indicators should be revised to give more emphasis to the pursuit of a few important goals and the alignment of school policies and practices to achieve those goals.
I now turn to more specific consideration of the sub-dimensions and indicators in Dimension 3: Leading and managing the school. This dimension is divided into the following 10 sub-dimensions:
- Establishing goals and expectations
- Strategic resourcing
- Designing, evaluating and coordinating the curriculum
- Coordinating and evaluating teaching
- Leadership opportunities
- Promoting professional learning
- Management in a positive environment that supports learning
- Self-review
- Analysis and use of assessment data
- Links with community.
There is considerable evidence that the leadership practices described under most of these sub-dimensions are linked to student outcomes (Leithwood, Harris, & Hopkins, 2008; Robinson, 2011; Robinson, Hohepa, & Lloyd, 2009). I suggest, however, that Leadership opportunities and Analysis and use of data should not be standalone categories. The reasons are elaborated in the following discussion of each sub-dimension.
1. Establishing goals and expectations
Numerous research studies employing both quantitative and qualitative methods have demonstrated the importance of this leadership dimension (Leithwood et al., 2008; Robinson et al., 2009). The current indicators emphasise the articulation of goals and an inclusive process for setting them. There are, however, other qualities that are critical for effective goal setting and goal achievement and the suggested revisions focus on these qualities. Discussion is needed about how these indicators can add value, given they are also monitored by the Ministry of Education when it reviews school charters.
- School goals and targets are set after inclusive discussions of the aspirations of students and whānau.
- School goals and targets are set to accelerate the learning of priority groups.
- Leadership uses evidence to monitor progress towards priority goals and to adjust strategies to ensure goals are met.
- Teachers, parents and students take collective responsibility for the achievement of relevant goals and are committed to achieving them.
2. Strategic resourcing
I recommend revising the current indicators so that they can be used to evaluate how well leaders align external regulatory and policy requirements with internal school priorities (Mintrop, 2012). Many leaders struggle to integrate external policy imperatives with internally generated priorities and, as such, risk setting up an opposition between the two that erodes commitment and trust. The suggested revisions also include indicators of leaders’ skill in strategic thinking about the relationship between problems, goals, causes and possible solutions. This skill is critical for avoiding a fragmented approach to improvement that ends up wasting teacher time and effort.
- School leadership buffers and integrates external policy requirements and initiatives in ways that protect and serve the achievement of priority school goals.
- School leadership adopts improvement strategies after collaborative investigation of the school-based causes of the existing situation and inquiry into the likely effectiveness of the proposed strategy.
- The allocation of staffing, money and time is clearly aligned to the school’s priority goals.
3. Designing, coordinating and evaluating the curriculum
There are six evaluative prompts and 10 indicators for this sub-dimension and, while they capture the key ideas of curriculum alignment, progression and engagement, they omit the issue of access to the most demanding pathways. Given the prevalence of within-school tracking in New Zealand schools and the associated large within-school variance, the allocation of students to curriculum pathways could be a focus.
- The curriculum is inclusive, responsive, and clearly aligned to the New Zealand Curriculum.
- The curriculum is reported by Māori and non-Māori students to be engaging.
- Leadership uses assessment data to evaluate the quality of the curriculum offered in various learning areas.
4. Coordinating and evaluating teaching
Many leaders are reluctant to exercise strong leadership in this area because they are not confident of their theory of effective teaching. Others may rely on a checklist or style-based approach which is not defensible in terms of the research evidence. A principled approach is needed that does not privilege a particular style of teaching but evaluates its effectiveness by the quality of the learning opportunities provided to students. In this view, effective teaching “maximises the time that learners are engaged with and successful in the learning of important outcomes” (Berliner, 1987). Rather than evaluate the teaching behaviours, the Berliner model evaluates teaching based on student engagement and success. The suggested revised indicators are:
- The school’s leadership acts as collectively responsible and mutually accountable for the quality of teaching.
- Teachers receive frequent feedback on their teaching and describe it as both challenging and supportive.
- Leaders ensure that undesirable variation in teaching effectiveness is identified and addressed in a timely and effective manner.
5. Leadership opportunities
Leadership development should be a fully integrated part of the development of every teacher. Strong, distributed teacher leadership is essential for the success of a professional learning community and is also critical for developing strong collective whānau/teacher responsibility for student achievement and well-being. Leadership involves the exercise of influence to progress important shared tasks. Since many opportunities for leadership, such as in team meetings, arise in daily interaction, it can be developed by observing, coaching and giving feedback about such interactions. Student outcomes are likely to be better served by such job-embedded leadership development than by the provision of more decontextualised leadership opportunities (Robinson, 2001).
6. Promoting professional learning
This dimension represents the most powerful way that leaders are able to have an impact on student outcomes (Robinson et al., 2009). The evaluative prompts for this dimension are too descriptive, and suggest that high quality involves such actions as providing a range of professional learning opportunities, which in themselves are unlikely to achieve excellence and equity purposes. The normative quality of these indicators can be improved by revising them in ways that include reference to purpose, alignment and differentiation.
Recent research on instructional leadership (Grissom, Loeb, & Master, 2013) has shown that instructional leadership practices such as making classroom visits and giving feedback have a positive impact on student outcomes if the purpose of the visits is to build teacher capability. If the purpose of visits or ‘walk-arounds’ is to check the implementation of particular teaching practices, or simply to be seen, the impact is likely to be negative.
The agenda for teacher professional learning should be derived from ongoing analysis of students’ learning needs, so that teachers become increasingly skilled in lifting engagement and achievement. Teacher professional learning should be differentiated so that teachers are learning what they need to learn next to improve their capability. For example, teachers may need to learn how to manage the classroom environment so that groups can work independently before they implement an initiative designed to foster a community of mathematical thinkers and problem-solvers. Revised indicators should address initiative and professional learning overload by stressing a progression of teacher learning opportunities that are differentiated and aligned to priority student learning goals.
These ideas could be captured in such indicators as:
- Teacher professional learning goals are based on evidence-based inquiry into student and teacher learning needs.
- Every teacher has clear professional learning goals, a plan for achieving them, and support for doing so.
- Human resource procedures (e.g. induction and appraisals) are aligned to and support the achievement of teacher professional learning goals.
- Senior leaders evaluate and build the capability of teacher leaders to carry out their delegated responsibilities for the improvement of teaching and learning.
- The school is organised in ways that enable focused and efficient individual and collective pursuit of teacher and leader professional learning goals.
7. Management in a positive environment that supports learning
This sub-dimension is foundational to the other leadership dimensions. If school routines are not effective and efficient there is little time and energy available for improving teaching and learning. The existing indicators are sound though the separation of communication is problematic as it is integral to all the indicators.
8. Self-review
In my suggested theory of improvement the key measure of the quality of self-review is the extent to which leaders can explain and defend their own evaluation – whether the school is high, medium or low (using whatever nomenclature is agreed) – on each of the sub- dimensions. When the internal and external reviewers discuss their respective evaluations the dialogue will provide valuable insight into the leaders’ capabilities on this sub- dimension.
One indicator will suffice:
- School leadership has made an accurate and defensible holistic evaluation of the school’s performance on each of the dimensions.
9. Analysis and use of assessment data
I suggest that sub-dimension 9 is integrated into the other sub-dimensions so that it is evaluated in the context of important tasks such as evaluating teaching and learning. This is already implied by the use of the phrases ‘evidence-based’ and ‘based on analysis of student learning’ in various indicators.
10. Links with community
There is very little evidence about the leadership practices involved in engaging the community in ways that make a difference to student outcomes. What there is, instead, is considerable evidence about the types of community engagement strategies that have greater or less positive impact on student outcomes. This evidence suggests that if efforts to engage the community are to have an educational pay-off, they need to involve educational activities. Non-educational forms of engagement, such as fundraising and governance, do not have any discernible impact on student achievement (Borman, Hewes, Overman, & Brown, 2003; Robinson et al., 2009). However, as some forms of educational engagement have negative effects on student outcomes, leaders need to know what differentiates the strategies that have positive effects from those that have negative effects. They also need to be able to work with community groups and leaders in ways that build trust so that previously absent voices can engage with teachers in the work of ensuring success (Bryk & Schneider, 2002). Some of the existing indicators do capture these points. My suggested revised indicators for sub-dimension 10 are:
- Leaders integrate community and cultural resources into relevant aspects of the school curriculum.
- Māori and non-Māori parents report that the school welcomes and guides their active involvement in their children’s learning.
- Leaders seek feedback from the whole parent community about the effectiveness of their various communications with the school community.
Conclusion
The table below summarises the recommended shifts in the number and names of the sub- dimensions included in Dimension 3.
|
|
Existing |
Revised |
|---|---|---|
|
1 |
Establishing goals and expectations |
Establishing goals and expectations |
|
2 |
Strategic resourcing |
Resourcing strategically |
|
3 |
Designing, evaluating and coordinating the curriculum |
Designing, evaluating and coordinating the curriculum |
|
4 |
Coordinating and evaluating teaching |
Coordinating and evaluating teaching |
|
5 |
Leadership opportunities |
Omit |
|
6 |
Promoting professional earning |
Leading professional learning |
|
7 |
Management in a positive environment that supports learning |
Managing in a positive environment that supports learning |
|
8 |
Self-review |
Leading self-review |
|
9 |
Analysis and use of assessment data |
Omit |
|
10 |
Links with community |
Engaging the community in education |
The revised indicators that I have suggested do not include the capabilities that leaders need in order to create the conditions described by the indicators. Leadership capabilities such as building relational trust, problem solving, cultural responsiveness, having Open-to- Learning™ conversations, and being highly knowledgeable about the research evidence, describe the dispositions, skills and knowledge that leaders need to create conditions that are conducive to achieving the school’s goals. Evaluation of these capabilities should be part of leadership appraisal and development rather than part of school effectiveness evaluation.
References
Berliner, D. (1987). Simple views of effective teaching and a simple theory of classroom instruction. In D. Berliner, & Rosenshine, B. (Ed.), Talks to Teachers. New York: Random House.
Borman, G. D., Hewes, G. M., Overman, L. T., & Brown, S. (2003). Comprehensive school reform and achievement: A meta-analysis. Review of Educational Research, 73(2), 125- 230.
Bryk, A. S., & Schneider, B. L. (2002). Trust in schools: A core resource for improvement. New York, NY: Russell Sage Foundation Publications.
Education Review Office (2011). Evaluation Indicators. Wellington, New Zealand: Education Review Office.
Fitz-Gibbon, C. T., & Tymms, P. (2002). Technical and ethical issues in indicator systems: Doing things right and doing wrong things. Education Policy Analysis Archives, 10(6)
Goddard, R. D., Hoy, W. K., & Hoy, A. W. (2003). Collective efficacy beliefs: Theoretical developments, empirical evidence, and future directions. Educational Researcher, 33(3), 3-13.
Goddard, Y. L., Goddard, R. D., & Tschannen-Moran, M. (2007). A theoretical and empirical investigation of teacher collaboration for school improvement and student achievement in public elementary schools. Teachers College Record, 109(4), 877-896.
Grissom, J. A., Loeb, S., & Master, B. (2013). Effective instructional time use for school leaders: Longitudinal evidence from observations of principals. Educational Researcher, 42(8), 433-444.
Leithwood, K., Harris, A., & Hopkins, D. (2008). Seven strong claims about successful school leadership. School Leadership & Management, 28(1), 27-42.
May, S., Cowles, S., & Lamy, M. (2013). PISA 2012: New Zealand summary report.
Wellington, New Zealand: Comparative Education Research Unit Research Division, Ministry of Education New Zealand.
Mintrop, H. (2012). Bridging accountability obligations, professional values and (perceived) student needs with integrity. Journal of Educational Administration, 50(5), 695-726.
New Zealand Ministry of Education (2007). The New Zealand Curriculum. Wellington: Learning Media.
OECD. (2014). PISA 2012 results: What students know and can do – Student performance in mathematics, reading and science. Paris: OECD Publishing.
Robinson, V. M. J. (2001). Embedding leadership in task performance. In K. Wong & C. Evers (Eds.), Leadership for quality schooling: International perspectives (pp. 90-102). London: Falmer Press.
Robinson, V. M. J. (2011). Student-centered leadership. San Francisco: Jossey Bass. Robinson, V. M. J., Hohepa, M., & Lloyd, C. (2009). School leadership and student outcomes: Identifying what works and why. Wellington: Ministry of Education.