aes17

Sandra Collins (Education Review Office)

The Education Review Office (ERO) plays a significant role in system-level improvement through its external evaluations.  Through a range of system-level evaluations ERO seeks to enhance both policy and practice by strengthening both the evaluation capability and capacity of the system and those who work within it. As well as undertaking external evaluation of all New Zealand schools and early learning services, ERO also undertakes national system-level evaluations.  In 2004, 2009 and 2017 ERO evaluated the Resource Teachers: Learning and Behaviour (RTLB) service.  The RTLB service aims to improve learning and teaching for students with moderate learning or behaviour difficulties in schools. RTLB are a group of trained itinerant specialist teachers, working across clusters of schools, who provide support to ensure good educational outcomes for Years 0-10 students. RTLB services are managed by full-time Cluster Managers, situated in 40 lead schools.Both the 2004 and 2009 evaluations found variability in the way the RTLB clusters were governed and managed. ERO noted "a lack of strong external and internal accountabilities for the use of funding and management of RTLB remains an issue in a large proportion of clusters".  Self review (internal evaluation), planning and reporting were found to be the weakest across the 40 clusters evaluated.      In response to the evaluation findings, the Ministry of Education worked in collaboration with key sector groups to undertake a significant transformation of the RTLB service.  This transformation included reducing the number of clusters from 199 to 40 and appointing cluster managers with responsibility for cluster management in collaboration with lead school principals.The Ministry of Education asked ERO to undertake a subsequent evaluation in 2017 to focus on the extent to which the transformation of the RTLB service had addressed the findings of the 2004 and 2009 evaluations. This presentation will focus on the use of the findings from the 2009 RTLB evaluation, and the design and implementation of the 2017 RTLB evaluation which has a strong focus on building evaluation capability and capacity in the RTLB service.  These evaluations are contributing to building evaluation capital by generating specific actionable recommendations for policy-makers and practitioners, while also supporting RTLB cluster managers and lead school principals to strengthen their capacity to engage in their own internal evaluation for continuous improvement. Justification StatementBecause the credibility of evaluation findings matters, priority has been given to working with key stakeholders to design the evaluation framework and approach, and provide a forum to discuss the findings prior to publishing them.  The evaluations of the RTLB service have focused on use from the early scanning and scoping phase by engaging with primary users.  Discussions with Ministry of Education staff/officials with funding and accountability responsibilities helped to clarify evaluation purpose and approach.  An external reference group comprised of RTLB cluster managers, lead school principals and Ministry of Education staff/officials contributed to the design of and approach to the 2009 and 2017 evaluations.  A workshop with key stakeholders at a national forum at the early stage of designing the 2017 evaluation invited critique of the 2009 evaluation framework and provided an opportunity for these stakeholders to make suggestions about the approach to the 2017 evaluation.  The evaluation framework, approach and tools are also shared with key stakeholders and those involved in the evaluation.Progressively embedding evaluative reasoning has been an integral part of these evaluations.  A synthesis rubric developed as part of the 2009 and published in the 2009 report Resource Teachers: Learning and Behaviour - An Evaluation of Cluster Management has been adopted by many of the RTLB clusters.  The synthesis rubric was modified for the 2017 evaluation to align it with Ministry expectations and drawing on adaptions made in individual RTLB clusters.  External evaluation processes are building the internal evaluation capability and capacity of those working in the RTLB service at both the local cluster level and at a system level