Reflexive thematic analysis and the challenge of leadership evaluation

How we measure our impact for New Zealand learners.

At Springboard Trust, we work with more than 300 volunteers and principals every single year. Each of these individuals has their own unique experience, bringing their own expertise and background to courses like Springboard Coaching for Leadership and the Strategic Leadership for Principals Programme.  

Throughout their journey with Springboard, and for some time afterwards, we gather information on the impact that Springboard’s work has had through surveys, interviews and specially designed assessment rubrics. This gives us a significant breadth of qualitative and quantitative data about how our portfolio impacts principals, volunteers, organisations, schools, senior leaders and New Zealand students. 

Every year, we publish this data – along with substantial analysis of it - in our Impact Report. It includes our work on a new evaluation framework, reflexive thematic analysis, qualitative and quantitative findings that link the work we do with principals to positive outcomes for students. 

With the 2019 edition now available for you to read, it’s a good time to break down some of the ways we use this data to measure our impact.  

How we are measuring our impact: Reflexive thematic analysis 

In the past, we have presented impact data qualitatively, as stand-alone case studies or supporting evidence. By utilising reflexive thematic analysis and a dedicated statistician, we have been able to turn this wealth of information into statistically significant findings around our impacts on schools and learners.  

Thematic analysis (TA) is an overarching term for a set of practices in psychology, that have applications well beyond this field. In TA, researchers analyse qualitative data (like interviews, surveys or other expressive, open-ended responses) and identify statistically significant themes and outcomes. In short, it’s a more objective way of demonstrating results from data sets that can be highly subjective.  

Reflexive thematic analysis (RTA) is a subset of this, and was originally developed by the University of Auckland’s Virginia Braun and Victoria Clarke in 2006. Since then, it has become a hugely popular approach which has seen it being used as a methodological approach in hundreds of academic journals internationally. 

It is particularly well-suited to data sets that relate to people’s personal experiences or perceptions, and so forms a useful basis for analysing Springboard Trust’s impact.  

RTA consists of the following six key steps which are recursive - meaning that the researcher might move back and forth between these steps several times:  

1. Familiarising yourself with the data or information 

2. Giving each element of the data a name or label (coding) 

3. Developing the high-level themes or patterns within the data 

4. Reviewing these themes against the entire dataset 

5. Detailing and analysing each theme 

6. Writing up the findings 

We have adopted this approach, together with quantitative methods for the 2019 Portfolio Impact Report with the assistance of a dedicated Research and Evaluation professional, resulting in the most in-depth analysis of Springboard Trust’s impact to date. 

The challenge of measuring the impact of educational leadership 

Springboard Trust’s ultimate goal is to improve outcomes for New Zealand students, which we do by enhancing educational leadership. However, research on this area has indicated some complexity in linking leadership in schools directly to student outcomes.  

While it might seem reasonably straightforward to evaluate an SBT programme’s impact (just ask one of our principals), there has been ongoing debate about whether it is possible to determine the true and direct impact of principal development programmes. This debate is prevalent right across the development spectrum – not just in Springboard’s work.  

Specifically, while there has been consensus on the role of leadership in student achievement, there has been a general reluctance to confront the challenge of determining indicators of effectiveness, identifying what aspects to measure, how to measure them and how to interpret and respond to the results.  

This means the challenge for researchers – and Springboard Trust – is to go further in our evaluation than the bulk of educational research has gone before to navigate the complexity of tying school leadership, organisational function, teacher effectiveness and student learning together.

That means clarity in identifying the focus and outcomes, consideration of whether these outcomes can be achieved in the short-, medium- or long-term, the selection of relevant and varied data sources (e.g., multiple stakeholders, multiple methods) and the systematic collection of evidence over time. All of this must also be tied together in an agreed-upon evaluation framework and a commitment to gathering data from the short- to long-term.

Without that long-term commitment to evidence the impact of development programmes, we have - at best - a snapshot of delivery rather than evidence of impact over time.

The challenge of evaluation is not one we – or anyone working in this field – can solve overnight.

But we believe that with this Impact Report, we have laid the groundwork for some remarkable findings in how school leadership influences student outcomes.  

It forms part of our commitment to both improving student outcomes and continuously improving our portfolio, year on year, to better help principals and learners alike. We're thrilled with the results, hope you enjoy them too.