This project trialled the use of a clinical phone triage for low acuity 111 calls in two pilot areas. The ambulance services adopted different approaches to secondary triage, with Wellington Free Ambulance calls referred to Healthline nurses and St John calls referred to an advanced paramedic in the St John call centre. The pilots were part of a larger project aimed at people accessing the care most appropriate to their needs quickly, and ambulance resources being used more effectively.

The purpose of the evaluation was to identify what worked well and less well in each of the pilots, the features of the two pilot approaches that worked best, whether the pilot outcomes were different from the status quo, and the implications of these findings for scaleability.

The evaluation used mixed methods, including interviews, a phone survey of 111 callers, a clinical safety review of calls, and analysis of operational data. Evaluative criteria were developed to enable clear and transparent judgements about the findings and suggested improvements, process maps were used to illuminate the different elements of each approach, and a value for money discussion focussed on the efficiency implications of the pilots.

The evaluation included team members Julian King and Gary Strong.

The RMRRS was established to help address capacity issues in the rural midwifery workforce, which were impacting on access to midwifery services for rural women and their families.

The purpose of the formative evaluation was to identify what was working well and less well after 14 months of operation and any potential improvements, and to provide indicative evidence about the extent to which the RMRRS was meeting its intended objectives.

The evaluation used mixed methods, drawing on the success case method approach, and involved the development of a programme theory, evaluative criteria (rubrics), an on-line survey, in-depth interviews with the provider and rural midwives and other health and community practitioners in three sites, and a value-for-money analysis.

The evaluation included team members Cheryl Benn, Julian King and Kataraina Pipi.

The Learn to Fall (LTF) programme was one of five ACC-funded initiatives aimed at reducing the number or severity of falls of working-aged people in and around the home. LTF was a new form of exercise programme, designed to provide participants with strength, coordination and training to reduce injury through falls. Integral to LTF are components that focus on improving balance, so a secondary focus of the programme is falls prevention.

The evaluation explored ‘proof of concept’ (whether people could be taught to fall safely), the extent to which participants improved on a range of physical and psychological factors and increased their awareness and cognitive understanding about fall recovery, and whether the programme could be scaled up.

The mixed methods approach included the development of a programme logic and evaluative criteria, document analysis of course marketing and training manuals, analysis of pre- and post- physical tests, a participant survey, participant focus groups, interviews with trainers and key staff in the provider organization, phone interviews with course drop-offs, a 3 month post-course survey to explore retention.

The SafeHouse game was an ACC-funded initiative targeted at 20-40 year olds, and aimed at reducing falls in and around the home. Located within the serious game genre, the SafeHouse game challenges users to resolve a range of home safety hazards by moving various family members to undertake specific actions. At the same time, players have to defeat an invasion of zombies, often by placing hazards in their way. The game has five levels, with different hazards introduced at the different levels.

The evaluation explored the viability and the effectiveness of the online game as a communication channel to influence adults’ home safety knowledge and behaviours, and also the potential of serious games as a vehicle for other injury prevention initiatives.

Methods used for this evaluation included an online survey of game users two weeks after the game to test retention, analysis of Google Analytics data relating to the game, and post-implementation interviews with game designers, ACC marketing and communications staff, and the programme manager.

The evaluation included team member Bob Williams.

With funding support from ACC, SLSNZ developed its coastal public safety risk management initiative to identify, analyse and mediate risk in the coastal environment to help prevent drowning and water-based injuries.

The purpose of the evaluation was to explore how well the initiative had been implemented in two pilot sites and how effective the identified risk mitigation interventions had been, and to recommend improvements.

The evaluation used mixed methods, including the development of a logic model, evaluative criteria (rubrics), pre– and post-intervention site visits, in-depth interviews, and also drew on a survey of beach-users undertaken by SLSNZ.

Evaluation Works has provided capability mentoring services tailored to the needs of each the listed organisations. This has included:

• bringing a strategic lens to discussions about the best process for organisations undertaking a capability assessment and developing a plan
• facilitating organisational capability assessment workshops with management, staff and Board members
• facilitating ‘prioritising and planning’ workshops
• supporting the development of capability development plans through guidance and the provision of planning templates (aligned with subsequent funding application requirements)
• undertaking specific capability development actions such as designing and implementing an outcomes measurement system.

Robyn Bailey and Kataraina Pipi supported Barnardos to evaluate whether the organisation’s investment in Ngā Pou E Whā and its key implementation strategy — a series of three training ‘kete’ — had made a difference. Part of the process was to understand what ‘making a difference’ looked like.

A range of creative methods were used in workshops with management and staff, to elicit Barnardos’ story, rationale and theory informing their strategy, the key evaluation questions and their criteria for assessing success.

An evaluative process was designed which ‘cascaded’ through the organisation. All managers and staff were invited to participate in an online survey. The results of the survey were used as the basis for a reflective session in each work area, exploring where they ‘were at’ in relation to the strategy, what had worked well and where to next. The senior management team firstly undertook a facilitated evaluation session then ran their own sessions with their management teams, and so on.

The evaluative process was designed to address the dual objectives of ‘localising’ the discussion and informing national developments, in a manner which provided the ability both for frank assessment (via an anonymous survey) and group engagement (Barnardos preferred style), in a budget-constrained environment.

In 2004 the Pay and Employment Equity Taskforce had identified that a significant pay gap, and a range of gender inequity issues, continued to persist in the labour market between men and women. Massey was the first New Zealand university to embark on a comprehensive review of gender equity issues.

Rae Torrie was project manager and gender analyst for Massey’s pay and employment equity review. She worked with a large committee of academic and general staff, and facilitated their learning about gender analysis, and the review process and available processes and tools to support the examination of these issues, as developed in the Department of Labour.

The primary methods used in the review were an online survey to over 4,000 staff and analysis of human resource data. Analysis and interpretation of this data by gender was summarized in a series of papers for consideration by the committee. Rae drafted the early versions of both the review report and the action plan.

The report and action plan are available here: http://teu.ac.nz/wp-content/uploads/2009/02/Massey-PaEE-Review-Report-Final-Report.pdf

Robyn provided ‘writing services’ supporting the development of evaluator competencies for Aotearoa New Zealand. This involved preparing a précis of the evaluation competency literature, identifying key issues to inform discussion by the reference group, and preparing the draft set of competencies for feedback.

The project was undertaken in two phases — a developmental phase (refer the following project) and an impact evaluation. The aim of the impact evaluation was to evaluate the worth (value) of the inter-disciplinary, integrated Youth One Stop Shop approach, as developed and applied by KYS. The collection of data occurred over July-December 2012, and the analysis and reporting occurred over January-December 2013.

The impact evaluation utilised a mix of qualitative and quantitative research methods situated within an overall evaluation-specific methodology. The evaluation applied the outcomes model and measures (refer following project) to track changes for young people using KYS services over a three-five month period. This involved a pre- and post-intervention comparison of the young people’s health and wellbeing between July to December 2012 (with the ‘intervention’ being the ‘use of the KYS services’). In order to explore changes over the longer-term, a retrospective measure was also taken for the young people in the evaluation population who were pre-existing clients of KYS.

Semi-structured, individual face-to-face interviews were carried out with three groups of people: 25 young people, the KYS key worker for each of the 25 young people, and nine parents or significant other adults. The 25 young people who were interviewed were those with challenges in their lives, ranging from those with some challenges to those seriously at risk. These young people were invited to participate in the evaluation because they are arguably, where most positive change is required, and because KYS was considered to be able to make the most significant contribution with these young people. The purpose of this phase was to gather information to understand the factors that may be contributing to any changes for young people, and the contribution of KYS to these changes.

Download a copy of the findings in the Impact evaluation 2013 — Summary report.

The project was:

• funded by the Health Research Council (HRC) Research Partnerships in NZ Health Delivery (RPNZHD) 2011 programme, KYS and Evaluation Works Ltd
• supported by a project team of academics from Auckland and Massey universities, a youth specialist, a youth advisory group and a Māori advisory group, and additional evaluation expertise
• undertaken in two phases — a developmental phase and an impact evaluation.

While anecdotal evidence was strong that the KYS approach (way of operating) was making a significant difference, KYS sought to develop a robust way of measuring such changes and how KYS contributed. As there was no agreed set of national youth health and wellbeing outcome measures, the project needed to develop a way of measuring the changes for young people, before it could assess the impact of KYS’s approach on the young people using its services.

The developmental phase involved researching and developing a youth health and wellbeing outcomes model and measures, and a ‘theory about how KYS works’ to enable positive health and wellbeing outcomes for the young people who use their services.

Both the outcomes model and measures and the theoretical concepts were tested as part of the impact evaluation (refer to the above project). Over 300 young people were tracked over time, along with follow-up interviews with young people, their key worker at KYS and group interviews with a parent or other significant adult.

Download a copy of Report One: Introduction, background and methodology.