Independent programme evaluations and reviews

Benefits of programme evaluation

When conducted correctly, an independent programme evaluation can deliver significant benefits to an organisation. Evaluation is critical to identifying, monitoring, and managing programme success and to improving performance and efficiency.

In both the public and private sectors, gaining an understanding of whether programmes and initiatives have produced the desired outcome is vital to decisions on whether the programme should continue, what might be changed if it does continue or whether the programme should be retired or replaced. In addition to this, as problems and opportunities arise within an organisation, such as decreased funding, unmet service user needs, staffing changes etc., decisions may need to be made which require an in-depth objective understanding of the programme.

Internal vs external review

Organisations will often hire independent external reviewers, such as Crowe, instead of conducting an internal programme evaluation for a number of reasons:

• Objectivity. A third-party evaluation is likely to be more objective as the evaluators have some distance from the work and do not have a vested interest in the outcome
• Influence. Hiring a specialist team of evaluators will add more weight to the findings than an internal team
• Expertise. An internal team may not have the specific skills or training required to conduct a robust evaluation
• Experience. Industry specialist evaluators bring with them a vast amount of experience to help guide them through the process smoothly and to better evaluate the findings
• Time availability. The evaluation could take up a considerable amount of organisational time, which may not be available within the current workforce.

Our approach

Crowe has a vast experience conducting both mandatory and voluntary programme evaluations for both public and private sector organisations. We provide our clients with objective and robust evaluation findings that enable them to make important decisions. Our experience comes from evaluating a wide range of programmes including: training and education; industry support programmes; network development; programmes of activity for membership organisations; cultural events calendars; tourism events and festivals.

We understand that asking the right questions before, during and after a programme’s operation will answer whether it had a REAL IMPACT – allowing your organisation to look towards the future, whilst being informed by the past.

• Relevance – was the programme central to both the organisation’s objectives and relevant to the programme audience?
• Efficiency – did it use resources appropriately e.g. staff, time etc
• Awareness – did the programme build sufficient awareness about its services?
• Lessons – what lessons can be learned regarding how the programme operated?
• Implementation – were the programme’s activities actioned as originally planned?
• Measurement – was the programme monitored throughout its operation?
• Performance – how did the programme perform in comparison to expectations?
• Attribution – can the programmes successes and failures be shown to be related to the programme itself, as opposed to influencing factors e.g. macro environment
• Cost-effectiveness – did the programme create a good ROI and did it represent “value for money”?
• Targets – did the programme set, monitor and meet its targets?

During a programme evaluation, a number of structured processes and methods can be utilised to answer the above questions, uncover the successes and failures of the programme and inform recommendations or options for its future strategic direction or its potential termination. Below we outline the key stages in a programme evaluation.

Key stages to a programme evaluation

How did the programme operate?
A combination of interviews, focus groups, workshops and surveys can be used to ensure engagement across a range of stakeholders. Added to this, a variety of qualitative and quantitative research methods will be employed to gather depth and breadth of insights.

It is important that both internal and external stakeholders are consulted during this phase, to gain an understanding whether the programme satisfied both internal and external stakeholders’ needs and expectations. This approach ensures that stakeholders have an opportunity to input into the programme evaluation and are able to raise key issues.

If required, a financial assessment will be conducted to provide further context and background. This will often include an analysis of resource inputs and outputs which can inform decisions on future resource demands and resource utilisation.

An external environment analysis will also be carried out to assess relevant market trends, whether the programme has evolved in response to these trends, and, if requested how it compares to other international programmes.

How effective has the programme been?
A key stage in programme evaluations is analysing whether the programme achieved what it set out to and how it measured up against its KPIs. Key themes will have emerged from the previous phase which will required a more detailed analysis. These findings mapped against the programme objectives, external environment review, financial resource assessment and resource analysis will determine whether the programme has been a success and provided value for money.

At this stage in the process, initial findings can be presented at a workshop which allows key stakeholders time for review and any relevant questions to be asked and feedback provided.

Evaluation report and recommendations
Once the feedback and questions from the last stage have been answered or addressed, a report will be produced outlining all the findings, conclusions and recommendations. Crowe has made a number of recommendations where programmes are adapted: changing the resource allocation, modifying the target audiences, adapting the model etc. Occasionally recommendations have been made to terminate the programme entirely.

The methodical process we follow in our programme evaluations ensures we identify the programme’s strengths and weaknesses and uncover the possible future opportunities and threats. The evaluation will help an organisation determine the future viability of the programme.

If you would like to learn more about implementing a programme evaluation, please talk to a member of our consulting team.