Pillar 2: Evidence and accountability

'It is not feasible, cost-effective or appropriate to fully evaluate all government activities and programs. The cost of evaluation must be balanced against the risk of not evaluating, noting that sometimes performance monitoring by itself will be sufficient to meet the performance reporting requirements under the Public Governance, Performance and Accountability Act 2013.'

Commonwealth Evaluation Toolkit (Australian Centre for Evaluation)

We prioritise evaluation effort and resources with a decision-making framework (see table). This enables us to do a strategic, risk-based assessment of appropriate evaluation activities. 

To determine what evaluation activity may be required, including its scope and timing, we need to consider:

• the characteristics of the policy or program
• other assurance activities that have been, or will be undertaken
• the context in which the evaluation will be undertaken, including related policy developments.

The Evaluation Unit will work with responsible areas to assess the evaluation category for each initiative and the evaluation activity required. This is done as the NPP or program proposal is being considered and will allow appropriate resourcing to be provided.

When this has been determined, the Evaluation Unit will track the activity and provide regular reporting. This ensures strategic oversight of evaluation activities across the department. Evaluation activities will be included in the responsible area’s business planning.

Framework for determining appropriate evaluation activity for policies and programs

Purpose-driven evaluation

To maximise usefulness, all formal evaluations in the department need a clear purpose. Evaluations are shaped by:

• the purpose and context of the initiative (including where it is in its lifecycle)
• the information needs of stakeholders
• ethical and cultural considerations. 

Broad focus areas may include:

• Design – Did we do the right thing?
• Implementation – Did we do things in the right way?
• Impact – Did it work?

These focus areas inform the types of questions to ask in an evaluation and the most appropriate methodologies. Evaluations that focus on an initiative's impact must be timed so the expected outcomes have had time to occur. 

The Evaluation Unit will provide internal resources and guidance for staff on planning, scoping and conducting formal evaluations. This includes determining evaluation questions, appropriate methodologies and other considerations such as ethics and privacy. The resources will also include guidance on program health checks.

Using external expertise

The department may not always have the expertise or capacity for evaluation activities.

When needed, we may engage external partners or providers (such as academics or research providers) to conduct or help with evaluation. These partnerships are a chance to draw on expertise and build our internal evaluation capability.

2.3 Evaluation governance

Principle 2.3: We integrate oversight and accountability into our evaluation practices to produce credible and robust evidence.

We all have a role to play in ensuring evaluation is part of our policy and program processes as we seek continuous improvement. Figure 4 below provides an overview of the evaluation roles and responsibilities in the department.

Department-level oversight

Appropriate governance helps ensure accountable and credible evaluations.

The Performance and Risk Committee (PaRC) is responsible for strategic oversight of evaluation. It also provides advice to the Executive Board on the current status and approach to evaluation in the department, including interactions with other assurance activities. Through these governance arrangements, PaRC is responsible for ensuring the department is undertaking monitoring and evaluation in line with this strategy and Australian Government requirements. The Evaluation Unit supports this oversight function.

PaRC endorses the annual schedule of evaluations. This includes:

• considering any requests to reschedule or cancel planned evaluations
• prioritising where in-house evaluation and external expertise is used across the schedule.

PaRC only considers requests to cancel planned evaluations in very limited circumstances. Once PaRC has endorsed the schedule, it will be presented to the Executive Board for approval.

The Evaluation Unit in the Analysis and Insights Division provides strategic management of evaluation activities in the department. This includes:

• monitoring and reporting on evaluation activities across the department, including implementation of evaluation recommendations
• managing the annual schedule of evaluations and forward program
• determining the appropriate evaluation activity and governance arrangements for individual policies and programs
• providing guidance and advice on evaluation design, frameworks, plans and methodologies 
• undertaking some high-priority evaluations, as agreed with PaRC 
• maintaining the register of evaluation recommendations and lessons learned and the library of completed evaluation reports
• liaising with the Australian Centre of Evaluation and DISR portfolio entities.

Activity-level oversight

As well as considering the form of evaluations, we must also consider the appropriate level of oversight for specific evaluation activities. This needs to account for context and manage issues such as the risk of bias and ethical or cultural considerations.

For formal evaluations, the Evaluation Unit will work with the responsible team to create a reference group of appropriate decision-makers. The reference group will oversee the evaluation from beginning to end. The level of the reference group is informed by the evaluation category of the policy or program being evaluated (see Figure 3).

For less formal evaluation activities, such as ongoing monitoring or program health checks, oversight through existing reporting channels may be sufficient, combined with the departmental oversight set out above.

Overview of evaluation roles and responsibilities throughout the policy lifecycle