Improving outcomes for individuals and communities in Australia is complex. Disadvantage is multifaceted, with individuals facing compounding inequalities that can lead to cycles of disadvantage which are difficult to overcome. Similarly, the systems which aim to address disadvantage are complex, with varied governance, funding and structural arrangements.
The Paul Ramsay Foundation partners with a variety of organisations which can be classified as ‘intermediaries’, and which aim to achieve social change within complex systems. Intermediary organisations aim to break cycles of disadvantage by supporting an ecosystem of actors who also share this goal and often a common focus on a particular ‘offramp’ from disadvantage.
Intermediaries provide value by working in complex systems, doing one or more of the following:
- supporting other changemakers through capability building and evidence generation;
- coordinating efforts to foster community participation and to join-up services and other initiatives;
- influencing policy and systems, including advocating to government.
As such, intermediaries can boost the impact of many actors, increasing overall effectiveness and improving outcomes for individuals and communities most at risk of cycles of disadvantage.
Given their important role, there is considerable value in understanding and measuring the outcomes of intermediaries. Such an understanding can allow intermediaries to better innovate and improve, and can provide funders with evidence on the relative impact of intermediary partners, and on which ecosystem supports are most needed to address a given challenge.
To progress thinking on these issues, the Paul Ramsay Foundation commissioned Mark Cabaj to explore how intermediary organisations could evaluate their activities and results. We are pleased to publish his report publicly in the interests of sharing knowledge to strengthen the for-purpose sector.
The report draws on interviews with 12 of the Foundation’s partners who can be classified as one of four intermediary types:
- Capability Specialists
- Evidence Action Labs
- Place-Based Backbones
- Field Catalysts.
The report is also informed by Cabaj’s experience leading and evaluating intermediaries in Canada and elsewhere.
The report identifies the challenges intermediaries face in measuring their results. Intermediaries operate in complex systems, aim to address wicked problems, and are one of many actors working towards a particular social change. In this context, a full understanding of the causal mechanisms which lead to change, and any one organisation’s role in those mechanisms is not possible.
Cabaj argues that to understand their results and outcomes, intermediaries should consider the bespoke nature of their theories of change and avoid using one-size-fits all indicators or measurement methods. This supports conceptualising intermediary impact in aggregate rather than single specific projects, and evaluating results according to initial theories of degree of change (incremental to transformative). Methodologically, this requires mix-methods for measurement, and a focus on contribution to change, rather than sole attribution.
The report alludes to the different evaluation objectives that arise given different evaluation audiences. For intermediaries themselves, continual learning is vital for organisational improvement. While this should be of interest to funders and other stakeholders, these audiences also need to consider a wider set of questions to inform their decision-making. Cabaj proposes funders and other stakeholders assess portfolios of activities and results produced by intermediaries and other associated organisations, which are working towards a set of system changes. This can support funders’ attempts to assemble a set of mutually reinforcing partnerships and activities to address the complexity of disadvantage.
