Frameworks Simplified
How To Think About Impact Frameworks in Culture
Welcome to Measuring the Immeasurable, a newsletter helping cultural leaders understand the culture of evaluation. This piece explores the proliferation of explanatory frameworks in the cultural sector, and suggests a more modest approach to cultural evaluation, inspired by the Aboriginal fish traps at Brewarrina.
As always: if you also find it useful, please consider sharing and subscribing.
You could go big: The Public Value Measurement Framework (PVMF). The Cultural Development Network’s (CDN) Outcomes Schema. The UN Sustainable Development Goals (UNSDGs). Social Value International’s (SVI) Eight Principles. Measuring What Matters (MWM). Canada’s Arts Council’s Qualitative Impact Framework (QIF).
You could go niche: Brown and Novak’s Intrinsic Impacts of Live Performance Model. Brydie-Leigh Bartleet’s Community Music Framework. Lotterywest’s Community Investment Framework. Or, a favourite of mine: the CreaTures Framework.
Frameworks for assessing the impact of art and culture are ubiquitous. As a result, occasionally, someone suggests the horrid idea of building a ‘metaframework,’ a framework that encompasses every other framework. The Valuing the Arts1 report hinted at it recently for Creative Australia (I won’t accuse the authors of suggesting it). The result of ‘frameworks to end frameworks’ is pretty predictable:
The complexity of cultural change has led to a proliferation of frameworks, which in turn create a funding and reporting apparatus of impenetrable complexity. This is a vicious cycle similar to what I described in Bad Faith Evaluation several weeks ago, and it leads to both onerous reporting requirements (from funding agencies) and ritual gameplaying (from cultural organisations). The belief that any framework can fully capture the impact of a cultural intervention derives from a misunderstanding about what a framework is and what a framework is capable of doing.
I find it helpful to think of a framework as being like a net in a river. The purpose of a net is to catch some fish, but not others. A net which is too loose fails to catch all but the largest and dullest fish. A net which is too tight catches small fish, rocks, and other kinds of detritus. In the cultural sector, our general problem is that our nets are too tight: we’re catching too much and spend more time sorting through data than making effective use of it.
Funders (and, sometimes, arts advocates) think we need tighter nets, but we actually need better ones. I find it helpful to think of the Brewarrina Aboriginal Fish Traps2. The fish traps, like our contemporary technologies, are carefully constructed frameworks, but they are constructed in relation to something living and flowing. The principal purpose of the trap is to not disrupt the ecology it sits within. Its secondary purpose is to collect the minimum necessary fish to sustain the Ngemba peoples who share Country around the Brewarrina river.
As a lesson, we could say the good framework is one which doesn’t disrupt the flow of making and creating art and culture. A good framework is minimally instrusive. It is defined as much by what it chooses to exclude as what it chooses to include. A net without holes is, after all, not a net, but something completely different: a dam.
For cultural organisations and funders, this means ensuring that the reporting, acquittal, and impact assessment frameworks we develop are not onerous, address the specific needs of the organisation or project in question, and are as considered about what they exclude as they are about what they include.
Valuing the Arts, the Australia Council for the Arts (now Creative Australia).
Brewarrina Aboriginal Fish Traps, Department of Climate Change, Energy, the Environment and Water website.