Welcome to Measuring the Immeasurable, a newsletter helping arts and culture leaders understand the relationship between arts and public value. Today’s newsletter describes three basic attitudes towards evaluation I see in my consulting work with arts organisations - refusal, acceptance, and negotiation. I find it a helpful rubric.
If you also find it useful, please consider sharing and subscribing.
In my work I’ve observed there are three general orientations that arts leaders take towards evaluation: Refusal, Acceptance, and Negotiation. I’ve found this a helpful diagnostic tool, and I encourage you to see if any of these profiles resonates with you.
Refusal
In the arts and cultural sector, a kind of refusal of evaluation is common. This perspective frames evaluation as yet another burden imposed by funding agencies and philanthropists. Evaluation is an imposition: unwanted and unnecessary. In this frame, the arts only have intrinsic impacts. The word intrinsic does a lot of work here. It is used to mean both ineffable (unable to be communicated) and inherent (impact just happens, so why assess it?).
This refusal is always in principle, not practice. Organisations will comply with what is demanded by funders. However, evaluation becomes performative, and organisations only do the bare minimum required. Although I don’t agree with this attitude, I am sympathetic towards it. As I wrote about in an earlier post on the National Cultural Policy1, antagonistic relationships between funders and arts organisations has turned evaluation into something of a zero-sum game. Intrinsic impacts are also a real feature of many models of impact evaluation in the arts2, and yet tend to be broadly misunderstood and not well captured within existing frameworks.
However, arts organisations which refuse evaluation entirely often find themselves relying entirely on their history and brand to justify their worth. My observation is that this strategy works until it suddenly doesn’t. The recent funding cuts levelled at the BBC Singers3 are a recent case. Arts organisations which cannot connect their work to public value are often left relying on an outraged support base to push back against funding cuts.
Acceptance
This is a rare attitude in the arts and cultural sector. Organisations which adopt this perspective assume that evaluation brings value. They respond enthusiastically to opportunities to evaluate their work and integrate it with their business operations.
I only see this in social service organisations that are using arts projects for other purposes: like a wellbeing project in the prison system or arts therapy for people with trauma. These projects are evaluated because art is assumed to have instrumental rather than intrinsic impact.
The risk for these projects is that in putting art in service of something else, we often miss the unusual ways that art can work its magic on us. The intrinsic impact of the work is often missed. I wrote earlier about how the impacts of art take time4, and there is a need to keep a soft focus on longer-term consequences and other ripple effects. Evaluation can account for this, but those that accept evaluation on simplistic terms will miss these deeper impacts. Evaluation consultants who are unfamiliar with the arts are also likely to miss these impacts.
Negotiation
I believe that the proper attitude towards evaluation is always a negotiated one.
The negotiated response acknowledges that evaluation is always political. Like those who refuse to seriously engage with evaluation, the negotiators do not naively accept the terms of evaluation given to them by funders and consultants. Unlike those who refuse, the negotiators prefer to set their own terms.
This is particularly important in the arts, where funding is highly politicised. Arts organisations are not funded purely based on their social and civic impact. Government picks winners. Philanthropists have their preferred art forms. Ideas like ‘excellence,’ and ‘quality,’ complicate things: they aren’t interchangeable with ‘impact.’ The attitude that a symphony orchestra should take towards evaluation will be different to the attitude taken by a First Nations theatre company.
The negotiated response accepts the political reality of evaluation and chooses to be proactive within it. It is considered. It acknowledges that evaluation cannot tell the whole story, but they can tell a useful story. It acknowledges that there are gaps in its own knowledge and discerns the right questions to ask that will address those gaps. The negotiated approach is a martial art: It pushes, holds, grasps, but also yields, ducks, and weaves. Or, as Bruce Lee put it in his Wisdom for the Way:
“Absorb what is useful, discard what is useless and add what is specifically your own”
Canada’s Qualitative Impact Framework is a good example. The CATA Public Value Framework and Culture Counts also use the term (example). I’ll unpack what we mean by ‘intrisic’ impact in a future post, as it’s a very slippery term.
See this open letter. The cuts were subsequently reversed.