Welcome to Measuring the Immeasurable, a newsletter helping arts and culture leaders understand the language of public value. This week I’d like to share a little bit of practical advice, tying together a few themes from earlier posts, and demonstrating an interesting framework in active use in Canada. If you find this work useful, please consider sharing and subscribing.
In an earlier post, Dukkah, Bread, and a Survey, I shared two stories about two evaluation surveys I received as part of two separate performances. In brief, the argument I made there was that even identical surveys cannot necessarily be compared if the artform, context, and methodology are different. The statements “I love icecream,” and “I love my partner,” use the same verb and yet mean completely different things: this is the problem of comparability in arts evaluation.
However, the piece touched on a particularly insidious difference that is often ignored in arts evaluation: when an evaluation activity takes place. Imagine you are evaluating a large, outdoor public art event like Waterlicht1. You could deliver a survey:
During an arts experience (with intercept interviews)
Immediately after an experience (as audiences are leaving the grounds)
Much later (say, an email survey six months later)2.
Naturally, our responses at each point are going to be quite different. The way we feel during a performance is very different to how we feel even five minutes after the performance finishes. We shift from reporting on what we are feeling to reporting on what we felt. Six months later, we’re now dealing with what we remember about what we felt and we may struggle to find words to describe our experience. We may need to flip through our calendar to remember exactly what it was we saw (“oh yes, that outdoor water thing right before COVID…”). An art experience is a little bit like a dealer used car: the moment you’re out of the parking lot, there is some kind of value lost. Six month later, our new car is just the car, and all the features that delighted us are now completely taken for granted. We may already be debating the virtues of buying a fancy new EV, like our neighbours. Our memory fades, and lived experience is slowly replaced with judgment.
This may all seem theoretical, but this basic division along three time periods informs the Canada Council for the Arts’ Qualitative Impact Framework3. So, for my possibly cynical Australian friends, this demonstrates that it is possible within an arts political economy to recognise the profound importance of time in considering art impacts. This separation between immediate, concurrent impacts (now), post-event reporting on experience (soon), and long-term impacts (later) is essential if we are to understand the impact of art on audiences and communities. In the arts marketing world, this distinction between the immediacy of an art experience and the need to consolidate it is recognised in the visitor engagement cycle. The point here is that this is a useful, practical impact framework already employed in a variety of arts contexts.
Of course, in Australia, it is the ‘later’ impacts that are most poorly understood in the arts. Long-term readers will recognise this is an argument I made in Art Impact Takes Time a few months ago. The Canada Council for the Arts is inclined to agree. As stated in their impact framework:
“But, in preparing this framework, we have come to believe that the larger challenge – and the greatest potential reward – lies in developing stronger theoretical frameworks and measurement approaches for understanding [long-term] impact... These frameworks would articulate how individuals construct a larger narrative from their arts experiences, how this narrative contributes to their quality of life.”
This way of thinking strikes me as something practical and immediately available to arts organisations and funding bodies. Thinking about art impact along different time horizons (now, soon, and later) is both theoretically sound and intuitively true. It therefore presents a way for arts organisations to assess and communicate what they do more effectively.
Waterlicht, which I mentioned in Dukkah, Bread, and a Survey.
An attentive reader may notice another common option: Within a day or two of the art experience. Generally, I think of this as similar to #2 above, but with lower quality data and poorer response rates.