Welcome to Measuring the Immeasurable, a newsletter helping cultural leaders understand the culture of evaluation. A particular welcome to my UK-based readers: it is encouraging to see this rather specific blog reach international audiences.
I work with organisations to develop data and evaluation strategies which ‘zig the zag’ of mainstream arts reporting in a way that supports advocacy and insight with minimum administrative burden. If you’re keen to discuss how we can work together, please message me on LinkedIn or reply to this post.
Today I unpack a few developments in evaluation in the UK to better unpack governments’ pursuit of certainty and standardisation — a fool’s quest that costs the sector millions and teaches us very little. As always, if you enjoy reading it, please share and subscribe.
Mark Robinson, a former Executive Director at Arts Council England (ACE) and author of
, recently alerted me to a new research collaboration between two research groups in the United Kingdom. The six projects, which together cost an eye-watering £3.1 million (close to $6m AUD), aim to develop a ‘formal and multidisciplinary approach to valuing the benefits of culture and heritage assets to society.’This seems to be a recurring theme at the ACE. In 2016, they implemented Culture Counts in order to ‘standardise measuring artistic quality’ for their National Portfolio Organisations (NPO). Apparently, this standardisation was not good enough: ACE has now also engaged big-four consulting firm PWC on a project called ‘Illuminate,’ to much the same end: To standardise reporting on box office, ticketing, and post-show surveys for NPOs.
Arts Professional UK reported a number of issues with the platform including “errors and omissions on the survey template, questions not mapping against Census data, the administrative burden of collecting both ticketing and survey data, and increases to the required sample sizes for audience survey submissions as being among them.” The platform also does not integrate with existing box office and ticketing software, meaning NPO organisations are “bending over backwards in order to get this data,” in the words of one consultant. The platform has been such a failure that they have had to delay NPO reporting by an entire year to fix the platform.
Australian readers who have recently submitted their multi-year funding applications may be either nodding in agreement or shaking their heads in dismay.
The ACE seeks certainty and standardisation. This fruitless quest has led to plenty of good money chasing after bad. Trying to answer the question “What is the value of art and culture?” is like trying to answer the question “Where is the fruit fly on the windowpane?” The value of culture is never a question that can be answered once and for all, because culture buzzes around, and its value to us shifts with it. This isn’t some quirk of arts policy but a feature of art itself. The whole purpose of art is to provide a kind of shifting, kaleidoscopic kind of value. If value were a static phenomenon, we would never rewatch our favourite movie or revisit our favourite gallery.
The other irony here, of course, is that there has never been a stronger case for the value of the arts in public life. The evidence is in: art changes us. We have all the knowledge we need and yet we keep seeking more of it. The ACE seeks data like the hungry ghosts of Buddhist mythology, wailing and wandering in the realm of Saṃsāra.
Why isn’t the knowledge we have good enough?
A friend pointed me towards an article in The Lancet which sought to unpack why the practical integration of arts into mainstream healthcare settings is trailing behind a rather convincing evidence base. The whole article is worth a read, but the sentence that best explains the phenomenon, in my view, is that:
“establishing causality [in arts interventions]… is often not possible. This bar of absolute certainty for arts interventions is arbitrary.”
The Arts Council England, and to a lesser extent certain Australian jurisdictions, pursue certainty and standardisation to the detriment of the sector. In the arts, if you seek certainty you will be forever dissatisfied. This is as true for funders as it is for the novice gallery visitor who treats a work of art like a math problem on a chalkboard. “Yes, but what does it mean?” is the catch-cry of both the novice aesthete and the art bureaucrat.
If we believe that the value of arts and culture, or any subset thereof, is a problem of simply gathering more information, then we are only committing ourselves to endless wandering and searching about. I am reminded of the old joke about the drunkard’s search for his keys:
A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, "this is where the light is."
Illuminating indeed.
What we need from arts evaluation is not more information about its impact, but perhaps those very qualities art bestows on us, qualities like openness, curiosity, courage, and above all else, a kind of comfort within uncertainty — a willingness to look in the dark.