Bad Faith Evaluation
The Political Economy of Accountability in the Arts
Welcome to Measuring the Immeasurable, a newsletter helping arts and culture leaders understand the relationship between arts and public value. Firstly, I note that Treasury has released their Measuring What Matters wellbeing framework. I made a submission to that policy here, which I’m happy to see was briefly acknowledged in the framework. I’ll be diving in to that over the next few weeks, mostly on LinkedIn. Follow me there if you’re keen to hear my thoughts on what the framework means for our sector.
Now: Today’s post is a longer, foundation piece looking at the political economy of evaluation, and how assumptions of bad faith drive increasingly onerous grant and evaluation requirements across the sector. I propose what good faith evaluation might look like and how to get there.
As always: if you also find it useful, please consider sharing and subscribing.
Art and evaluation are like two worlds separated by a vast ocean. In this newsletter, I act as an emissary from the evaluation world, trying to establish a trade pact with the arts world. My argument to arts and culture leaders is that evaluation is just data-driven storytelling, and so can serve as a Rosetta Stone which allows the language of the arts to be understood by philanthropists, politicians and civil servants.
However, readers should know that I’m not just in the employ of the evaluators. What I learn from the arts I bring back into the world of evaluation1. The arts helpfully illustrate how we think about evaluation. Art presents a kind of limit case, a situation in which the machinery of evaluation starts to break down, and in so doing the assumptions behind evaluation become apparent and, therefore, available for questioning. Arts evaluation is like doing a wheelie on a bicycle: it isn’t something the bicycle is supposed to be able to do, yet the bicycle still handles it adequately (if with a fair bit of wobble). The delicate manoeuvring required to keep the bicycle vertical teaches us something about how it is built and what it is designed to do.
This post is about the way the evaluation bicycle wobbles under the pressure of evaluating the arts. I diagnose two problems here, both of which I trace back to what I call Bad Faith Evaluation. Bad Faith Evaluation is what happens when evaluation becomes a tool of enacting and evading discipline rather than reaching mutual understanding. These problems are particularly prevalent in the arts and cultural sector, where the relationship between funders and recipients has long been somewhat antagonistic.
Problem 1: Evaluation tends to be gamed
It may be simpler to ground this observation outside the arts. Consider NAPLAN (National Assessment Program – Literacy and Numeracy) which provides a series of standardised tests to school students across Australia. The results of these tests are then directly linked to school funding and are considered an unofficial measure of school performance. This assessment, rather than producing better schools, instead produces the ‘teaching to the test’ phenomenon. As a number of Australian researchers put it:
It is widely accepted in the literature that high-stakes testing and practices such as ‘teaching to the test’ do unintentionally affect curriculum, assessment and pedagogical practices.
In Victoria, the Victorian Education Department went so far as to tell schools to “explicitly teach for NAPLAN'' and to give students a ''daily NAPLAN item'' in class2.
Other examples include the gaming of environmental policies and protections. On carbon credits, one whistleblower noted that there was an “inability to operate schemes like this with high integrity,” and that the market had “degenerated to become a rort." He goes on:
Payments are being made to people to not chop down forests that were never going to be chopped down, to grow forests that are already there, [and] to grow forests in places that will never sustain permanent forests.
A simple, useful heuristic in evaluation design is that anything explicit tends to be gamed. NAPLAN, environmental approvals, grant submissions and acquittals, and program evaluations are all examples of this. In the arts, a benign example of gaming the system is the significant expenditure of the sector on consultant grant writers and grant writing workshops. Indeed, one report found that the sector’s main capacity development demand is grant writing, rather than anything around operational efficiency or artistic production3.
Problem 2: Accountability is considered the solution to gaming the system
Evaluation is often framed in the language of accountability, which shares obvious etymological roots with accounting. Financial accountants are concerned with accountability: every single dollar must be traced and accounted for. What is on my books must precisely match what is in your books. The accountant’s world is a world of perfect transparency in which fraud is not so much immoral as impossible.
Groups like Social Value International propose evaluation as a way to build an accounting-like system for ‘non-financial forms of value,’ or, more simply, social and environmental value. Although they strive for a world beyond the narrow limitations of monetary finance, they borrow its metaphors and structures. They seek to ‘optimise’ for social value, build ‘societally agreed’ frameworks and standards, and help organisations maximise their ‘social return on investment.’
SVI’s focus on accountability is a response to Problem #1: that actors tend to game the system to their advantage. However, it is not as if SVI and the UN are watching the small-to-medium Australian arts sector. Their observations derive primarily from big finance: institutional investors, impact investing, development finance, and government procurement, where the stakes for bad faith are considerably higher. However, the logic of accountability applied to institutional finance has trickle-down effects. As John Maynard Keynes said, many of us are “the slaves of some defunct economist.”
Problem 3: A vicious cycle
John Elkington, who developed the original idea of triple bottom line reporting (an early iteration of social impact accounting) 25 years ago, had hoped to “provoke deeper thinking about capitalism and its future.” However, he instead found that the additional accounting data was “[not] being aggregated and analyzed in ways that genuinely help.” He recently issued a recall of his original proposal. It’s also worth pointing that even within the world of financial accounting, the strict standards therein did not prevent the likes of Enrol, Freddie Mac, AIG, and the architects of the Global Financial Crisis from creatively re-engineering their accounts to suit their own ends.
This is to say that stricter standards of accountability are often ineffective. Sometimes a better trap just produces a smarter animal.
So, what about the arts?
I shared in an earlier post that the new onus on arts organisations to conduct post-show surveys led one arts organisation to engineer the post-show experience to produce higher survey scores. After a performance, my friends and I were invited to:
sit and reflect on the show. We were given a platter of bread, dukkah and olive oil while we discussed... After we had been discussing for a short while, we were approached by a volunteer with a survey. The dukkah was there to woo us.
The arts are perhaps the most undervalued and overmeasured sector of civil society, and so gaming the system is understandable. Kate Larsen, in a piece for the Overland Journal4, found that organisations were spending “between thirty and fifty hours on each expression of interest [in multi-year funding].” However, assessors have comparatively little time to assess each application, such that the end result is that each assessor will spend “just one or two minutes of consideration for each hour of [the organisation’s] work [on their expression of interest].”
This is a perfect storm for gaming the system: funders are seeking more information than they can credibly audit from organisations who are desperate for funding. This drives sector organisations to spend even more time crafting word perfect proposals or spending their limited discretionary funding on consultant grantwriters. This is, of course, just the tip of the iceberg. I’ll leave the reader to imagine what other kinds of behaviour such a perverse system might incentivise.
The end result of onerous grants, acquittals and evaluations (from agencies and funders) and gaming of the system (from organisations) is a vicious cycle of mistrust and bad faith.
Breaking the Cycle
I believe that evaluation, when practiced in good faith, provides an opportunity for parties to seek to understand themselves and each other more truthfully. This is more than just professional courtesy: it is an essential democratic virtue.
With the launch of Creative Australia and the renewed hopes of the sector in having a clear, valued place in civic life, we have a rare opportunity to create new channels of communication in good faith. These channels would enable arts organisations to speak truthfully about what they do and to permit funders to make sensible, reasoned judgments about what to fund in alignment with civic and social priorities.
The cycle must be broken in both places at once. Arts organisations should engage in evaluation in earnest. This means evaluating on their own terms in a language that makes sense to them and their stakeholders.
Evaluators, whether government or philanthropy, must seek a kind of evaluation that shifts from a logic of accountability (which assumes bad faith), to one of honest communication (which assumes good faith). From my perspective, this also includes the professional evaluation industry, which has often supported a vision of the sector aligned with the logic of accounting. As every artist knows, there are some things which are beyond measure, logic, and rigour. This is the gift artists have to give to the world of evaluation.
I sit on the board of SIMNA, the main professional network for social impact assessment in Australia. SIMNA is a member of the broader Social Value International network which drives measurement practice and standards globally, increasingly in partnership with the United Nations.
Teach for tests, teachers told, Sydney Morning Herald.
Audit of Capacity Building Needs and Initiatives Across the WA Arts and Culture Sector, Chamber of Arts and Culture WA.
Tears for peers: the hidden costs of arts funding, Kate Larsen, Overland Journal.