Think Data, Not Evaluation
Why Evaluation Is Easy, Simple, and Cheap (or Free).
Welcome to Measuring the Immeasurable, a newsletter helping cultural leaders understand the culture of evaluation. This month I am on a quest to understand how arts organisations are collecting and storing data about customers, clients, donors and artists. You might think of this as the unsolved ‘CRM’ problem in the arts sector.
If you’re in the arts and culture sector and would like to catch up for coffee (real or virtual) on this topic please message me on LinkedIn or reply to this post.
Today’s post is long overdue: an attempt to show how evaluation can be easy, simple, and free. As always, if you enjoy reading it, please share and subscribe.
When I was growing up, I couldn’t stand the taste or texture of eggs. I was like the man Sam-I-Am spoke to in Green Eggs and Ham: Whichever way you sliced it (literally or figuratively), I could not, and would not, stand the taste of eggs.
That was, until, one day my grandmother offered to make Toad in the Hole. I didn’t even ask questions. The name reminded me of Toad from the Super Mario games and so I said ‘yes!’ without hesitation. With probably too much tomato sauce and salt (this same grandmother was never told you’re supposed to dilute cordial), I discovered I could not only stomach, but enjoy, the taste of eggs. Eggs on toast became a breakfast favourite for many years to come. The secret to enjoying eggs was to enjoy it smuggled inside something else, hiding under a psuedonym.
Evaluation, I think, is the art’s sector’s runny egg. Evaluation is seen as expensive, specialised, and complicated. We do acknowledge evaluation is good for us, most of the time, and a CaCWA sector report identified it as a priority for future sector development efforts1. We know that the road to relevance involves some element of proof, and that proof can only come from evaluation. Despite this, evaluation is consistently a middling priority for overwhelmed, overworked and underfinanced arts organisations.
Of course, evaluation isn’t necessarily expensive, specialised, or complicated2. A small investment in time and energy in evaluation can produce enormous gains in understanding and relevance for arts organisations and funders. We become overwhelmed by the scale of what we imagine evaluation is asking of us. It’s like refusing to go for a walk because you’re unwilling to run a marathon, even though a thirty minute walk delivers most of the health benefits.
I’ve found it helpful lately, borrowing from my grandmother, to start talking about data collection rather than evaluation. Just as she smuggled an egg into my morning toast, ‘data collection’ is something routine and ordinary (the toast) that every art organisation does, and yet it contains the thing we dare not eat (evaluation).
As a practical example, a client once asked for help collecting unsolicited testimonials for their outcomes reporting. They assumed this process would be expensive, specialised, and complicated: that they would need a qualitative analysis tool like NVivo or a PhD social scientist to make sense of the data. I asked them where most of these testimonials came from and they said usually email or over a conversation. In the end, our ‘data collection’ plan was this:
When you receive an email with an unsolicited testimonial, tag it in Gmail as ‘testimonial’ using Gmail’s tagging feature.
If you get a testimonial any other way, ask them if they can send it to you as an email.
Evaluation tastes good when it’s familiar. Positive feedback shared through our existing channels, like audience attendance or revenue diversity, is a form of data collection we don’t ordinarily think of as “evaluation.” If this seems a bit too easy, consider that systematically collecting anecdotal feedback was, according to one funder, a sufficient level of rigour to apply for grants up to $50,000.
Evaluation, properly understood, is more of a mindset than a method. It’s not a hyper-specialised set of tools that only an expensive consultant can apply3. Rather, evaluation is a commitment to methodically collect qualitative and quantitative data on how the work you’re doing is changing people’s lives.
If you’re doing that, you’re doing evaluation.
Audit of Capacity Building Needs and Initiatives Across the WA Arts and Culture Sector, from the Chamber of Arts and Culture WA.
To be quite explicit here, I highly encourage any arts organisation reading this to download the CSI’s Outcomes Roadmap or check out Lotterywest’s Community Impact Hub. Both resources are free and have enough detail to support implementation: There’s no need for any consultants or software.
Of course, there are evaluations that are like this, such as this recent SROI on the value of the Sydney Opera House. However, evaluations of this scale are not necesary or the best place to start. Start with a Toad in the Hole before worrying about a soufflé.