We all know of the importance of measurement and evaluation but in this time and resource-poor era are you focusing too much on the easily countable at the expense of the true picture? Panel writer Sarah Lay looks at the danger of McNamaras fallacy in the evaluation of communications.

by PANEL WRITER Sarah Lay

There is a saying that I am fond on when looking at measurement and evaluation of digital and communications: ‘A number has no meaning other than that which you place upon it. Its a reminder that while qualitative data may well be easy to collect, it rarely tells the true or full picture. While the headline figures may be made attractive by time pressures, without that full story how well informed can your future efforts be?

The McNamara Fallacy

Relying too heavily on that which can be easily gathered and stacked up has become known as the quantitative fallacy, or the McNamara Fallacy.

Robert McNamara was the secretary of defence during the Vietnam War and developed an approach to measurement based on factory production lines. It relied on assumed predictable and repeatable behaviour, as well as advising to discount anything which wasnt easily counted.

In a move that saw body count being used as a measure of success it was blind-sided by the messy chaos of human behaviour – a realisation he was reportedly haunted by in his later life.

“The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which cant be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what cant be measured easily really isnt important. This is blindness. The fourth step is to say that what cant be easily measured really doesnt exist. This is suicide.” (Yankelovitch 1972)

High numbers dont mean success

Those easy numbers are the ‘new website draws double the daily visitsheadline. But if you only count visits and look no further you are unlikely to understand the real issues and opportunities, the successes and failures of your current offer.

So – your web page is bringing in a good amount of visitors, perhaps youve even been able to show this has corresponded to a sought-after uptake of a service, or a reduction in contact through other channels. But have you looked into whether that page is meeting the users need? How sustainable is the change if your content fails to satisfy.

You need deeper stats here about bounce rates, time spent on page, completed actions, user comments. If people were following a search result were the keywords they entered relevant to your page – if they cant find the answer does a larger number of them visiting mean anything (other than maybe a large number of frustrated people)?

Perhaps you persuaded someone to sign up to your mailing list but they left within a week, or they never open the mail you send to them – in which case, was it ever really success? Or the video you posted on Facebook got 5000 views, a handful of Likes and a couple of shares. But how many of those engagements resulted in an action?

Maybe your content does ok and achieves what you want it to most of the time, but by looking at the detail could you make an improvement that takes it from functionally meeting a user need to satisfying extra desires as well, from that passive engagement to action.

With shorter time and greater pressure to prove the value of your efforts its tempting to take the number which looks good and pin your value to it. This might get your short-term praise internally but may mean youre missing the chance to really understand how youre meeting a users need, and how you can do even better next time.

Sarah Lay is a digital content strategist, consultant and comms2point0 Lifetime Achievement UnAward Winner, with experience of delivering across the public and private sector. You can find out how to work with her, and read more from her, on her website www.sarahlay.com or find her on Twitter @sarahlay.

image via Thels Kofoed Hjorth

Original source – comms2point0 free online resource for creative comms people – comms2point0

Comments closed

Bitnami