Rubbish in…rubbish out….stuff that surveys are made of.

I took part in a recent survey for the Soil Association – despite the fact that it was clearly the most biased, marketing driven survey I’ve ever come across.

The questions were so leading as to make objectivity impossible, such as:

“Annual reports from the Pesticides Residues Committee regularly show pesticides showing up in non-organic food. 150 of the 311 pesticides commonly used in non-organic farming have been identified as potentially causing cancer. Do you think this is cause for concern?”

Well, how would you respond? Oh no, I’m fine about eating potentially cancer causing pesticides…(and don’t even get me started on the word “potentially”), so, it fails on one of the most primary quality control checks for surveys: maximise objective assessments, and reduce subjective ones.

Surveys are also supposed to be balanced – otherwise what is the point? With surveys that contain leading questions such as the above, you’ll only learn what you already think you know – which may well have been their objective, who knows? And if the Soil Association (who I don’t have issue with as a whole) wants to back up their marketing messages with data from this survey (I can see it now: 97% of people think non-organic food causes cancer), then that’s up to them and their conscience.

Which is my point here….surveys are most valuable in situations where you don’t know what you don’t know. Validating a marketing message, or an assumption about a marketplace for example, and the quality of surveys should be tightly controlled to ensure objectivity and balance. Why? Well, if you’re planning on using the results for marketing/messaging purposes, you don’t want to be caught out with inaccuracies that have come about through leading questions. At best, your messaging could be based on non-existent industry needs, at worst, your reputation could be impacted negatively.

I am guessing that actually what the Soil Association were trying to achieve was to get my contact details, and given that it was presented to me after completing my shop on Ocado, they probably figured I fit a certain profile, and these statements would be what I wanted to hear. Which is all fine, but don’t dress it up as “one of the most important debates of our time” (their words, not mine), a more credible approach would have been “if you agree with these statements then support the soil association”

So what would a non-leading version of that question above have looked like? How about:

Are you concerned about pesticides in food? If so, please state your reasons:

– because the media tells me I ought to be

– because I have heard that they may be harmful to people’s health

– I don’t know

– I just don’t like the idea of eating chemicals, but don’t know what effect that has on my health

That way, if you wanted value from the survey, you would understand if people are actually concerned in the first place, then you would see what’s driving that concern.

Putting time into planning the questions properly, and avoiding the temptation to slant them to your solution, will ensure that not only do you get the most value from the results, your subsequent actions are based on real fact, and you will have established credibility with your audience.

Another good article, from the US, on this subject, is available here. This article also has some good specifics around how to measure the quality of your surveys.

About the Author

Charlotte Graham-Cumming

Director, Ice Blue Sky Ltd

Leave a Reply

Your email address will not be published. Required fields are marked *