If you insist on launching a survey, there are several simple ‘rules’ that will ensure that the information you get is meaningful. There are also some crass mistakes that ensure that it will be meaningless. There are books written on market research, and with good reason. If you don’t know anything about market research then either read a few such books, or enlist the help of someone who does know what they are doing.
If you, or your organisation can’t afford an expert, then one REALLY simple ‘golden rule’ is to get a handful of friends to try to complete the survey for you – before you send it out. When you send out a survey you are representing your organisation. A poorly worded survey, or one that many people won’t be able to complete accurately, gives the recipients a bad impression of your organisation. It also means that the information that you gather will be meaningless too.
The example above is from a survey by a British police force. This is question 3. Incidentally, the first contained a typo. Think through this question. “Which of the following methods” implies that more than one answer is likely to be possible. Would most people have one preference? After all, if there’s an axe murderer loose in my immediate neighbourhood I’d probably like the face-to-face reassurance of a police person. If there’s been a spate of littering, I probably don’t need to hear about it – as I will be able to see it for myself. If thieves are targeting cars with belongings visible on their back seats, then perhaps a message online would suffice. There is no one answer to this question. So, make sure that your survey allows more than one. This is the kind of stupid mistake (yes, you do need to be pretty stupid to make it) that a friend will point out to you. Assuming, of course, that the friends aren’t afraid of you and aren’t as stupid as you.
Here’s the next question in the same survey, merely as an illustration of course. I think they meant to ask “Which face to face methods would you like us to use to keep you informed?” which is a very different question, but let’s go with what they asked. Here, we can select all the answers that apply, which is progress, but wait… what if, there’s another method that we would like to use? Perhaps to know that there would be a police officer in a particular place at a given time, the pub for example. Perhaps to have one knock on your door when that axe murderer is about? Perhaps to have one living in your village who you can speak to ‘off the record’. They are all valid responses, but there’s nowhere to give them.
Here’s the very next question….. (Notice the completion bar, used to indicate the respondent’s progress through the survey (in other words how much more of their time you are expecting them to donate)… it has dropped BACK. In other words, completing the last question has meant that we have to spend more time than previously…. How can that be so?)
Which question would they like us to answer? They are all different. “How well-informed do you feel?“, “How well-informed do you feel and would you like to join any of these groups” (well that’s one and a half questions, really, since they forgot to finish the second one!), “Do you feel informed enough about how to prevent [yourself from] becoming a victim of rural crime?”
This particular example gets worse. Here’s the next question (you can guess that I answered “No” to the last one).
Feeling not well enough informed to avoid being a victim is VERY different to being able to prevent crime. Seriously, this police force is officially asking the public for advice on how to do their job properly? “How can we help prevent rural crime?” Just as there are degrees in market research, so too are there ones in criminology and policing. Perhaps sending some of their colleagues on them would be the best answer? But to ask us? That isn’t our job. We don’t ask you how we should do ours…
It may seem as though I am picking on this police force, but these issues are SO common that it really does apply to most online surveys; I have simply taken the latest example to arrive in my inbox.
As I said at the beginning, if you’re thinking of sending out a survey simply ask a few friends to complete it first. Colleagues won’t do. They probably think the same way that you do, from the same perspective, and they may not give you honest straight-forward feedback for fear of retribution or of hurting your feelings. If you often find yourself having to publish information, which is what a survey is, then maybe it is worth having a small panel of people with whom you can pilot your poll and test out other publications with on a regular basis?
Whatever you do, remember that most people will find themselves spending 5 to 10 minutes either completing it or reaching the point where they give up in despair. On average, that means that every 9 people who do so will represent a wasted hour of effort. In this case, I guess they are hoping for a response from 300 or so members of the public. That represents a colossal 33hrs of voluntary time that has been frittered away. Could it have been better spent? (That’s a rhetorical question – you don’t need to answer it!)
Best wishes, Graham
