Questionnaire design: garbage in, garbage out

‍

One of the most useful ways to collect data when conducting market research is via the use of HUMINT (Human Intelligence). Data can be collected via in-depth qualitative interviews or large scale quantitative surveys. In this blog, the focus is on the latter. Quantitative surveys are a great method to gather representative information about a specific target group.

In order to obtain valuable input for analysis you need a well-designed questionnaire. If your questionnaire isn’t well designed, you will have trouble getting valuable results out of it: ‘garbage in, garbage out’. This blog will provide some guidance on how to prevent garbage input (and thus garbage output) when designing a questionnaire for a quantitative survey.

‍

Research purpose and KPI’s

When you plan to conduct a quantitative survey, you need a clear view on what you want to achieve after collecting and analyzing the results: the purpose of your survey. Ask yourself the following questions during the questionnaire design:

• What is the intended use of the insights resulting from the survey? What should the survey results clarify to make a substantiated decision?

The answer to these questions will not result into survey questions yet, but rather into measurable Key Performance Indicators (KPI’s), e.g.: market share, brand awareness, buying criteria, satisfaction etc. Also, keep in mind that if you want to show the final results in a data visualization tool (e.g. PowerBI, Tableau) that the KPI’s are easy to visualize and to understand for the user.

‍

Creating key questions

Once you have a clear purpose to conduct a survey and awareness which KPI’s you need to measure, it is time to create the key questions. These questions provide a result for the sample of respondents. While creating questions there will probably pop up all kinds of secondary questions. It is vital to clearly distinguish the key questions measuring your KPI’s from the ‘nice to know’ questions to position the key questions in such a way that they can be answered unbiased. From those ‘nice to know’ questions, only keep those questions that really add value.

A longer questionnaire leads to distracted, less motivated respondents.

‍

Respondent profile

A key aspect of analysis of a quantitative survey is segmentation. When you analyze the results of the survey, it is beneficial to segment your sample to identify significant differences between types of respondents. This is why your questionnaire should start with questions that create a respondent profile. Obvious questions that come to mind are demographics like age and gender, but it is important not to miss out on any indicator where a segmentation of the sample could be useful.

If we would for example conduct a survey among 100 CEO’s of European companies, it could be useful to segment the CEO’s by personal indicators like how long they have been in the position of CEO, how long they have worked at that particular company, what other jobs they have had etc. It could also be useful to segment the companies they work for on indicators like company size, the sector they are active in, whether they operate B2B or B2C etc.

‍

Crystal clear questions

One thing that should be present throughout the entire questionnaire is clarity for the respondent. Be sure that it is 100% clear what you mean to the respondent, every single question. You can achieve this by providing context with text, images or video for any question or term that may possibly be interpreted in multiple ways

A good way to provide context about specific terms is by adding a sentence right after a question where you have used a difficult or multi-interpretable term. For instance if you ask the open question: which products of term X can you name? You should follow with: By term X we mean A, B and C, excluding X, Y and Z.

Besides providing clarity about definitions, it is also important to provide clarity on what is expected from the respondent. For instance: ranking buying criteria on importance on a scale of 1-  5, answering with a number, multiple answers being possible etc. Make this explicit! It will not only provide clarity to the respondent but also make life easier once you analyze the results.

A proper routing will also provide clarity to the respondent and make the analysis easier. Think thoroughly about which answers will lead to skipping or jumping to other questions. If I ask the 100 CEO’s whether they have heard of Hammer and I want to follow up with a question about their perception of Hammer, I should be sure that the respondents who have never heard of Hammer skip the follow-up question about perception!

‍

Preventing bias

In order to get valid results out of your survey, it is key to have your respondents answer questions as unbiased as possible. The two most common ways bias can sneak into your questionnaire are by:

• ‍Loaded wording: when you ask your respondents about sentiment it is key to ask the question as neutral as possible. Avoid adding loaded words about your topic. A question like: ‘How beautiful do you find logo X?’ is a no go for instance. Instead, you could ask ‘What is your opinion about logo X?’ Rate from 1 – not beautiful at all to 5 – very beautiful
• ‍Order of text and questions: quite often in surveys questions are asked about a topic that has already had attention earlier. Be sure to avoid giving any information that sends the respondent in a certain direction about a topic you will ask later on.

‍

Questionnaire feedback

Last, but not least, be sure to always obtain feedback before approaching respondents. No matter how carefully you designed your questionnaire, it is very common to overlook minor mistakes that are easily made. Secondly, it is important to gain feedback from a content perspective. Are all KPI’s properly measured? Are there any questions missing? Or are there questions in you questionnaire that should be removed? It is best to have your draft questionnaire checked by three types of people:

• ‍Stakeholders: those people who benefit from a successful survey. At Hammer, when we conduct a survey assigned by a client, we align the questionnaire narrowly with their demands. In the end, they have to make decisions supported by the results of the survey.
• ‍Experts: those people who have the knowledge and experience to improve your questionnaire by viewing it from their perspective. This can be either experts on the topic or experts on the research method. If I want to conduct a survey among dairy farmers for instance, I will ask both a dairy/agriculture expert from our network and one of my experienced co-workers with market research knowhow.‍
• People part of the target group: asking feedback from someone who fits the criteria of your target group is invaluable. This is a great check to find out if the questionnaire is completely clear to respondents. Running the questionnaire by a potential respondent can also be used as a pilot to see whether the types of answers that come out of it are analyzable.

‍

Designing a solid questionnaire can easily be underestimated. Make sure your research objective is well defined, as well as the responding KPI's. Make the questionnaire as clear as possible for the respondent, prevent bias and ask for feedback from different types of stakeholders.