It’s easy to become a market researcher today. It is easy nowadays to make polls and surveys and to ask questions about almost everything. You can make polls even on social media using the tools available in applications such as Instagram, Whatsapp, LinkedIn and I can go on. This post was inspired by a conversation that I had on line with one “guru marketing influencer” following a poll that he made on a social media network, whose results were strange to use a soft word.
What to expect?
I have no issue using the available tools such as forms, polls and anything else available on the Internet. The problem here, comes from what you should expect from the data collected, which most of the cases is crap, from a data analysis point of view. Let me explain!
One of the most important issues when you analyse data is the quality of the quantitative data collected. The old saying of “You get what you give” applies here. You might have the most comprehensive data analysis in the world. If your data is crap, your conclusions are crap. It applies in market research, sociological research, consumer behaviour analysis, medical research… you name it.
Therefore for any kind of research, collecting good data is the most important starting point (assuming you have correctly defined your problem).
What does it mean?
What do we mean by good data? Simply put, by good data we mean that the collected data is valid and relevant for the problem we want to assess. We really measure what we wanted to measure. And this is not trivial at all.
Actually this is more difficult than one would thing because (at least in market research studies) we need to take into account different biases that might interfere with our data collection process:
- sampling issues
- semantic bias
- perception bias
- timing
to name a few related to questionnaire design.
Sampling
For your data to be relevant, it must be representative. The representation is achieved by the way we select the respondents to our survey or poll. A representative sampling is obtain by following some statistical rules (some may call it laws).
Sampling selection is not a simple process, that we might develop in another article. It requires some statistics. There are different sampling methodologies that can be used according to the problem we are studying.
Most of the cases, in social sciences in calculating sampling characteristics we use gaussian distribution.
On short, in order to be representative, a sample should allow equal (or a mathematically computed – according to the sampling methodology) chance of each individual to be represented in the sample. This is achieved through randomisation methodology used in the sample.
Another important feature of the sample is the size. Sample size allows us to calculate the statistical error of the phenomena or the perception we want to measure. If the selection of respondents in the sample was not done according to statistical rules, even if you have a bigger sample, it might not be usable as it can introduce non statistical errors in the sample (you don’t want that). You can have a representative sample starting even with 35 respondents – of course with a high error.
When talking about samples we need to assess the sample error and the confidence interval. They are influenced by the sample size. The bigger, the better.
In statistics we do not have a problem with statistical errors – they are considered controlable.
Semantic and Perception Bias
The data collection tool is basically the questionnaire. Here is where everybody is an expert 🙂
Only that in order to collect good quality data, you need some knowledge. The methodology of developing the questionnaire is qualitative and might involve in depth interviews or questionnaire testing. Anyway you might need to take into account the following:
- the Logic of the questionnaire – certain logic can mislead the respondent as some of the question can be logically related to other
- the way the Questions are Formulated as this is the most major source of semantic bias – in day to day life we might enter an argument with someone because we understand the meaning of the words in the context differently. Same holds true for questionnaires. Different people might understand different things from a question – therefore might answer in another direction. This hold true especially when you use jargon among large populations with different social strata. Similar expectations we have when we apply a questionnaire to people with different expertise on the subject.
- respondents possible Perception toward the way questions are asked. When answering a question some might think that they are evaluated through the questionnaire and there is a right or wrong answer which leads to a type of bias that cannot be evaluated.
- the State of the respondent can influence the way she responds – if the questionnaire requires some attention it should not be applied when the respondent is tired or hungry
- maybe there is a lot to say on this subject, but you’ve got the idea
All of the above mentioned biases, introduce a kind of bias that cannot be statistically measured, therefore these errors cannot be evaluated, producing data intoxication. This type of error is not controllable.
If you cannot afford a professional to help you with the questionnaire, here are some tips:
- Avoid long questions with explanations
- Avoid jargon, use simple words
- KYC (knowing who you are asking questions is important)
- think of the native language of the subject – for instance I am not a native English speaker – some of my formulations might be strange for a native (it is true for large geographical areas with same language speaking populations)
- keep in mind that on closed questions, the options are introducing bias, so use it wisely
- put yourself in the respondent shoes
Data Collection Timing
Keep in mind that every studied phenomena or characteristic has a time evolution. In social sciences almost every characteristic of a population has its own dynamic.
Each survey or poll is in a certain time frame, therefore select the time frame and close the survey, or take into account the time when you collected the data (they can both work – it depends on what you are really researching).
Should I not use SM for polls and survey?
I’ve never said that!
Just keep in mind that social media is a connection and communication tool. Use it for what it is. Make a survey, engage, exchange ideas, but take all the results and the outcomes with a pinch of salt.
It might not be representative for what you intend to do. Please keep in mind that people that tend to respond to social media survey – have a different profile than the average population, because the average Joe or Jane, does not have a tendency to respond to surveys. They have a different psychological profile.
But on the other hand sometimes it can be the only available tool you have. I hope you will think at least twice when you design the questionnaire.
I hope you liked it! I’ve put a huge deal of work in creating this material. If you have any comments, please contact us.