Now that you have your survey and design all lined up, it’s time to invite respondents to your survey. This may seem like the easiest step in research, how hard can gathering data be? Let me burst your bubble and tell you, this is one of the most important aspects of survey research and can determine the fate of the entire research endeavor. No pressure.

You need to make sure that you target the right respondents so that they accurately represent what you wish to look into. But, how do you ensure accurate targeting without introducing a source of bias to your research? Bias is the concurring evil of all research, so here are 4 things to consider:

1. The what and the who: research interest and population size

When determining who should answer your survey and how they get a hold of your survey, you first need to figure out what your research interest is. It doesn’t make sense to survey all students at the University of Oxford if you’re trying to figure out how satisfied students are with the dorms at the university. For this, you need to define your population, which is the entirety of your research subjects. For this example, the population is the number of students living in the University of Oxford’s dorms. However, it’s often impossible to survey an entire population due to time and cost issues. Luckily, most of the time, surveying the entire population isn’t necessary. Drawing inferences from samples often get you pretty close to the actual population, just be mindful of the inherent uncertainty they carry.

2. Random sampling

In order to achieve a result that comes as close to the truth as possible, you need to carefully sample your respondents. A sample frame is a list containing the entire population, from which a random sample is drawn. Continuing with the above example, this would be a list of all students living in the dorms, from which random names are selected to participate in the survey. You need to make sure that each student has the same chance of being selected so that you can minimize the sampling error (the natural deviation between sample and population). This is called probability sampling.

3. Sample size

We know you’ve been waiting for this one. The answer to the question; how many respondents do I need? Well… that depends on many factors but don’t worry we have a formula for you.

Let’s not get ahead of ourselves and take this one step at a time. You have to set the maximum error you’re willing to accept in your survey. When doing this, you should be aware of the following two parameters: margin of error and confidence level.

The margin of error is the interval within which you expect to find the value from the population you’re measuring. For instance, let’s say you wish to determine how large the proportion of the 10.000 students in the dorm are exchange students. If we assume that 1.000 are exchange students (10% of the population) with 5% margin of error, it really means it is about 500 (5%) and 1500 (15%).

The confidence level expresses how confident you feel about the value you look for within the margin of error. For example, in the previous case, if you choose a 95% confidence level, we could say the percentage of students in the dorm that are exchange students in 95% of the cases, is between 5% and 15%. In other words, if we repeat the survey 100 times, the proportion we’re looking for would be within the interval 95% of the cases and it would be out the interval in the remaining 5% of the cases.

The margin of error, confidence level, and sample size are always linked and co-dependent. Modifying any of these values will change the others:

• Minimizing the margin of error will require a bigger sample size.
• Increasing the confidence level will require a bigger sample size.

So, once you’ve decided on the margin of error and the confidence level you can use the formula to determine your sample size!

Pro tip: As a reference, the margin of error in political polls is usually 3%.

4. Non-response and response bias

Once you’ve determined your ideal sample size, add a couple extra! Why, you ask? Because there will most likely be some who don’t want to answer your survey. To counter the effects of people not responding, you may want to increase your sample size by the expected non-response rate. So, why should you care?

This has got to do with two more biases related to your respondents that may influence your data. A response bias is mainly on the side of the respondent, who doesn’t understand the question or is lying while answering the question. You can counter this by making sure your questions are properly phrased and that respondents trust that their answers are anonymous and/or confidential.

It takes people 1/10th of a second to form an opinion about a person, and surveys are no different. Always keep in mind that surveys speak in two languages; words and visuals. So, your survey’s first impression relies on these two complementing each other.

Images, colors, and fonts

We are sure you already know this, but we’ll say it again – colors, fonts, and images are important! Plenty of studies (shout out to Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method by Don Dillman, Jolene Smyth, and Leah Melani) show that these incite feelings and attitudes on people, which means you shouldn’t overlook the impact these can have on your respondents.

You can find endless research on colors, fonts, and images and we encourage you to do so, however, today we’re taking it a step further. We want to focus on matching your survey’s visuals to its content.

Avoid bias by conveying the same message through visuals and words

Make sure your visuals and text are saying the same thing. If you have chosen a casual and informal tone in order to target a certain segment of respondents but your visuals are strict and rigid, you’re creating cognitive dissonance and influencing your respondents’ performance.

For example, if you conduct a survey on attitudes towards different social media platforms and you use Facebook’s color scheme, you will end up with invalid data

Your survey is part of your brand

Every and any interaction that customers have with your brand is defined as a touch point, and it can have a positive or negative effect. Your survey is a touch point, it’s part of your brand, that’s why your survey design should align with your brand identity and be consistent.

We live in an online world, which means customers experience brands throughout a variety of channels and it’s important to keep their experience consistent. So, if your website and social media are branded then your survey should reflect your company’s brand presence as well.

Engage your audience through visuals

You have chosen a specific tone and words to address a specific audience, however, have you chosen the right visuals? For example, let’s say you’re conducting a survey about homeschooling and include images of classrooms, however, your target audience is homeschooled children, then these images are not going to resonate with them, they might even hit the wrong note. Make sure your visuals align with your content and brand, as well as speak directly to your target audience.

Ultimately, you can jeopardize your data by overlooking your survey’s visual design. Remember, it is the first thing your respondents will notice and you don’t want to lose them before they even read the content of your survey. Visual design done well can increase response rates and minimize bias, however, when done poorly it can have the opposite effect.

Enalyzer provides multiple survey design templates, which can serve as a great starting point for your survey design – the possibilities are endless.

When collecting data from an audience, you need to be respectful of their time and you want to make sure your survey keeps the respondents engaged from “hello” to “thank you”. A key aspect of keeping your respondents engaged is avoiding survey fatigue.

Survey fatigue can be divided into two different types, both of which can have a profound effect on your survey’s response rate, as well as the quality of the data collected.

The first type of survey fatigue starts before your survey even begins and stems from the increased amount of surveys currently being circulated. People are constantly being asked for feedback, whether it be by the local grocery store or their workplace, and are simply tired of answering surveys. This type of survey fatigue is the hardest to battle as this is not really dependent on your specific survey, but an overload of surveys in general.

The second type of survey fatigue is related to the fatigue your respondents may feel when actually taking a survey. This type of fatigue happens if your survey is too long, complicated, or confusing and may lead to the respondents rushing through or exiting the survey prematurely- leaving you with a lack of quality data. 

Though the above might sound bleak, don’t fret! We have comprised a set of tips for how you can avoid your respondents getting survey fatigue.

Don’t drown your audience in surveys
Don’t send more surveys than absolutely needed. This way the chances are higher that your target audience will respond to the survey, as opposed to if they have already received four surveys from you this month. If your organization is dependent on sending out a lot of surveys, try to keep track of when different departments are sending out surveys, so the same people are not answering surveys from multiple departments at the same time.

Communicate the survey’s value clearly
If the respondents know how their responses will be used and what the aim of the survey is, they are more inclined to stick through the survey. So make sure to communicate this clearly. When designing your survey reflect on and efficiently communicate the following to your respondents:

• Why should they take your survey?
• What will the answers be used for?
• How time-consuming is the survey?

Always think of your respondents
Though the survey might ultimately be beneficial for the respondents themselves, through for example improved work or customer experience, while taking the survey they are doing you a ‘favor’ by sacrificing their time. This is not something that should be taken lightly, so make sure that their time isn’t wasted which can be done by:

• Using behavior and conditions to make your survey as ‘respondent-friendly’ as possible. This way you won’t confuse your respondents with irrelevant questions.
• Asking the right questions and not asking too many of them. Keep it simple and only ask the questions you absolutely need to. Short and sweet is the way to go. Put your survey to a ‘nice to know vs. need to know’ test. If questions in your survey are ‘nice to know’ rather than ‘need to know’, drop them. This will give you better quality data and a higher response rate ensuring that you get the information that you ‘need to know’.
• Get creative. A good looking survey is inherently more pleasant to answer so put some effort into your survey design. If you need inspiration check out our blog post on flawless survey designs.

What is a good question? A good question, is a question that asks the right thing in the right way.

Last week we talked about asking the right things by transforming your objectives into survey questions. Today, we will look at how to ask the questions the right way, to ensure higher response rates and better data.

For this purpose, we have comprised a list of 7 golden rules for survey question writing:

• Clear questions are the best questions
  If your question is not clear, your answer won’t be either. So keep it simple, to make sure that your respondents understand what you’re asking.
  A question should only include a single idea, including several questions will confuse respondents and it will be impossible for you to interpret their answers.  Let’s try this in practice:

  
  
  If a respondent answers “satisfied” to this question, how will you know what it means? Is the respondent satisfied with the teacher or the catering? Or maybe the respondent was “very satisfied” with the teacher and “unsatisfied” with the catering? See, it’s confusing!
  A simple mistake as this, creates invalid feedback on the teacher and catering during the course, making it impossible to come up with solutions. These types of double-barreled questions can often be spotted by the use of the word ‘and’, signaling the connection of two different focuses: “… the course teacher AND the catering”.  In other words, by applying the one-idea-per-question rule, you won’t confuse your respondents and collect sound data.

• Avoid hypothetical questions
  When you ask hypothetical questions, it often results in unreliable data caused by respondents not being able to understand your hypothetical scenario. The question “Imagine that you’re buying a new car, what kind of financing will you prefer?” is virtually impossible for someone that has never considered buying a car, or doesn’t have the knowledge of the different financing options, to answer. Instead, it would be better to ask someone who has recently bought a car how they financed the purchase.

• It’s all about the context
  In some cases, questions and their answers will only give insights if understood in a certain context established by other questions. For example, if asking about a respondent’s attitude towards Buddhism, can you adequately interpret this without finding out about their attitudes towards religion in general, or other religious groups? In such a case, contextual questions are your friend since they ensure that you’re getting the full picture and the valid information you need.

• Your response options have to be all-inclusive
  Make sure that your response options allow respondents to answer your question. Let’s look at an example:
  
  
  
  Here, a respondent who has worked at Enalyzer for over a year but less than 2, can’t adequately answer the question. This will inherently have an effect on your data’s validity, plus he/she is now feeling left out and no one wants that. In this case, you need to make sure that your response options fit all possible answers. For the above example, one could add an extra response option, ‘1-2 years’ or extend one of the other options to include this time span.

• Find the balance between being too specific and too broad
  When writing survey questions, you need to keep your survey’s goals and objectives in mind at all times in order to make sure that your questions allow for the answers you need. So, it might be necessary to reflect on the correlation between being more specific or sufficiently general and the possible answers you can get.
  General questions can sometimes lead to information that is difficult to interpret. For example, let’s say that you’re a business owner that is interested in knowing what customers think about your service. To find this out you could ask “how well do you like my services?” rated on a scale ranging from “not at all” to “extremely well”, but what would a possible response to this mean? What exactly does it mean that someone likes your services? Instead, you could ask more specific questions such as “would you recommend my services to others?” or “would you use my services again?”.
  In other instances, you may need to evaluate whether your question is sufficiently general in order to make sure that the answers you are getting accurately reflects the respondent’s attitude towards the topic of choice. For example, if you ask someone how they have thrived at their workplace for the last week, you could get a very different answer than if you asked them how they have thrived there the past year. Perhaps, the respondent had a bad week, but this doesn’t necessarily reflect their sentiments at their workplace in general.

• Keep them relevant
  When making a survey always keep in mind that you’re ‘borrowing’ time from your respondents that they could have otherwise used on something else. Therefore, it is important not to waste this time by asking irrelevant questions. Avoid this by going through all of your questions before sending your survey, making sure that you actually need to ask the question and whether you need to ask it at the level of detail you currently have. For example, if you’re asking a question about your respondents income, do you need to know the exact number, or would your reporting needs be satisfied by income ranges?

• Make them neutral
  Survey questions and response options should be neutrally formulated so that you don’t lead respondents to a particular response. Also, respondents should be able to answer questions both positively and negatively. Here is an example:
  
  
  In this example, the response poles, disagree and strongly agree, are not balanced and there are more positively loaded options than negative ones. This should be avoided as it can sway the respondents’ replies and no one likes a manipulator!