Step 2: Designing and a structuring survey(1)

2.1 Provide context

The EO should first position the survey in terms of the EO's overall goals. What is the EO trying to achieve? (e.g. surveying members on these issues because it wants to ascertain whether "issue X" is a constraint on enterprise growth and want to use the information gathered in its dialogue with government). Include something from the outset that ensures participants that confidentially will be respected; below is such an example:

"The goal of this survey is to gather information and opinions about constraints on enterprise. Ultimately, the information gathered here will help the EO to advocate to governments' policies and programmes that enhance employment and economic growth.

The information obtained here will be held in the strictest confidence. Neither your name, nor the name of your business will be used in any document based on this survey."

The EO should only undertake a questionnaire or survey if it knows exactly what it wants.

2.2 Keep it simple

In general, in terms of design and length the easiest rule is to make the survey simple and straightforward.

In order to write effective questions, you need to consider four important factors: directness, simplicity, specificity, and discreteness.

  • Write questions in a straightforward fashion using direct language.
  • Don't use jargon.
  • Specifically tailor questions to the membership.
  • Keep questions short and simple.
  • Specific questions are usually better than general ones. The more general a question, the wider the range of interpretation of an answer.
  • Avoid questions that are overly personal or direct, especially when dealing with commercially-sensitive issues.

Test it thoroughly in terms of these indicators before it is sent to members.

2.3 Survey Structure

Putting effort into the design of the survey will save the EO time when it comes to analysing the data. Respondents should be able to navigate the questions clearly and logically and the survey should flow in a structured manner.

Questions should be designed with an eye towards those being surveyed, as the data gathered can additionally serve as supporting tools for interviews and focus group discussions.

Survey design should allow for easy compilation of the data obtained (for example, by designing questionnaires with closed questions). Design should allow for a systematic analysis and documentation of the responses, particularly if the sample is quite large.

2.3.1 Question and Language design

It is extremely important that the EO design the survey with neutral, gender sensitive language. Use straightforward language and never use jargon.

Be careful with any definitions used and have absolute clarity in what is being asked of those being surveyed. For example in asking the question "Are young people more employable today than five years ago?", without exactly defining the term 'young people', there is the possibility of the term being interpreted as anyone from 15 to 40 years of age.

The EO should always give the option of "No" even on an issue the EO thinks 100 per cent of employers will respond "Yes". For example, if the question was 'Do you favour higher taxes on your profits?", providing options for both "Yes" and "No" responses would allow the EO the ability to report the percentage of employers against such a proposal. It is also vital for the advocacy process.

Surveys can suffer from reactivity, in that companies or people can offer responses that are socially desirable rather than accurate; however, good survey design can improve this. Much depends on how the question is asked, for example, passive voices, maybe using slightly different (less emotive) language can sometimes get around this (unless the subject matter is of a particularly sensitive nature).

The EO should not make assumptions in the questions it asks.

In its approach to the survey the EO should try and ask questions in a way that allows companies to provide answers that deviate from the replies that may be expected.

Ensure that you avoid 'off-putting' questions. These are questions that ask such sensitive issues that they may cause a respondent to not take the survey at all. There are ways of arriving and extracting the information you may want by taking a more subtle and incremental approach.

If questions are structured in a logical sequence and ease the respondent into the process it is also possible to get information that the EO may have not expected to arrive at.

Ask the easiest questions first. Once a respondent has invested time in a survey they are more reluctant to not finish it, and will choose to ignore a difficult question. Whereas if the difficult questions come early in the survey they are more likely to not continue.

Avoid leading questions, for example: "How big an issue is climate change?" with a follow up questions such as: "Do you think the Green party has the best climate change policies?"

With respect to the time span of questions asked they should not seek information more than five years in the past. This is about as long a time horizon as should be used. Institutional memory beyond this point can be difficult to capture.

Questions can be closed ("Have the services delivered by the EO been good or bad?"), or open ("What do you think about the services?"). Both types of questions are examined below.

2.3.1 Closed questions

Closed questions limit respondents' answers to the survey, but they make analysis easier. The participants are allowed to choose from either a pre-existing set of dichotomous answers, such as Yes/No, True/False, multiple choice questions with an option for "Other" where a respondent can write their own answer, or a ranking scale response option.

The most common of the ranking scale questions is called the Likert scale question. This kind of question asks the respondents to look at a statement (such as "The most important issue facing the business community this year is the deterioration in export markets") and then to "rank" this statement according to the degree to which they agree ("I strongly agree, I somewhat agree, I have no opinion, I somewhat disagree, I strongly disagree"). Experience suggests that an odd number of options tends to generate a high proportion of answers in the middle – using an even number of choices forces respondents to come down on one side or the other of 'average'.(2)

Advantages of closed questions:

  • More easily analysed;
  • can be more specific, thus more likely to communicate similar meanings;
  • require less time from the interviewer.

In terms of the closed (tick box type) questions, one of the major issues in survey design is that any pre-set list must be comprehensive and mutually exclusive, with only one item at each tick box. The EO should try and limit the number of answer options and certainly no more than 10 or 12.

2.3.2 Open-ended questions

The reason for using an open-ended question is to get as much information as possible. Open-ended questions do not give respondents answers from which to choose, but are phrased in a manner where respondents are encouraged to explain their answers and reactions to the question with a sentence, a paragraph, or even a page or more, depending on the survey. The EO should try to ask questions in such a way as to facilitate the most detailed analysis possible.

Open questions are like: "What do you think is the most important issue facing the business community this year?". If you want the respondents to focus on their answer, the question can be phrased as "Do you think that the most important issue facing the business community this year is the deterioration of export markets? Explain your answer below." (Although you should note that writing questions in this way is likely to introduce bias into the answers).

Advantages of open questions:

  • They allow for more information;
  • they cut down on error by eliminating the option of not reading the questions and just "filling in" the survey with all the same answers (e.g., filling in the "No" box on every question).
  • they allow for obtaining extra information, and can be used more readily for secondary analysis.

Open-ended questions are also difficult to analyse and often they do not allow the EO to make definitive statements. For example, if the following question is asked: 'What is the most important challenge for your firm in the next 12 months?' The EO can only deal with each of the answers individually; what it can then say in its analysis is something like: "Energy costs were seen as the most important challenge for 10 per cent of respondent companies.' The EO cannot make any statements about the relative importance of energy costs to firms who did not list this as the most important challenge, although it may have been important to them nonetheless. Therefore this important comparative ability can be lessened by using an open-ended style.

The EO will need to make a decision on which type of questions it will pursue, dependent on the resources and expertise it has. Both types of question style have pros and cons, and both have limitations.

 

Biases commonly found in surveys:(3)

Hawthorne Effect:

One bias commonly found in survey research is the Hawthorne Effect, which states that respondents tend to respond differently simply because they have been selected for a survey. Because of the special recognition which has been given them, it is sometimes found that the respondents tend to answer in the way which will most please the researcher. To minimize this bias, the questioner should be as neutral as possible in presenting the survey.

"Self-lifting" bias:

Closely associated with the Hawthorne Effect, the "self-lifting" bias recognizes respondents want to make themselves appear in a positive light, and will respond accordingly. This bias can be minimized by positioning personal questions about respondents at the end of the questionnaire, where they would tend not to affect other, more substantive responses.

The "Habit" bias:

If given a series of similar questions, respondents will fall into a habit of answering them similarly without considering each on its own merit. This bias can be minimized by changing the format of questions throughout the questionnaire. The format may range from simple "tick the box" questions, to one-word responses, to open-ended responses, to completing information on simple graphs and maps. Through these variations, each question is given its own personality, thus avoiding the "habit" response.

Non-respondent bias:

One type of bias often found in surveys is based on the assumption that individuals who haven't responded in a survey tend to feel the same as those who have responded. However, studies have found that non-respondents generally have a more negative outlook, which may not be otherwise represented in the collected data; although non-respondent bias can also be redolent of a more positive viewpoint. Sometimes with surveys verifying a "problem issue" only those with the problem respond, so non-respondents may in fact be more positively biased.

Stopping Rule:

A bias sometimes found in survey research is based on the fact that the survey procedure is terminated when a researcher has obtained the desired results, and the amount of data collected is determined by results. This can be avoided by establishing survey parameters beforehand.


(1) David S. Walonick: Survival Statistics: Designing and Using Surveys (1997-2004).

(2) The Survey System: http://www.surveysystem.com/sdesign.htm

(3) Eastern Michigan University: Education First Developing a Downtown Survey.