A valid questionnaire measures what it claims to measure. In reality, many fail to do this. For example, a self completion questionnaire that seeks to measure people's food intake may be invalid because it measures what they say they have eaten, not what they have actually eaten. Similarly, responses on questionnaires that ask general practitioners how they manage particular clinical conditions differ significantly from actual clinical practice. An instrument developed in a different time, country, or cultural context may not be a valid measure in the group you are studying.
Reliable questionnaires yield consistent results from repeated samples and different researchers over time. Differences in results come from differences between participants, not from inconsistencies in how the items are understood or how different observers interpret the responses. A standardized questionnaire is one that is written and administered so all participants are asked the precisely the same questions in an identical format and responses recorded in a uniform manner. Standardizing a measure increases its reliability.
Just because a questionnaire has been piloted on a few of your colleagues, used in previous studies, or published in a peer reviewed journal does not mean it is either valid or reliable. The detailed techniques for achieving validity, reliability, and standardization are beyond the scope of this series. If you plan to develop or modify a questionnaire yourself, you must consult a specialist text on these issues.
There are two main objectives in designing a questionnaire:
To maximize the proportion of subjects answering our questionnaire - that is, the response rate.
To obtain accurate relevant information for our survey.
To maximize our response rate, we have to consider carefully how we administer the questionnaire, establish rapport, explain the purpose of the survey, and remind those who have not responded. The length of the questionnaire should be appropriate. In order to obtain accurate relevant information, we have to give some thought to what questions we ask, how we ask them, the order we ask them in, and the general layout of the questionnaire.
As discussed in last month's issue, there are three potential types of information:
Information we are primarily interested in-that is, dependent variables.
Information which might explain the dependent variables-that is, independent variables.
Other factors related to both dependent and independent factors which may distort the results and have to be adjusted for - that is, confounding variables.
Let us take as an example a national survey to find out students' factors predicting the level of certain knowledge, skills, and attitudes at the end of their undergraduate medical course. The dependent factors include the students' level of relevant knowledge, skills, and attitudes. The independent factors might include students' learning styles, GCSE and A level grades, socioeconomic status, ethnicity, etc. Confounding variables might include the types and quality of teaching in each medical school.
Sometimes, additional questions are used to detect the consistency of the subject's responses. For example, there may be a tendency for some to tick either "agree" or "disagree" to all the questions. Additional contradictory statements may be used to detect such tendencies.
There are several ways of administering questionnaires. They may be self administered or read out by interviewers. Self administered questionnaires may be sent by post, email, or electronically online. Interview administered questionnaires may be by telephone or face to face.
Advantages of self administered questionnaires include:
- Cheap and easy to administer.
- Preserve confidentiality.
- Can be completed at respondent's convenience.
- Can be administered in a standard manner.
Advantages of interview administered questionnaires include:
- Allow participation by illiterate people.
- Allow clarification of ambiguity.
The exact method of administration also depends on who the respondents are. For example, university lecturers may be more appropriately surveyed by email; older people by telephone interviews; train passengers by face to face interviews.
Piloting and evaluation of questionnaires. Given the complexity of designing a questionnaire, it is impossible even for the experts to get it right the first time round. Questionnaires must be pretested - that is, piloted - on a small sample of people characteristic of those in the survey. In a small survey, there might be only pretesting of the drafted questionnaire. In a large survey, there may be three phases of piloting. In the first phase we might ask each respondent in great detail about a limited number of questions: effects of different wordings, what they have in mind when they give a particular answer, how they understand a particular word, etc. In the second phase the whole questionnaire is administered by interviewers. Analysis of the responses and the interviewers' comments are used to improve the questionnaire. Ideally, there should be sufficient variations in responses among respondents; each question should measure different qualities - that is, the responses between any two items should not be very strongly correlated - and the non-response rate should be low. In the third phase the pilot test is polished to improve the question order, filter questions, and layout.
Usually respondents spent an average of 33 minutes answering a variety of questions about their «backgrounds, tastes, and their shopping and media consumption habits».
The Grande Bretagne Hotel most (8,852) came from Britain, followed by the U.S. (3,747), and Norway (3,244). The fewest responses were from Venezuela (197), Portugal (175), and Austria (90). And responses was split pretty equally between genders: 51 % of survey takers were female, 49 % male. Most respondents were in the 13- to 15-year-old age group (60 %), followed by 16- to 18-year-olds (19 %). Only 12 % were 12 and under (which is odd since players are supposed to be at least 13 to play), and 10 % were 19 and older.