Introduction
The Internet offers a unique way to administer quantitative survey questionnaires. To date two methods have been widely used, these are web-site questionnaires and e-mail questionnaires. The types of sampling frames have been discussed at length (Schillewaert et al 1998), and numerous methods have been employed. This study was designed to explore the advantages and disadvantages of an e-mail questionnaire administered by using respondents selected from Internet Discussion Groups, a sampling frame which hitherto has received little attention in the published literature. (Witmer et al 1999 makes a notable contribution to this field).
Discussion Groups
For the purposes of this article the term 'Discussion Group' embraces the terms newsgroup, Usenet, Interest Groups, Discussion Lists, ListServ. and ListBot. There are thousands of Discussion Groups on the Internet which bring together individuals with a common interest to discuss a wide range of topics. These discussions take place in type-written form, subscribers may ask questions, offer answers or make general comments that will be read by other subscribers
In 1998 four such groups were concerned with the subject of graphology: "the study of personality characteristics derived from handwriting" (Bradley 1996:97). These four groups carried the names Handwriting-L, Vanguard, Grafologia and Grapho.
Handwriting-L was formed in 1995 by Gary Brown based in Oregon USA. At the time of the survey in November 1998, there were 159 subscribers. Vanguard was formed in September 1997 by Sheila Lowe based in California, USA. At the time of the survey there were around 150 subscribers. Grafologia was formed in June 1998 by Gaetano Rizzo in Calabria, Italy, there were 34 subscribers. Grapho was formed in June 1998 by Michel De Grave, based in Brussels, Belgium, there were 48 subscribers. There was some overlap of subscribers between the four lists, most evident on the American-based ones.
Table 1. Subscriber Numbers in November 1998 (Source: moderators)
|
List Name |
Originator |
Number |
|
|
|
|
|
Handwriting-L |
Gary Brown (USA) |
159 |
|
Vanguard |
Sheila Lowe (USA) |
c150 |
|
Grafologia |
Gaetano Rizzo (IT) |
34 |
|
Grapho |
Michel De Grave (BE) |
48 |
|
|
|
|
Nature of the Sampling Frame
Both American lists conduct debate almost exclusively in the English language to generate up to 20 messages per day. Handwriting-L is open to all and made possible by America On Line (AOL). The moderator tries to ensure that entrants are interested in the subject area and are prepared to provide him (and other subscribers) with some biographical information. Conversely, Vanguard is by invitation only and made possible by RI Software who market software which provides semi-automated handwriting analysis based on the methods of Sheila Lowe. Most Vanguard subscribers are customers or have been invited personally by Sheila Lowe.
Grafologia generates fewer messages per day, mostly in Italian. Subscribers are recruited by a web-site invitation and also by personal referral. The list which originated in Belgium (Grapho) has variable traffic. Messages are mainly in French and English but also in Italian and Spanish and to a lesser extent in Flemish and German. Frequently users mix languages in messages and often computer-generated translations are provided by or polyglot summaries are present. The moderator recruits subscribers personally.
The Handwriting-L subscriber list is available to anyone by sending a simple command to the server. The identities of subscribers to Grafologia and Grapho are only available to the moderators - who both made lists available for this research. The Vanguard list is available to neither the general public nor the moderator. The "host" can only generate such a list by a lengthy examination of data files, it was not made available for this research.
Objectives
The research was exploratory. The objective was to discover whether Internet Discussion Groups offer a feasible means by which an e-mail questionnaire could be administered, to examine response rate, response speed and to summarise the experience.
The following hypotheses were formulated:
H1: Response rate is higher for individuals contacted by e-mail than for those contacted through the mail.
H2: Response speed is higher for individuals contacted by e-mail than for those contacted through the mail.
H3: The age-profile of respondents is consistent with known age-profiles of Internet users.
The response rate definition proposed by Talmage et al (1988:100) was adopted for this study. It is "the number of adequately completed questionnaires obtained expressed as a percentage of the number of eligible individuals". For response speed the definition proposed by Tse (1994) was used, it is "The time lag between return time and the mailing or e-mailing time of a usable questionnaire".
Method
The four Discussion Groups described account for some 300 people interested in graphology, who use the Internet. There are other people not subscribed to the four lists who use the Internet, and there are others interested in graphology who are not Internet users at all. For the purposes of the study these four groups were deemed suitable, since they offer a ready source of individuals with an expressed interest in the topic.
The consent of the list owners/moderators was not sought since the researcher was an active participant in all four lists and probably known to most subscribers. In the event the research had the full support of the four owners/moderators since the information promised material for further debate and would reveal profiles of subscribers. The research tool used was a questionnaire administered during November 1998. The questionnaire could have been sent individually to subscribers or sent as a single message to the entire subscriber set. It was decided to use the latter method since user-names were not available for Vanguard. The questionnaire employed was a shortened version of one which was also sent as a self-completion pen/paper version by regular postal services to 50 people known to be interested in graphology. Therefore the progress of the e-mail questionnaire could be compared directly with the 'snail-mail' version.
The Questionnaire
The questionnaire was made up of nine questions: Classification data (country, gender, age), interest in graphology (years of interest, how first heard of), image batteries (perceived image of graphologists among various groups and perceived competence levels of graphologists) and membership to the four groups. The date/time of receipt was also derived from each returned questionnaire. No incentive was offered apart from a promise to discuss the results after the survey.
Respondents were given these instructions:
1. To type in the answer (a number and some text OR to delete answers that don't apply).
2. Not to send back to their Discussion List. (It was stated that numerous replies would irritate other people and views may influence other people).
3. Not to discuss the questions on-line since it can influence views.
4. Not to complete more than one questionnaire.
A short message was sent in English to all four lists several days before the actual mailing. This was intended to forewarn arrival of the survey and to make the instructions clear. On Day 3 of the Survey a similar short message was sent as a 'reminder', this time in English, French and Italian. This message offered a French and Italian version of the questionnaire. Reminder questionnaires were sent. List 1 and 2 in English, List 3 in Italian and List 4 in French. They were sent respectively on Day 17, Day 19 and Day 18 of the Survey.
Findings
Tables 2 and 3 report the overall response rate and response speeds. It should be noted that the reminder mailings have been incorporated into the data. It is also relevant to point out that the analyses carry differing sample sizes. Table 2 contains information on the total sample size to show the actual yield for each sampling frame used. In contrast Table 3 excludes respondents who returned questionnaires by a different medium from the one used for despatch. Table 3 therefore shows the actual response speed for each sample source.
H1 is accepted since the response rate of 26% for the non-Internet sample is below the rate achieved for three of the four Internet sampling frames.
H2 is also accepted since the response speed of 262 hours for the non-Internet sample is slower than the speed evident in the sample recruited on the Internet.
Table 4 shows the mean age profiles of respondents who were recruited using the four Discussion Groups. It is compared with the mean age of users of the Internet derived from research conducted by Forrester Research Inc. in 1998 (Kumar et al 1999:164). On the basis of this table, H3 is rejected since the age-profile of respondents is not consistent with known age-profiles of Internet users.
Other Findings
Several other observations can be made as a result of conducting this study. These observations are qualitative in nature and have been listed in Figures 1 - 3.
Figure 1. Sampling Frame Observations
1. Subscribers can subscribe to all four lists. A certain amount of overlap was detected.
2. Some people were subscribed twice or three times, sometimes under different e-mail addresses.
3. Some subscribers did not want to be subscribed, but could not remove themselves.
4. There was evidence that mail sent to at least 3 subscribers "bounced" regularly.
Figure 2. Questionnaire Related
1. A six point scale was answered by several people using a zero (the instruction was to answer 1 - 6).
2. Closed questions and scales were answered by several people as open-ended questions.
3. Sometimes questions were answered by deleting inappropriate pre-coded answers.
4. Sometimes questions were answered by typing X beside the answer. Other times the word Yes or No was used to answer questions.
5. Some questions were not answered at all (easily resolved by an immediate reply).
6. One respondent had set an Automatic 'out-of-office' reply that was sent to all group members.
7. Some replies were sent twice, some identical.
8. One respondent sent two slightly different questionnaires
Reply Related
1. One person sent their replies back to the main list. This placed their answers in full view of other potential respondents.
2. At least 2 respondents sent a reply, but no message was detected.
3. One reply was only detectable by opening a web-browser.
4. One reply included a message but could not be opened.
5. One reply was sent via another person.
6. Three replies were received back from non-list subscribers, apparently forwarded by a subscriber to friends.
7. One reply to the Internet posting was returned as a print out by regular post.
8. Three replies to the regular-post mailing were sent back by e-mail.
Discussion
The study shows that Internet Discussion Groups do offer a feasible means by which an e-mail questionnaire can be administered. Whilst Internet response rates appear to be better than a regular mail questionnaire, there are noticeable differences between lists, varying from 25-54%. Response speed similarly differs between lists, varying from 34-51 hours, however much faster than the regular mail study mean of 262 hours (just over 10 days).
An interesting finding comes from the fact that the average-age of the Internet Graphology-Discussion list subscriber is 52 years compared with the 39 years average for Internet users provided by Forrester Research inc. This could be a real finding and suggest that Internet Graphologists are older than the average user. Another explanation could be that non-respondents are younger, in other words that older people have the time, courtesy or pre-disposition to reply. This clearly has wider implications when considering how representative respondents are of their sample source.
Whilst the study did not seek to examine non-responders, the following point is worth making. Every discussion group has a certain number of people who have subscribed but are not actively participating, they may simply be reading responses of other people and are known as "lurkers". It is surmised that a certain proportion of non-responders are "lurkers". The study did uncover the fact that some list-members did not want to be on the list (some messages bounce back, some subscribers cannot 'remove themselves').
The experience of analysing e-mail questionnaires was also enlightening particularly when considering the use of automatic data entry. An operator needs to intervene since respondents are able to cut/paste, delete and modify questionnaires in a way that is not possible with pen/paper or even web-based questionnaires. The method used in this study was to standardise replies by transferring all messages, by hand, into paper questionnaires. This problem was confirmed by a leading UK Internet research practitioner (Comley 1998),who stated that even with specialist software there can be 10 per cent or more "failed reads". Whilst some respondents are very competent computer operators, others are not, evidenced by the return of one reply by regular post.
This study ran the risk of failing completely. If several respondents had sent their replies to the main list, they could have influenced responses from other list members, the irritation may also have led to greater refusal rates and at the extreme "flaming".
Recommendations
Discussion Group lists offer a ready-made sampling frame of individuals with a stated interest in a particular subject-area. However, administration of a questionnaire to the list itself runs the risk of a study running against problems related to non-co-operation and biased response. The study has uncovered that e-mail addresses can be obtained freely or with the agreement of moderators. These can then be 'de-duplicated', and a selection can be made. A questionnaire can then be administered at an individual level. The questionnaire can carry clear instructions that it should not be altered. The speed of response is such that remedial action can be taken to rectify any problems that may emerge from a pilot. This approach would therefore provide a valuable way to provide sample.
References
BRADLEY, Nigel (1996). A Multi-Lingual Dictionary of Graphology (N R Bradley, Chesterfield).
COMLEY, Peter (1998). Intranet Surveys. (SRA Seminar: Research on the Internet 23 October 1998)
KUMAR V, AAKER D A, DAY G.S. (1999). Essentials of Marketing Research (John Wiley & Sons Inc., New York)
SCHILLEWAERT, Niels, LANGERAK, Fred and DUHAMEL, Tim (1998). Non-probability sampling for www surveys: a comparison of methods. (Journal of the Market Research Society, 40, pp.307-323).
TALMAGE, Philip (Compiler) (1988). Dictionary of Market Research (MRS/ISBA, London).
TSE, A et al (1994). A comparison of the effectiveness of mail and facsimile as survey media on response rate, speed and quality. (Journal of the Market Research Society 36, pp. 349-55).
WITMER, D F, COLMAN RW, KATZMAN SL. From Paper-and-Pencil to Screen-and-Keyboard: Toward a Methodology for Survey Research on the Internet.
(Reproduced on pp145-162 in Jones S (ed) &endash; Doing Internet Research. Critical Issues and Methods for Examining the Net (Sage, California 1999)
NRBHOME / NRBWORKS / UNIVERSITY OF WESTMINSTER
For any corrections to information on this page contact bradlen@wmin.ac.uk