Classic forms of experiments in cognitive and social science are the lab experiment and the field experiment. Both forms are distinctive from simple observation in being active ways of gaining knowledge. They come with certain theoretical and practical disadvantages, which limit their power of explanation in a basic way. The new World Wide Web experiment category offers the chance of extending the validity and acceptance of experimental research.
The following is a preliminary view of what is intended to become a comprehensive overview of methodological issues of online experiments - with your help. At the bottom of this page you will find the opportunity to mail a commentary.
A more comprehensive discussion of web experimental methodology has been published in German: Reips, U.-D. (1997). Das psychologische Experimentieren im Internet [Psychological experimenting on the internet]. In B. Batinic (Ed.): Internet für Psychologen (pp. 245-265). Göttingen, Germany: Hogrefe. There is also a Website for this book, which appeared in its second, completely revised edition in 2000.
Also in 2000 appeared Reips, U.-D. (2000). The Web Experiment Method: Advantages, Disadvantages, and Solutions. In M. H. Birnbaum (Ed.), Psychological Experiments on the Internet (pp. 89-118). San Diego, CA: Academic Press. This publication contains a deepened and extended discussion of the points listed below, and describes empirical results, some from our research program on online and offline research methodology. More new and old publications about web experiments and online research. Watch for Dimensions of Internet Science (appearing in May 2001).
Disadvantages of Traditional Forms of Experiments
samples are not representative of the general population (subjects are mostly psychology students, young, mobile, of local nationality)
it is hard to know whether results are specific for the culture where the experiment was conducted
for practical reasons (finances, time, lab space), often N cannot get large enough
control of potentially relevant factors, which might create an ecologically invalid situation (in the lab) or not be possible at all (in the field)
institutional and bureaucratic limitations often create ecologically skewed situations (e.g. lab hours are limited to [non-lunch :-)] daytime during weekdays)
experimenter bias is rarely taken care of
the traditional methodology narrowly limits what CAN be done experimentally (partly to avoid the above issues becoming too relevant) and -worse- what IS done (due to the ease of conducting some experiments and the hardship of conducting others)
Advantages of Internet Experiments
not local (international, sample is not limited)
potentially high N possible
experiment comes to the subject instead of the other way round
no experimenter needs to be present during the experiment
financially attractive (less money for experimenters, subjects, rooms etc.) <-- I bet this will be decisive! ;-)
sometimes the only way to reach a sufficient number of subjects from the target population (e.g. people age 20-30 with diabetes, an IQ higher than 130, and a traumatic crime experience)
detectability of motivational confounding
mostly easier and faster to update during pilot phase
in some ways ecologically more valid
experimenting around the clock - independent of time of day and day of week
completely voluntary participation throughout the whole experiment
Disadvantages of Internet Experiments
less (?) control (several subjects at one address, one subject at many computers)
samples are not representative of the general population - in a different sense (internet users: currently 38.7% female in USA; internet access; refering web site)
self selection
technical variance (different computers, monitors, browsers, net connections) <-- this can also be an advantage, because the result of an experiment is less likely due to a technical artefact in one apparatus
motivation? Drop out is much more likely
individual questions regarding the comprehension of the material (subject --> experimenter) are not possible
Conclusion
Internet Experiments are applicable in many research areas, especially those in which a perfectly controlled lab situation is not necessary.
Some of the possible disadvantages of this new method can be avoided by taking appropriate measures, e.g. avoiding data contamination from multiple participation by analyzing only the first data set from one IP or limiting drop out during the experimental phase by having subjects go through a "warm up".
The advantages of having a large world wide pool of subjects always right at one's fingertips are extremely attractive. For the first time in the history of social science it seems possible to overcome some of the essential objections against the traditional experiment. Data from the emerging field of Web Experimentation might help us to be able to generalize much better than ever across demographic, cultural and numerical boundaries - and this while saving money.
Therefore, have no fear of internet experiments:
Commentaries? Additions?
My view of the opportunities of Web Experimentation are quite subjective, perhaps especially in their optimism. Hence, I would be glad to receive your suggestions and commentaries, which I might put up here as a whole or in parts. (If you are generally interested in discussing Web Research Methodology, please subscribe to the "Psycresearch-online" mailing list by typing
subscribe psycresearch-online <your email address here>
in the body of the email message). If you send mail with comments from this page, please indicate if you do NOT want them to appear on the comments page.
Send e-mail by clicking the following button:
