They originated in the fields of optimal experimental design in statistics (Lindley, 1956 Atkinson & Donev, 1992) and active learning in machine learning (Cohn et al., 1994 Settles, 2009). Its ease of implementation and generally good results have made it a popular method across many fields in psychology.įormal approaches to achieving these same ends (good efficiency and precision) have also been developed. The staircase method is efficient because the general region of the threshold is identified after a relatively small number of trials, after which the remaining trials concentrate on obtaining a precise threshold estimate. The experiment is stopped after a given number of reversals in direction has been observed. Intensity is increased if the stimulus was not detected, decreased if it was. The procedure operates by a simple heuristic rule, of which there are a handful of variants: The stimulus to present on one trial is determined by the response on the previous trial. Stimuli differ along a one-dimensional continuum (intensity). The most widely used one is the staircase procedure for estimating a threshold (Cornsweet, 1962 Feeny et al., 1966 Rose et al., 1970), such as when measuring hearing or visual acuity. Methods of optimizing efficiency and precision have been developed for some experimental paradigms. By optimizing stimulus selection in the design space, efficiency and precision can be balanced. What then is the optimal number of trials that will provide the most precise performance estimates? A partial answer lies in recognizing that not all stimuli are equally informative. However, it may not be efficient to add too many trials, especially with a clinical population where time is really of the essence and when participants can easily get fatigued or bored. Adding more trials can increase precision along with practice effects. Too few trials yield poor precision (low signal-to-noise ratio) there are simply not enough data to make an inference, for or against a prediction, with confidence. How much data must be collected to be confident that differences between conditions could be found? This question is similar to that asked when performing a power analysis, but is focused on the performance of the participant during the experiment itself. This issue is often guided by two competing goals: efficiency and precision. The design space, the stimulus set that arises from decisions about the independent variable (number of variables, number of levels of each variable) is critically important for creating a high-signal experiment.Ī similarly important consideration is the stimulus presentation schedule during the experiment. Because human data always contain various types of noise, researchers need to design experiments so that the signal of interest (experimental manipulations) is amplified while unintended influences from uncontrolled variables (noise) are still present. Scientific discovery is guided in part by statistical inference, and the strength of any inference depends on the quality of the data collected. Simulation data are also provided to demonstrate how ADO designs compare with other designs (random, staircase).Ī main goal of psychological research is to gain knowledge about brain and behavior. In this paper, we first provide a tutorial introduction to ADOpy and ADO itself, and then illustrate its use in three walk-through examples: psychometric function estimation, delay discounting, and risky choice. The package, available on GitHub, is written using high-level modular-based commands such that users do not have to understand the computational details of the ADO algorithm. To increase its accessibility to experimentalists at large, we introduce an open-source Python package, ADOpy, that implements ADO for optimizing experimental design. The nontrivial technical skills required to use ADO have been a barrier to its wider adoption. ADO is a general-purpose method for conducting adaptive experiments on the fly and can lead to rapid accumulation of information about the phenomenon of interest with the fewest number of trials. It works by maximizing the informativeness and efficiency of data collection, thereby improving inference. Adaptive design optimization (ADO Cavagnaro, Myung, Pitt, & Kujala, 2010 Myung, Cavagnaro, & Pitt, 2013) is one such method. Advances in Bayesian statistics and machine learning offer algorithm-based ways to identify good experimental designs. Experimental design is fundamental to research, but formal methods to identify good designs are lacking.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |