Fleiss, J. L. Measure of nominal scale agreement among many advisors. Psychological Bulletin 1971,76, 378-382. House, A.E., House, B.J. – Campbell, M.B. Measures of interobserver agreement: Calculation formulas and distribution effects. Journal of Behavioral Assessment 3, 37-57 (1981). doi.org/10.1007/BF01321350 1In effect of this calculator based on Chapter 5 (Improving and Assessing the Quality of Behavi measurement; 102-124) by Cooper, Heron and Heward (2007) tested IOA data. For all algorithms, there was a 100% match between the values derived from the IOA with the computer described in this article and those reported in Cooper et al.
This is not a very strict agreement procedure, since a total of 100% could be determined if two observers recorded totally different cases of target responses within the same 15-metre observation. In the data flow example shown in Figure 1, Observer 1 records three target response instances during the 3 m (one per minute) of its observation, two instances per minute 4 and misses all other instances for the remaining 12 meters. During the same hypothetical observation, Observer 2 missed all three instances during 1-3 minutes, recorded a target response instance in minute 4, but recorded four instances in minute 15. Although these are totally different events, the total number of IAOs that would result would still be 100%. Hawkins, R. P., and Dotson, V. A. Reliability scores that delude: An Alice in Wonderland trip through the misleading characteristics of interobserver agreement scores in interval recording. In E. Ramp and G.
Semb (Eds.), behavioral analysis: areas of research and application. N.J.: Prentice Hall, 1975, 539-376. Time-based IOA algorithms evaluate the agreement between the temporal data of two observers. These measures consist of (a) the total duration and (b) the average duration of the incident. Table 3 summarizes the strengths of the two algorithms. Consider as a permanent example of the permanent IOA the hypothetical data flow represented in Figure 3, in which two independent observers recorded the duration of a target response over four deposits. A procedure to improve the credibility of the data by comparing the independent observations of two or more people from the same event. The IOA is calculated by calculating the number of agreements between independent observers and divided by the total number of agreements plus disagreements.
The coefficient is then multiplied by 100 to calculate the percentage (%) Consent.