Krippendorff's alpha

Krippendorff's alpha coefficient[1] is a statistical measure of the agreement achieved when coding a set of units of analysis in terms of the values of a variable. Since the 1970s, alpha is used in content analysis where textual units are categorized by trained readers, in counseling and survey research where experts code open-ended interview data into analyzable terms, in psychological testing where alternative tests of the same phenomena need to be compared, or in observational studies where unstructured happenings are recorded for subsequent analysis.

Krippendorff’s alpha generalizes several known statistics, often called measures of inter-coder agreement, inter-rater reliability, reliability of coding given sets of units (as distinct from unitizing) but it also distinguishes itself from statistics that are called reliability coefficients but are unsuitable to the particulars of coding data generated for subsequent analysis.

Krippendorff’s alpha is applicable to any number of coders, each assigning one value to one unit of analysis, to incomplete (missing) data, to any number of values available for coding a variable, to binary, nominal, ordinal, interval, ratio, polar, and circular metrics (Levels of Measurement), and it adjusts itself to small sample sizes of the reliability data. The virtue of a single coefficient with these variations is that computed reliabilities are comparable across any numbers of coders, values, different metrics, and unequal sample sizes.

Software for calculating Krippendorff’s alpha is available.[2][3][4][5]

Reliability data

Reliability data are generated in a situation in which m ≥ 2 jointly instructed (e.g., by a Code book) but independently working coders assign any one of a set of values 1,...,V to a common set of N units of analysis. In their canonical form, reliability data are tabulated in an m-by-N matrix containing N values vij that coder ci has assigned to unit uj. Define mj as the number of values assigned to unit j across all coders c. When data are incomplete, mj may be less than m. Reliability data require that values be pairable, i.e., mj ≥ 2. The total number of pairable values is nmN.

To help clarify, here is what the canonical form looks like, in the abstract:

u1 u2 u3 ... uN
c1 v11 v12 v13 ... v1N
c2 v21 v22 v23 ... v2N
c3 v31 v32 v33 ... v3N
... ... ... ... ... ...
cm vm1 vm2 vm3 ... vmN

General form of alpha

We denote the set of all possible responses an observer can give, . The responses of all observers for an example is called a unit (it forms a multiset). We denote a multiset with these units as the items, .

Alpha is given by:

where is the disagreement observed and is the disagreement expected by chance.

where is a metric function (see below), is the total number of pairable elements, is the number of items in a unit, number of pairs in unit , and is the permutation function. This can be seen to be the (weighted) average observed distance from the diagonal.

where is the number of ways the pair can be made. This can be seen to be the average distance from the diagonal of all possible pairs of responses that could be derived from the multiset of all observations.

The above is equivalent to the usual form of once it has been simplified algebraically.[6]

One interpretation of Krippendorff's alpha is:

indicates perfect reliability
indicates the absence of reliability. Units and the values assigned to them are statistically unrelated.
when disagreements are systematic and exceed what can be expected by chance.

In this general form, disagreements Do and De may be conceptually transparent but are computationally inefficient. They can be simplified algebraically, especially when expressed in terms of the visually more instructive coincidence matrix representation of the reliability data.

Coincidence matrices

A coincidence matrix cross tabulates the n pairable values from the canonical form of the reliability data into a v-by-v square matrix, where v is the number of values available in a variable. Unlike contingency matrices, familiar in association and correlation statistics, which tabulate pairs of values (Cross tabulation), a coincidence matrix tabulates all pairable values. A coincidence matrix omits references to coders and is symmetrical around its diagonal, which contains all perfect matches, viu = vi'u for two coders i and i' , across all units u. The matrix of observed coincidences contains frequencies:

,
and ,

omitting unpaired values, where I(∘) = 1 if is true, and 0 otherwise.

Because a coincidence matrix tabulates all pairable values and its contents sum to the total n, when four or more coders are involved, ock may be fractions.

The matrix of expected coincidences contains frequencies:

,

which sum to the same nc, nk, and n as does ock. In terms of these coincidences, Krippendorff's alpha becomes:

.

Difference functions

Difference functions [7] between values v and v' reflect the metric properties (Levels of Measurement) of their variable.

In general:

In particular:

For nominal data , where v and v' serve as names.
For ordinal data , where v and v' are ranks.
For interval data , where v and v' are interval scale values.
For ratio data , where v and v' are absolute values.
For polar data , where vmin and vmax define the end points of the polar scale.
For circular data , where the sine function is expressed in degrees and U is the circumference or the range of values in a circle or loop before they repeat. For equal interval circular metrics, the smallest and largest integer values of this metric are adjacent to each other and U = vlargestvsmallest + 1.

Significance

Inasmuch as mathematical statements of the statistical distribution of alpha are always only approximations, it is preferable to obtain alpha’s distribution by bootstrapping.[8][9] Alpha 's distribution gives rise to two indices:

The minimum acceptable alpha coefficient should be chosen according to the importance of the conclusions to be drawn from imperfect data. When the costs of mistaken conclusions are high, the minimum alpha needs to be set high as well. In the absence of knowledge of the risks of drawing false conclusions from unreliable data, social scientists commonly rely on data with reliabilities α ≥ .800, consider data with 0.800 > α ≥ 0.667 only to draw tentative conclusions, and discard data whose agreement measures α < 0.667.[10]

A misunderstanding of Krippendorff's alpha has become an instructive public controversy.[11]

A computational example

Let the canonical form of reliability data be a 3-coder-by-15 unit matrix with 45 cells:

Units u: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Coder A * * * * * 3 4 1 2 1 1 3 3 * 3
Coder B 1 * 2 1 3 3 4 3 * * * * * * *
Coder C * * 2 1 3 4 4 * 2 1 1 3 3 * 4

Suppose “*” indicates a default category like “cannot code,” “no answer,” or “lacking an observation.” Then, * provides no information about the reliability of data in the four values that matter. Note that unit 2 and 14 contains no information and unit 1 contains only one value, which is not pairable within that unit. Thus, these reliability data consist not of mN=45 but of n=26 pairable values, not in N =15 but in 12 multiply coded units.

The coincidence matrix for these data would be constructed as follows:

o11 = {in u=4}: {in u=10}: {in u=11}:
o13 = {in u=8}: o31
o22 = {in u=3}: {in u=9}:
o33 = {in u=5}: {in u=6}: {in u=12}: {in u=13}:
o34 = {in u=6}: {in u=15}: o43
o44 = {in u=7}:
Values v or v' : 1 2 3 4 nv
Value 1 6 1 7
Value 2 4 4
Value 3 1 7 2 10
Value 4 2 3 5
Frequency nv' 7 4 10 5 26

In terms of the entries in this coincidence matrix, Krippendorff's alpha may be calculated from:

.

For convenience, because products with and , only the entries in one of the off-diagonal triangles of the coincidence matrix are listed in the following:

Considering that all when for nominal data the above expression yields:

With , , and , for interval data the above expression yields:

Here, because disagreements happens to occur largely among neighboring values, visualized by occurring closer to the diagonal of the coincidence matrix, a condition that takes into account but does not. When the observed frequencies ov≠ v' are on the average proportional to the expected frequencies ev ≠ v', = .

Comparing alpha coefficients across different metrics can provide clues to how coders conceptualize the metric of a variable.

Alpha's embrace of other statistics

Krippendorff's alpha brings several known statistics under a common umbrella, each of them has its own limitations but no additional virtues.

where , and
When data are nominal, alpha reduces to a form resembling Scott’s pi:
Scott’s observed proportion of agreement appears in alpha’s numerator, exactly. Scott’s expected proportion of agreement, is asymptotically approximated by when the sample size n is large, equal when infinite. It follows that Scott’s pi is that special case of alpha in which two coders generate a very large sample of nominal data. For finite sample sizes: . Evidently, .
where , and
When sample sizes are finite, K can be seen to perpetrate the inconsistency of obtaining the proportion of observed agreements by counting matches within the m(m-1) possible pairs of values within u, properly excluding values paired with themselves, while the proportion is obtained by counting matches within all (mN)2=n2 possible pairs of values, effectively including values paired with themselves. It is the latter that introduces a bias into the coefficient. However, just as for pi, when sample sizes become very large this bias disappears and the proportion in nominalα above asymptotically approximates in K. Nevertheless, Fleiss' kappa, or rather K, intersects with alpha in that special situation in which a fixed number of m coders code all of N units (no data are missing), using nominal categories, and the sample size n=mN is very large, theoretically infinite.
,
where is the sum of N differences between one coder’s rank c and the other coder’s rank k of the same object u. Whereas alpha accounts for tied ranks in terms of their frequencies for all coders, rho averages them in each individual coder's instance. In the absence of ties, 's numerator and 's denominator , where n=2N, which becomes when sample sizes become large. So, Spearman’s rho is that special case of alpha in which two coders rank a very large set of units. Again, and .

Krippendorff's alpha is more general than any of these special purpose coefficients. It adjusts to varying sample sizes and affords comparisons across a wide variety of reliability data, mostly ignored by the familiar measures.

Coefficients incompatible with alpha and the reliability of coding

Semantically, reliability is the ability to rely on something, here on coded data for subsequent analysis. When a sufficiently large number of coders agree perfectly on what they have read or observed, relying on their descriptions is a safe bet. Judgments of this kind hinge on the number of coders duplicating the process and how representative the coded units are of the population of interest. Problems of interpretation arise when agreement is less than perfect, especially when reliability is absent.

Naming a statistic as one of agreement, reproducibility, or reliability does not make it a valid index of whether one can rely on coded data in subsequent decisions. Its mathematical structure must fit the process of coding units into a system of analyzable terms.

Notes

  1. Krippendorff, K. (2013) pp. 221-250 describes the mathematics of alpha and its use in content analysis since 1969.
  2. Hayes, A. F. & Krippendorff, K. (2007) describe and provide SPSS and SAS macros for computing alpha, its confidence limits and the probability of failing to reach a chosen minimum.
  3. Reference manual of the irr package containing the kripp.alpha() function for the platform independent statistics package R
  4. The Alpha resources page.
  5. Matlab code to compute Krippendorff's alpha.
  6. Honour, David. "Understanding Krippendorff's Alpha" (PDF).
  7. Computing Krippendorff’s Alpha Reliability” http://repository.upenn.edu/asc_papers/43/
  8. Krippendorff, K. (2004) pp. 237-238
  9. Hayes, A. F. & Krippendorff, K. (2007) Answering the Call for a Standard Reliability Measure for Coding Data
  10. Krippendorff, K. (2004) pp. 241-243
  11. Brooks, R. “Sweet Jesus I love Bill Reilly!” Los Angeles Times, May 4, 2007, http://www.latimes.com/news/opinion/commentary/la-oe-brooks4may04,0,6548272.column?coll=la-home-commentary; Mitchell, R. “Stop calling O’Reilly names.” Los Angeles Times, May 10, 2007, http://www.latimes.com/news/opinion/la-oew-mitchell9may09,0,3143633.story?coll=la-opinion-center; Conway, M., Grabe, M. E., & Grieves, K. "Bill O'Reilly and Krippendorff's Alpha." Los Angeles Times May 16, 2007, http://www.latimes.com/news/opinion/la-oew-conway16may16,0,3767872.story?coll=la-opinion-center; Convey, M., et al. “Peas in a pod; LA Times op-ed.
  12. Scott, W. A. (1955)
  13. Fleiss, J. L. (1971)
  14. Cohen, J. (1960)
  15. Siegel, S. & Castellan, N. J. (1988), pp. 284-291.
  16. Spearman, C. E. (1904)
  17. Pearson, K. (1901), Tildesley, M. L. (1921)
  18. Krippendorff, K. (1970)
  19. Cohen, J. (1960)
  20. Krippendorff, K. (1978) raised this issue with Joseph Fleiss
  21. Zwick, R. (1988), Brennan, R. L. & Prediger, D. J. (1981), Krippendorff (1978, 2004).
  22. Nunnally, J. C. & Bernstein, I. H. (1994)
  23. Cronbach, L. J. (1951)
  24. Bennett, E. M., Alpert, R. & Goldstein, A. C. (1954)
  25. Goodman, L. A. & Kruskal, W. H. (1954) p. 758
  26. Lin, L. I. (1989)

1. K. Krippendorff, 2013, Content Analysis: An Introduction to Its Methodology, 3rd ed. Thousand Oaks, CA, USA: Sage, PP. 221–250

References

External links

This article is issued from Wikipedia - version of the 8/20/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.