Dataset describing the Altman's Benchmarking Scale

Description

This dataset contains information describing the Altman scale for benchmarking chance-corrected agreement coefficients such as Gwet AC1/AC2, Kappa and many others.

Usage

altman

Format

Each row of this dataset describes an interval and the interpretation of the magnitude it represents.

lb.AL

The interval lower bound

ub.AL

The interval upper bound

interp.AL

The interpretation

Source

Altman, D.G. (1991). Practical Statistics for Medical Research. Chapman and Hall.

Computing Altman's Benchmark Scale Membership Probabilities

Description

Computing Altman's Benchmark Scale Membership Probabilities

Usage

altman.bf(coeff, se, BenchDF = altman)

Arguments

Value

A one-column matrix containing the membership probabilities (c.f. http://agreestat.com/research_papers/inter-rater%20reliability%20study%20design1.pdf)

Function for computing the Bipolar Weights

Description

Function for computing the Bipolar Weights

Usage

bipolar.weights(categ)

Arguments

Value

A square matrix of quadratic weights to be used for calculating the weighted coefficients.

Brennan-Prediger's agreement coefficient among multiple raters (2, 3, +) when the input dataset is the distribution of raters by subject and category.

Description

Brennan-Prediger's agreement coefficient among multiple raters (2, 3, +) when the input dataset is the distribution of raters by subject and category.

Usage

bp.coeff.dist(ratings, weights = "unweighted", categ = NULL,
  conflev = 0.95, N = Inf)

Arguments

Value

A vector containing the following information: pa(the percent agreement),pe(the percent chance agreement),coeff(Brennan-Prediger coefficient), stderr(the standard error of Brennan-Prediger coefficient),conf.int(the p-value of Brennan-Prediger coefficient), p.value(the p-value of Brennan-Prediger coefficient),coeff.name ("Brennan-Prediger").

Source

Brennan, R.L., and Prediger, D. J. (1981). “Coefficient Kappa: some uses, misuses, and alternatives," Educational and Psychological Measurement, 41, 687-699.

Examples

#The dataset "distrib.6raters" comes with this package. It represents the distribution of 6 raters 
#by subject and by category. Note that each row of this dataset sums to the number of raters, which
#is 6. You may this dataset as follows:
distrib.6raters
bp.coeff.dist(distrib.6raters) #BP coefficient, precision measures, weights & list of categories
bp <- bp.coeff.dist(distrib.6raters)$coeff #Yields Brennan-Prediger coefficient alone.
bp
q <- ncol(distrib.6raters) #Number of categories
bp.coeff.dist(distrib.6raters,weights = quadratic.weights(1:q)) #Weighted BP with quadratic weights

Brennan \& Prediger's (BP) agreement coefficient for an arbitrary number of raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Description

Brennan \& Prediger's (BP) agreement coefficient for an arbitrary number of raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Usage

bp.coeff.raw(ratings, weights = "unweighted", categ.labels = NULL,
  conflev = 0.95, N = Inf)

Arguments

Value

A data list containing 3 objects: (1) a one-row data frame containing various statistics including the requested agreement coefficient, (2) the weight matrix used in the calculations if any, and (3) A vector of categories used in the analysis. These could be categories reported by the raters, or those available to the raters whether they used them or not. The output data frame contains the following variables: "coeff.name" (coefficient name), "pa" (the percent agreement), "pe" (the percent chance agreement), coeff.val (Brennan-Prediger coefficient estimate), "coeff.se" (standard error), "conf.int" (the confidence interval), "p.value"(Brennan-Prediger coefficient's p-value), "w.name"(the weights' identification).

References

Brennan, R.L., \& Prediger, D. J. (1981). “Coefficient Kappa: some uses, misuses, and alternatives." Educational and Psychological Measurement, 41, 687-699.

Examples

#The dataset "cac.raw4raters" comes with this package. Analyze it as follows:
cac.raw4raters
bp.coeff.raw(cac.raw4raters) #BP coefficient, precision measures, weights & categories
bp.coeff.raw(cac.raw4raters)$est #Brennan-Prediger coefficient with precision measures
bp <- bp.coeff.raw(cac.raw4raters)$est$coeff.val #Yields Brennan-Prediger coefficient alone.
bp
bp.coeff.raw(cac.raw4raters, weights = "quadratic") #weighted Brennan-Prediger coefficient

Brenann-Prediger coefficient for 2 raters

Description

Brenann-Prediger coefficient for 2 raters

Usage

bp2.table(ratings, weights = identity.weights(1:ncol(ratings)),
  conflev = 0.95, N = Inf)

Arguments

Value

A data frame containing the following 5 variables: coeff.name coeff.val coeff.se coeff.ci coeff.pval.

Examples

#The dataset "cont3x3abstractors" comes with this package. Analyze it as follows:
bp2.table(cont3x3abstractors) #Yields Brennan-Prediger's coefficient along with precision measures
bp <- bp2.table(cont3x3abstractors)$coeff.val #Yields Brennan-Prediger coefficient alone.
bp
q <- nrow(cont3x3abstractors) #Number of categories
bp2.table(cont3x3abstractors,weights = quadratic.weights(1:q)) #Weighted BP coefficient

Ratings of 12 units from 2 raters named Ben and Gerry

Description

This dataset contains ratings that 2 raters named Ben and Gerry assigned to 12 units distributed in 2 groups "G1" and "G2".

Usage

cac.ben.gerry

Format

Each row of this dataset describes an interval and the interpretation of the magnitude it represents.

Group

Group Name

Units

Unit number

Ben

Ben's Ratings

Gerry

Gerry's Ratings

The first 2 columns "Group" and "Units" play a descriptive role here and are not used by any fucntion included in this package. One will typically use cac.ben.gerry[,c(3,4)] or cac.ben.gerry[,c("Ben","Gerry")] as input dataset.

Distribution of 4 raters by subject and by category, for 14 Subjects that belong to 2 groups "G1" and "G2"

Description

This dataset contains rating data in the form of a subject-level distribution of 4 raters by category the subject was classified into. A total of 4 raters had to classify 14 subjects into one of 5 categories "a", "b", "c", "d", and "e". This dataset is different version of the more detailed cac.raw.g1g2 dataset. While cac.raw.g1g2 tells you about the exact category into which each rater classified all subjects, cac.dist.g1g2 on the other hand, can only tell you how many raters classified a given subject into a particular category.

Usage

cac.dist.g1g2

Format

This dataset contains ratings obtained from an experiment where 4 raters classified 14 subjects into 5 possible categories labeled as a, b, c, d, and e. None of the 4 raters scored all 14 units. Therefore, some missing ratings appear in each of the columns associated with the 4 raters.

Note that only the the 4 last columns are to be used with the functions included in this package. The first 2 columns only play a descriptive role and are not used in any calculation.

Group

This variable represents the group name.

Units

This variable represents the unit number.

a

Number of raters who classified the subject represented by the row into category "a"

b

Number of raters who classified the subject represented by the row into category "b"

c

Number of raters who classified the subject represented by the row into category "c"

d

Number of raters who classified the subject represented by the row into category "d"

e

Number of raters who classified the subject represented by the row into category "e"

Distribution of 4 raters by Category and Subject - Subjects allocated in 2 groups A and B.

Description

This dataset summarizes the ratings assigned by 4 raters who classified 15 subjects into one of 3 categories named "a", "b", and "c".

Usage

cac.dist4cat

Format

This dataset has 15 rows (for the 15 subjects) and 4 columns. Only the last 3 columns representing the categories into which subjects are classified are used in the calculations - unless the sub-group analysis is required.

Group

This variable repsents the subject number.

a

category a

b

Category b

c

Category c

Dataset of raw ratings from 4 Raters on 14 Subjects that belong to 2 groups named "G1" and "G2"

Description

This dataset contains data from a reliability experiment where 4 raters identified as Rater1, Rater2, Rater3 and Rater4 scored 14 units on a 5-point alphabetical scale based on the values a, b, c, d and e. These 14 units are allocated to 2 groups named G1 and G2.

Usage

cac.raw.g1g2

Format

This dataset contains ratings obtained from an experiment where 4 raters classified 14 subjects into 5 possible categories labeled as a, b, c, d, and e. None of the 4 raters scored all 14 units. Therefore, some missing ratings appear in each of the columns associated with the 4 raters.

Note that only the the 4 last columns are to be used with the functions included in this package. The first 2 columns only play a descriptive role and are not used in any calculation.

Group

This variable repsents the unit number.

Units

This variable repsents the unit number.

Rater1

All ratings from rater 1

Rater2

All ratings from rater 2

Rater3

All ratings from rater 3

Rater4

All ratings from rater 4

Rating Data from 4 Raters and 15 human Subjects, 9 of whom are female and 6 males.

Description

This dataset contains data from a reliability experiment where 4 raters scored 15 units on a 3-point alphabetic scale based on the values a, b, and c.

Usage

cac.raw.gender

Format

This dataset contains ratings obtained from an experiment where 4 raters classiffied 15 subjects into 3 possible categories labeled as a, b, and c.

Note that only the the 4 last columns are to be used with the functions included in this package. The first column only plays a descriptive role and is not to be used in any calculation.

Group

This variable repsents the unit number.

RaterA

All ratings from rater 1

RaterB

All ratings from rater 2

RaterC

All ratings from rater 3

RaterD

All ratings from rater 4

Rating Data from 4 Raters and 12 Subjects.

Description

This dataset contains data from a reliability experiment where 5 observers scored 15 units on a 4-point numeric scale based on the values 0, 1, 2 and 3.

Usage

cac.raw4raters

Format

This dataset contains ratings obtained from an experiment where 4 raters classified 12 subjects into 5 possible categories labeled as 1, 2, 3, 4, and 5. None of the 4 raters scored all 12 units. Therefore, some missing ratings in the form of "NA" appear in each of the columns associated with the 4 raters.

Note that only the the 4 last columns are to be used with the functions included in this package. The first column only plays a descriptive role and is not used in any calculation.

Units

This variable repsents the unit number.

Rater1

All ratings from rater 1

Rater2

All ratings from rater 2

Rater3

All ratings from rater 3

Rater4

All ratings from rater 4

Source

Gwet, K.L. (2014) Handbook of Inter-Rater Reliability, 4th Edition, page #120. Advanced Analytics, LLC.

Scores assigned by 5 observers to 20 experimental units.

Description

This dataset contains data from a reliability experiment where 5 observers scored 15 units on a 4-point numeric scale based on the values 0, 1, 2 and 3.

Usage

cac.raw5obser

Format

This dataset has 15 rows (for the 15 subjects) and 6 columns. Only the last 5 columns associated with the 5 observers are used in the calculations. Of the 5 observers, only observer 3 scored all 15 units. Therefore, some missing ratings in the form of "NA" appear in the columns associated with the remaining 4 observers.

Unit

This variable repsents the unit number.

Observer1

All ratings from Observer 1

Observer2

All ratings from Observer 2

Observer3

All ratings from Observer 3

Observer4

All ratings from Observer 4

Observer5

All ratings from Observer 5

Source

Gwet, K.L. (2014) Handbook of Inter-Rater Reliability, 4th Edition. Advanced Analytics, LLC. A larger version of this table can be found on page #125

Function for computing the Circular Weights

Description

Function for computing the Circular Weights

Usage

circular.weights(categ)

Arguments

Value

A square matrix of quadratic weights to be used for calculating the weighted coefficients.

Conger's generalized kappa coefficient for an arbitrary number of raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Description

Conger's generalized kappa coefficient for an arbitrary number of raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Usage

conger.kappa.raw(ratings, weights = "unweighted", categ.labels = NULL,
  conflev = 0.95, N = Inf)

Arguments

Value

A data list containing 3 objects: (1) a one-row data frame containing various statistics including the requested agreement coefficient, (2) the weight matrix used in the calculations if any, and (3) A vector of categories used in the analysis. These could be categories reported by the raters, or those available to the raters whether they used them or not. The output data frame contains the following variables: "coeff.name" (coefficient name), "pa" (the percent agreement), "pe" (the percent chance agreement), coeff.val (Conger's Kappa estimate), "coeff.se" (standard error), "conf.int" (Conger Kappa's confidence interval), "p.value"(agreement coefficient's p-value), "w.name"(the weights' identification).

References

Conger, A. J. (1980), “Integration and Generalization of Kappas for Multiple Raters," Psychological Bulletin, 88, 322-328.

Examples

#The dataset "cac.raw4raters" comes with this package. Analyze it as follows:
cac.raw4raters
conger.kappa.raw(cac.raw4raters) #Conger's kappa, precision stats, weights & categories
conger.kappa.raw(cac.raw4raters)$est #Conger's kappa with precision measures
conger <- conger.kappa.raw(cac.raw4raters)$est$coeff.val #Yields Conger's kappa alone.
conger
conger.kappa.raw(cac.raw4raters, weights = "quadratic") #weighted Conger's kappa

Distribution of 100 pregnant women by pregnancy type and by abstractor.

Description

This dataset contains pregnancy type data collected from 100 women who entered an Emergency Room with a positive pregnancy test and a second condition, which is either abdominal pain or vaginal bleeding. After reviewing their medical records, 2 reviewers (also referred to as abstractors) classified them into one of the following three pregnancy categories: Ectopic Pregnancy (Ectopic), Abnormal Intrauterine pregnancy (AIU) and Normal Intrauterine Pregnancy (NIU).

Usage

cont3x3abstractors

Format

Each row of this dataset describes an interval and the interpretation of the magnitude it represents.

Type

Pregnancy Type. This variable is shown here for information only and is never used by any function in the irrCAC package.

Ectopic

Ectopic Pregnancy

AIU

Abnormal Intrauterine Pregnancy

NIU

Normal Intrauterine Pregnancy

Source

Gwet, K.L. (2014). Handbook of Inter-Rater Reliability, 4th Edition. Advanced Analytics, LLC.

Distribution of 223 Psychiatric Patients by Type of of Psychiatric Disorder and Diagnosis Method.

Description

This dataset shows the distribution of 223 psychiatric patients by diagnosis category and by the method used to obtain the diagnosis. The first method named “Clinical Diagnosis" (also known as “Facility Diagnosis") is used in a service facility (e.g. public hospital, or a community unit) and does not rely on a rigorous application of research criteria. The second method known as “Research Diagnosis" is based on a strict application of research criteria. Column 1 contains the diagnosis categories into which patients are classified with Method 1. The first row on the other hand, shows categories into which patients are classified with Method 2.

Usage

cont4x4diagnosis

Format

This dataset contains a 4x4 squared table. The first column is never used in the calculations and only contains row names. Only the last 4 columns are used for computing agreement coefficients.

Diagnosis

Pregnancy Type. This variable is shown here for information only and is never used by any function in the irrCAC package.

Schizophrenia

Ectopic Pregnancy

Bipolar.Disorder

Abnormal Intrauterine Pregnancy

Depression

Normal Intrauterine Pregnancy

Other

Normal Intrauterine Pregnancy

Source

Gwet, K.L. (2014). Handbook of Inter-Rater Reliability, 4th Edition. Advanced Analytics, LLC.

Distribution of 6 psychiatrists by Subject/patient and diagnosis Category.

Description

This dataset summarizes the ratings assigned by 6 psychiatrists classifying 15 patients into one of five categories named "Depression", "Personal Disorder", "Schizophrenia", "Neurosis" and "Other".

Usage

distrib.6raters

Format

This dataset has 15 rows (for the 15 subjects) and 7 columns. Only the last 6 columns representing the categories into which subjects are classified are used in the calculations.

Subject

This variable repsents the subject number.

Personality.Disorder

Personality disorder category

Schizophrenia

Schizophrenia Category

Neurosis

Neurosis category

Other

"Other" category

Source

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters, Psychological Bulletin, 76, 378-382.

Dataset describing Fleiss' Benchmarking Scale

Description

This dataset contains information describing Fleiss' scale for benchmarking chance-corrected agreement coefficients such as Gwet AC1/AC2, Kappa and many others.

Usage

fleiss

Format

Each row of this dataset describes an interval and the interpretation of the magnitude it represents.

lb.FL

The interval lower bound

ub.FL

The interval upper bound

interp.FL

The interpretation

Source

Fleiss, J. L. (1981). Statistical Methods for Rates and Proportions. John Wiley & Sons.

Computing Fleiss Benchmark Scale Membership Probabilities

Description

Computing Fleiss Benchmark Scale Membership Probabilities

Usage

fleiss.bf(coeff, se, BenchDF = fleiss)

Arguments

Value

A one-column matrix containing the membership probabilities (c.f. http://agreestat.com/research_papers/inter-rater%20reliability%20study%20design1.pdf)

Fleiss' agreement coefficient among multiple raters (2, 3, +) when the input dataset is the distribution of raters by subject and category.

Description

Fleiss' agreement coefficient among multiple raters (2, 3, +) when the input dataset is the distribution of raters by subject and category.

Usage

fleiss.kappa.dist(ratings, weights = "unweighted", categ = NULL,
  conflev = 0.95, N = Inf)

Arguments

Value

A vector containing the following information: pa(the percent agreement),pe(the percent chance agreement),coeff(Fleiss' agreement coefficient), stderr(the standard error of Fleiss' coefficient),conf.int(the confidence interval of Fleiss Kappa coefficient), p.value(the p-value of Fleiss' coefficient),coeff.name ("Fleiss").

Source

Fleiss, J. L. (1981). Statistical Methods for Rates and Proportions. John Wiley & Sons.

Examples

#The dataset "distrib.6raters" comes with this package. It represents the distribution of 6 raters 
#by subject and by category. Note that each row of this dataset sums to the number of raters, which
#is 6. You may this dataset as follows:
distrib.6raters
fleiss.kappa.dist(distrib.6raters) #Fleiss' kappa, precision measures, weights & list of categories
fleiss <- fleiss.kappa.dist(distrib.6raters)$coeff #Yields Fleiss' kappa alone.
fleiss
q <- ncol(distrib.6raters) #Number of categories
fleiss.kappa.dist(distrib.6raters,weights = quadratic.weights(1:q)) #Weighted fleiss/quadratic wts

Fleiss' generalized kappa among multiple raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Description

Fleiss' generalized kappa among multiple raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Usage

fleiss.kappa.raw(ratings, weights = "unweighted", categ.labels = NULL,
  conflev = 0.95, N = Inf)

Arguments

Value

A data list containing 3 objects: (1) a one-row data frame containing various statistics including the requested agreement coefficient, (2) the weight matrix used in the calculations if any, and (3) the categories used in the analysis. These could be categories reported by the raters, or those that were available to the raters whether they used them or not. The output data frame contains the following variables: "coeff.name" (coefficient name-here it will be "Fleiss' Kappa"), "pa" (the percent agreement), "pe" (the percent chance agreement), coeff.val (the agreement coefficient estimate-Fleiss' Kappa), "coeff.se" (the standard error), "conf.int" (Fleiss Kappa's confidence interval), "p.value"(Fleiss Kappa's p-value), "w.name"(the weights' identification).

References

Fleiss, J. L. (1981). Statistical Methods for Rates and Proportions. John Wiley \& Sons.

Examples

#The dataset "cac.raw4raters" comes with this package. Analyze it as follows:
cac.raw4raters
fleiss.kappa.raw(cac.raw4raters) #Fleiss' kappa, precision measures, weights & categories
fleiss.kappa.raw(cac.raw4raters)$est #Yields Fleiss' kappa with precision measures
fleiss <- fleiss.kappa.raw(cac.raw4raters)$est$coeff.val #Yields Fleiss' kappa alone.
fleiss
fleiss.kappa.raw(cac.raw4raters, weights = "quadratic") #weighted Fleiss' kappa/quadratic wts

Gwet's AC1/AC2 agreement coefficient among multiple raters (2, 3, +) when the input dataset is the distribution of raters by subject and category.

Description

Gwet's AC1/AC2 agreement coefficient among multiple raters (2, 3, +) when the input dataset is the distribution of raters by subject and category.

Usage

gwet.ac1.dist(ratings, weights = "unweighted", categ = NULL,
  conflev = 0.95, N = Inf)

Arguments

Value

A vector containing the following information: pa(the percent agreement),pe(the percent chance agreement), coeff(Gwet's AC1 or AC2 dependending on whether weights are used or not),stderr(the standard error of Gwet's coefficient), conf.int(the confidence interval of Gwet's coefficient), p.value(the p-value of Gwet's coefficient),coeff.name (AC1/AC2).

Source

Gwet, K. L. (2008). “Computing inter-rater reliability and its variance in the presence of high agreement," British Journal of Mathematical and Statistical Psychology, 61, 29-48.

Examples

#The dataset "distrib.6raters" comes with this package. It represents the distribution of 6 raters 
#by subject and by category. Note that each row of this dataset sums to the number of raters, which
#is 6. You may this dataset as follows:
distrib.6raters
gwet.ac1.dist(distrib.6raters) #AC1 coefficient, precision measures, weights & list of categories
ac1 <- gwet.ac1.dist(distrib.6raters)$coeff #Yields AC1 coefficient alone.
ac1
q <- ncol(distrib.6raters) #Number of categories
gwet.ac1.dist(distrib.6raters,weights = quadratic.weights(1:q)) #AC2 with quadratic weights

Gwet's AC1/AC2 agreement coefficient among multiple raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Description

Gwet's AC1/AC2 agreement coefficient among multiple raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Usage

gwet.ac1.raw(ratings, weights = "unweighted", categ.labels = NULL,
  conflev = 0.95, N = Inf)

Arguments

Value

A data list containing 3 objects: (1) a one-row data frame containing various statistics including the requested agreement coefficient, (2) the weight matrix used in the calculations if any, and (3) the categories used in the analysis. These could be categories reported by the raters, or those that were available to the raters whether they used them or not. The output data frame contains the following variables: "coeff.name" (coefficient name), "pa" (the percent agreement), "pe" (the percent chance agreement), coeff.val (the agreement coefficient estimate-AC1 or AC2), "coeff.se" (the standard error), "conf.int" (AC1/AC2 confidence interval), "p.value" (Gwet AC1/AC2 p-value), "w.name"(the weights' identification).

References

Gwet, K. L. (2008). “Computing inter-rater reliability and its variance in the presence of high agreement." British Journal of Mathematical and Statistical Psychology, 61, 29-48.

Examples

#The dataset "cac.raw4raters" comes with this package. Analyze it as follows:
cac.raw4raters
gwet.ac1.raw(cac.raw4raters) #AC1 coefficient, precision measures, weights & categories
gwet.ac1.raw(cac.raw4raters)$est #Yields AC1 coefficient with precision measures
ac1 <- gwet.ac1.raw(cac.raw4raters)$est$coeff.val #Yields AC1 coefficient alone.
ac1
gwet.ac1.raw(cac.raw4raters, weights = "quadratic") #AC2 coefficient with quadratic wts

Gwet's AC1/AC2 coefficient for 2 raters

Description

Gwet's AC1/AC2 coefficient for 2 raters

Usage

gwet.ac1.table(ratings, weights = identity.weights(1:ncol(ratings)),
  conflev = 0.95, N = Inf)

Arguments

Value

A data frame containing the following 5 variables: coeff.name coeff.val coeff.se coeff.ci coeff.pval.

Examples

#The dataset "cont3x3abstractors" comes with this package. Analyze it as follows:
gwet.ac1.table(cont3x3abstractors) #Yields AC1 along with precision measures
ac1 <- gwet.ac1.table(cont3x3abstractors)$coeff.val #Yields AC1 coefficient alone.
ac1
q <- nrow(cont3x3abstractors) #Number of categories
gwet.ac1.table(cont3x3abstractors,weights = quadratic.weights(1:q)) #AC2 with quadratic weights

Function for computing the Identity Weights

Description

Function for computing the Identity Weights

Usage

identity.weights(categ)

Arguments

Value

A square matrix of identity weights to be used for calculating the unweighted coefficients.

Kappa coefficient for 2 raters

Description

Kappa coefficient for 2 raters

Usage

kappa2.table(ratings, weights = identity.weights(1:ncol(ratings)),
  conflev = 0.95, N = Inf)

Arguments

Value

A data frame containing the following 5 variables: coeff.name coeff.val coeff.se coeff.ci coeff.pval.

Examples

#The dataset "cont3x3abstractors" comes with this package. Analyze it as follows:
kappa2.table(cont3x3abstractors) #Yields Cohen's kappa along with precision measures
kappa <- kappa2.table(cont3x3abstractors)$coeff.val #Yields Cohen's kappa alone.
kappa
q <- nrow(cont3x3abstractors) #Number of categories
kappa2.table(cont3x3abstractors,weights = quadratic.weights(1:q))#weighted kappa/quadratic wts

Krippendorff's agreement coefficient among multiple raters (2, 3, +) when the input dataset is the distribution of raters by subject and category.

Description

Krippendorff's agreement coefficient among multiple raters (2, 3, +) when the input dataset is the distribution of raters by subject and category.

Usage

krippen.alpha.dist(ratings, weights = "unweighted", categ = NULL,
  conflev = 0.95, N = Inf)

Arguments

Value

A vector containing the following information: pa(the percent agreement),pe(the percent chance agreement),coeff(Krippendorff's alpha), stderr(the standard error of Krippendorff's coefficient),conf.int(the confidence interval of Krippendorff's alpha coefficient), p.value(the p-value of Krippendorff's alpha), coeff.name ("krippen alpha").

Source

Gwet, K. (2014). Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Multiple Raters, 4th Edition. Advanced Analytics, LLC Krippendorff (1970). “Bivariate agreement coefficients for reliability of data," Sociological Methodology,2,139-150 Krippendorff (1980). Content analysis: An introduction to its methodology (2nd ed.), New-bury Park, CA: Sage.

Examples

#The dataset "distrib.6raters" comes with this package. It represents the distribution of 6 raters 
#by subject and by category. Note that each row of this dataset sums to the number of raters, which
#is 6. You may this dataset as follows:
distrib.6raters
krippen.alpha.dist(distrib.6raters) #Krippendorff's alpha, precision measures, weights & categories
alpha <- krippen.alpha.dist(distrib.6raters)$coeff #Yields Krippendorff's alpha coefficient alone.
alpha
q <- ncol(distrib.6raters) #Number of categories
krippen.alpha.dist(distrib.6raters,weights = quadratic.weights(1:q)) #Weighted alpha

Krippendorff's alpha coefficient for an arbitrary number of raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Description

Krippendorff's alpha coefficient for an arbitrary number of raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Usage

krippen.alpha.raw(ratings, weights = "unweighted", categ.labels = NULL,
  conflev = 0.95, N = Inf)

Arguments

Value

A data list containing 3 objects: (1) a one-row data frame containing various statistics including the requested agreement coefficient-in this case, Krippendorff's alpha, (2) the weight matrix used in the calculations if any, and (3) the vector of categories used in the analysis. These could be categories reported by the raters, or those that were available to the raters whether they used them or not. The output data frame contains the following variables: "coeff.name" (coefficient name), "pa" (the percent agreement), "pe" (the percent chance agreement), coeff.val (Krippendorff's alpha estimate), "coeff.se (standard error), conf.int" (Krippendorff alpha's confidence interval),"p.value" (Krippendorff alpha's p-value), "w.name" (the weights' identification).

References

Gwet, K. (2014). Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Multiple Raters, 4th Edition. Advanced Analytics, LLC.

Krippendorff (1970). “Bivariate agreement coefficients for reliability of data." Sociological Methodology,2,139-150.

Krippendorff (1980). Content analysis: An introduction to its methodology (2nd ed.), New-bury Park, CA: Sage.

Examples

#The dataset "cac.raw4raters" comes with this package. Analyze it as follows:
cac.raw4raters
krippen.alpha.raw(cac.raw4raters) #Alpha coeff. , precision measures, weights & categories
krippen.alpha.raw(cac.raw4raters)$est #Krippendorff's alpha with precision measures
alpha <- krippen.alpha.raw(cac.raw4raters)$est$coeff.val #Krippendorff's alpha alone.
alpha
krippen.alpha.raw(cac.raw4raters, weights = "quadratic") #weighted alpha/ quadratic wts

Krippendorff's Alpha coefficient for 2 raters

Description

Krippendorff's Alpha coefficient for 2 raters

Usage

krippen2.table(ratings, weights = identity.weights(1:ncol(ratings)),
  conflev = 0.95, N = Inf)

Arguments

Value

A data frame containing the following 5 variables: coeff.name coeff.val coeff.se coeff.ci coeff.pval.

Examples

#The dataset "cont3x3abstractors" comes with this package. Analyze it as follows:
krippen2.table(cont3x3abstractors) #Krippendorff's alpha along with precision measures
alpha <- krippen2.table(cont3x3abstractors)$coeff.val #Krippendorff's alpha alone.
alpha
q <- nrow(cont3x3abstractors) #Number of categories
krippen2.table(cont3x3abstractors,weights = quadratic.weights(1:q)) #Weighted alpha coefficient

Dataset describing the Landis & Koch Benchmarking Scale

Description

This dataset contains information describing the Landis & Koch scale for benchmarking chance-corrected agreement coefficients such as Gwet AC1/AC2, Kappa and many others.

Usage

landis.koch

Format

Each row of this dataset describes an interval and the interpretation of the magnitude it represents.

lb.LK

The interval lower bound

ub.LK

The interval upper bound

interp.LK

The interpretation

Source

Landis, J.R. & Koch G. (1977). The measurement of observer agreement for categorical data, Biometrics, 33, 159-174.

Computing Landis-Koch Benchmark Scale Membership Probabilities

Description

Computing Landis-Koch Benchmark Scale Membership Probabilities

Usage

landis.koch.bf(coeff, se, BenchDF = landis.koch)

Arguments

Value

A one-column matrix containing the membership probabilities (c.f. http://agreestat.com/research_papers/inter-rater%20reliability%20study%20design1.pdf)

Function for computing the Linear Weights

Description

Function for computing the Linear Weights

Usage

linear.weights(categ)

Arguments

Value

A square matrix of quadratic weights to be used for calculating the weighted coefficients.

Function for computing the Ordinal Weights

Description

Function for computing the Ordinal Weights

Usage

ordinal.weights(categ)

Arguments

Value

A square matrix of quadratic weights to be used for calculating the weighted coefficients.

Percent agreement coefficient among multiple raters (2, 3, +) when the input dataset is the distribution of raters by subject and category.

Description

Percent agreement coefficient among multiple raters (2, 3, +) when the input dataset is the distribution of raters by subject and category.

Usage

pa.coeff.dist(ratings, weights = "unweighted", categ = NULL,
  conflev = 0.95, N = Inf)

Arguments

Value

A vector containing the following information: pa(the percent agreement),pe(the percent chance agreement),coeff(Brennan-Prediger coefficient), stderr(the standard error of Brennan-Prediger coefficient),conf.int(the p-value of Brennan-Prediger coefficient), p.value(the p-value of Brennan-Prediger coefficient),coeff.name ("Brennan-Prediger").

Source

Brennan, R.L., and Prediger, D. J. (1981). “Coefficient Kappa: some uses, misuses, and alternatives," Educational and Psychological Measurement, 41, 687-699.

Examples

#The dataset "distrib.6raters" comes with this package. It represents the distribution of 6 raters 
#by subject and by category. Note that each row of this dataset sums to the number of raters, which
#is 6. You may this dataset as follows:
distrib.6raters
pa.coeff.dist(distrib.6raters) #percent agreement, precision measures, weights& list of categories
pa <- pa.coeff.dist(distrib.6raters)$coeff #Yields the percent agreement coefficient alone.
pa
q <- ncol(distrib.6raters) #Number of categories
pa.coeff.dist(distrib.6raters,weights = quadratic.weights(1:q)) #Weighted percent agreement

Percent agreement among multiple raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Description

Percent agreement among multiple raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Usage

pa.coeff.raw(ratings, weights = "unweighted", categ.labels = NULL,
  conflev = 0.95, N = Inf)

Arguments

Value

A data list containing 3 objects: (1) a one-row data frame containing the estimates, (2) the weight matrix used in the calculations, and (3) the categories used in the analysis. The data frame of estimates contains the following variables "coeff.name" (coefficient name), "pa" (the percent agreement), "pe" (percent chance-agreement-always equals 0), "coeff.val" (agreement coefficient = pa), coeff.se (the percent agreement standard error), "conf.int" (the percent agreement confidence interval), "p.value"(the percent agreement p-value), "w.name"(the weights' identification).

Examples

#The dataset "cac.raw4raters" comes with this package. Analyze it as follows:
cac.raw4raters
pa.coeff.raw(cac.raw4raters) #Percent agreement, precision measures, weights & categories
pa.coeff.raw(cac.raw4raters)$est #Yields percent agreement with precision measures
pa <- pa.coeff.raw(cac.raw4raters)$est$coeff.val #Yields percent agreement alone.
pa
pa.coeff.raw(cac.raw4raters, weights = "quadratic") #weighted percent agreement/quadratic weights

Percent Agreement coefficient for 2 raters

Description

Percent Agreement coefficient for 2 raters

Usage

pa2.table(ratings, weights = identity.weights(1:ncol(ratings)),
  conflev = 0.95, N = Inf)

Arguments

Value

A data frame containing the following 5 variables: coeff.name coeff.val coeff.se coeff.ci coeff.pval.

Examples

#The dataset "cont3x3abstractors" comes with this package. Analyze it as follows:
pa2.table(cont3x3abstractors) #Yields percent agreement along with precision measures
pa <- pa2.table(cont3x3abstractors)$coeff.val #Yields percent agreement alone.
pa
q <- nrow(cont3x3abstractors) #Number of categories
pa2.table(cont3x3abstractors,weights = quadratic.weights(1:q)) #Weighted percent agreement

Function for computing the Quadratic Weights

Description

Function for computing the Quadratic Weights

Usage

quadratic.weights(categ)

Arguments

Value

A square matrix of quadratic weights to be used for calculating the weighted coefficients.

Function for computing the Radical Weights

Description

Function for computing the Radical Weights

Usage

radical.weights(categ)

Arguments

Value

A square matrix of quadratic weights to be used for calculating the weighted coefficients.

Function for computing the Ratio Weights

Description

Function for computing the Ratio Weights

Usage

ratio.weights(categ)

Arguments

Value

A square matrix of quadratic weights to be used for calculating the weighted coefficients.

Scott's coefficient for 2 raters

Description

Scott's coefficient for 2 raters

Usage

scott2.table(ratings, weights = identity.weights(1:ncol(ratings)),
  conflev = 0.95, N = Inf)

Arguments

Value

A data frame containing the following 5 variables: coeff.name coeff.val coeff.se coeff.ci coeff.pval.

Examples

#The dataset "cont3x3abstractors" comes with this package. Analyze it as follows:
scott2.table(cont3x3abstractors) #Yields Scott's Pi coefficient along with precision measures
scott <- scott2.table(cont3x3abstractors)$coeff.val #Yields Scott's coefficient alone.
scott
q <- nrow(cont3x3abstractors) #Number of categories
scott2.table(cont3x3abstractors,weights = quadratic.weights(1:q)) #weighted Scott's coefficient

An r function for trimming leading and trealing blanks

Description

An r function for trimming leading and trealing blanks

Usage

trim(x)

Arguments

Value

A string variable where leading and trealing blanks are trimmed.