Chapter Measures of interrater agreement when each target is evaluated by a different group of raters
Most measures of interrater agreement are defined for ratings regarding a group of targets, each rated by the same group of raters (e.g., the agreement of raters who assess on a rating scale the language proficiency of a corpus of argumentative written texts). However, there are situations in which...
-д хадгалсан:
| Үндсэн зохиолч: | |
|---|---|
| Формат: | Online |
| Хэл сонгох: | англи |
| Хэвлэсэн: |
Firenze University Press, Genova University Press
2023
|
| Нөхцлүүд: | |
| Онлайн хандалт: | ONIX_20230803_9791221501063_96 |
| Шошгууд: |
Шошго байхгүй, Энэхүү баримтыг шошголох эхний хүн болох!
|
| _version_ | 1869519185508302848 |
|---|---|
| author | BOVE, Giuseppe |
| author_browse | BOVE, Giuseppe |
| author_facet | BOVE, Giuseppe |
| author_sort | BOVE, Giuseppe |
| collection | Directory of Open Access Books |
| description | Most measures of interrater agreement are defined for ratings regarding a group of targets, each rated by the same group of raters (e.g., the agreement of raters who assess on a rating scale the language proficiency of a corpus of argumentative written texts). However, there are situations in which agreement between ratings regards a group of targets where each target is evaluated by a different group of raters, like for instance when teachers in a school are evaluated by a questionnaire administered to all the pupils (students) in the classroom. In these situations, a first approach is to evaluate the level of agreement for the whole group of targets by the ANOVA one-way random model. A second approach is to apply subject-specific indices of interrater agreement like rWG, which represents the observed variance in ratings compared to the variance of a theoretical distribution representing no agreement (i.e., the null distribution). Both these approaches are not appropriate for ordinal or nominal scales. In this paper, an index is proposed to evaluate the agreement between raters for each single target (subject or object) on an ordinal scale, and to obtain also a global measure of the interrater agreement for the whole group of cases evaluated. The index is not affected by the possible concentration of ratings on a very small number of levels of the scale, like it happens for the measures based on the ANOVA approach, and it does not depend on the definition of a null distributions like rWG. The main features of the proposal will be illustrated in a study for the assessment of learning teacher behavior in classroom collected in a research conducted in 2018 at Roma Tre University. |
| format | Online |
| id | doab-20.500.12854ir-112079 |
| institution | Directory of Open Access Books |
| language | eng |
| publishDate | 2023 |
| publishDateRange | 2023 |
| publishDateSort | 2023 |
| publisher | Firenze University Press, Genova University Press |
| publisherStr | Firenze University Press, Genova University Press |
| record_format | ojs |
| spelling | doab-20.500.12854ir-1120792025-07-17T10:01:34Z Chapter Measures of interrater agreement when each target is evaluated by a different group of raters BOVE, Giuseppe Interrater agreement Ordinal data Teacher evaluation thema EDItEUR::J Society and Social Sciences thema EDItEUR::J Society and Social Sciences Most measures of interrater agreement are defined for ratings regarding a group of targets, each rated by the same group of raters (e.g., the agreement of raters who assess on a rating scale the language proficiency of a corpus of argumentative written texts). However, there are situations in which agreement between ratings regards a group of targets where each target is evaluated by a different group of raters, like for instance when teachers in a school are evaluated by a questionnaire administered to all the pupils (students) in the classroom. In these situations, a first approach is to evaluate the level of agreement for the whole group of targets by the ANOVA one-way random model. A second approach is to apply subject-specific indices of interrater agreement like rWG, which represents the observed variance in ratings compared to the variance of a theoretical distribution representing no agreement (i.e., the null distribution). Both these approaches are not appropriate for ordinal or nominal scales. In this paper, an index is proposed to evaluate the agreement between raters for each single target (subject or object) on an ordinal scale, and to obtain also a global measure of the interrater agreement for the whole group of cases evaluated. The index is not affected by the possible concentration of ratings on a very small number of levels of the scale, like it happens for the measures based on the ANOVA approach, and it does not depend on the definition of a null distributions like rWG. The main features of the proposal will be illustrated in a study for the assessment of learning teacher behavior in classroom collected in a research conducted in 2018 at Roma Tre University. 2023-08-08T05:06:31Z 2023-08-08T05:06:31Z 2023-08-03T15:06:08Z 2023 chapter ONIX_20230803_9791221501063_96 2704-5846 https://library.oapen.org/handle/20.500.12657/74900 9791221501063 https://directory.doabooks.org/handle/20.500.12854/112079 eng Proceedings e report open access image/png image/jpeg Attribution 4.0 International Attribution 4.0 International https://library.oapen.org/bitstream/20.500.12657/74900/1/9791221501063-28.pdf https://library.oapen.org/bitstream/20.500.12657/74900/1/9791221501063-28.pdf Firenze University Press, Genova University Press 10.36253/979-12-215-0106-3.28 10.36253/979-12-215-0106-3.28 74113d79-2268-4658-88bb-6e8757c543b0 ASA 2022 Data-Driven Decision Making 9791221501063 6 Florence open access |
| spellingShingle | Interrater agreement Ordinal data Teacher evaluation thema EDItEUR::J Society and Social Sciences thema EDItEUR::J Society and Social Sciences BOVE, Giuseppe Chapter Measures of interrater agreement when each target is evaluated by a different group of raters |
| title | Chapter Measures of interrater agreement when each target is evaluated by a different group of raters |
| title_full | Chapter Measures of interrater agreement when each target is evaluated by a different group of raters |
| title_fullStr | Chapter Measures of interrater agreement when each target is evaluated by a different group of raters |
| title_full_unstemmed | Chapter Measures of interrater agreement when each target is evaluated by a different group of raters |
| title_short | Chapter Measures of interrater agreement when each target is evaluated by a different group of raters |
| title_sort | chapter measures of interrater agreement when each target is evaluated by a different group of raters |
| topic | Interrater agreement Ordinal data Teacher evaluation thema EDItEUR::J Society and Social Sciences thema EDItEUR::J Society and Social Sciences |
| topic_facet | Interrater agreement Ordinal data Teacher evaluation thema EDItEUR::J Society and Social Sciences thema EDItEUR::J Society and Social Sciences |
| url | ONIX_20230803_9791221501063_96 |
| work_keys_str_mv | AT bovegiuseppe chaptermeasuresofinterrateragreementwheneachtargetisevaluatedbyadifferentgroupofraters |