Chapter An experimental annotation task to investigate annotators’ subjectivity in a Misogyny dataset
In recent years, hatred directed against women has spread exponentially, especially in online social media. Although this alarming phenomenon has given rise to many studies both from the viewpoint of computational linguistics and from that of machine learning, less effort has been devoted to analysi...
Guardat en:
| Autors principals: | , , , , , |
|---|---|
| Format: | Online |
| Idioma: | anglès |
| Publicat: |
Firenze University Press, Genova University Press
2023
|
| Matèries: | |
| Accés en línia: | ONIX_20230803_9791221501063_117 |
| Etiquetes: |
Sense etiquetes, Sigues el primer a etiquetar aquest registre!
|
| _version_ | 1869515503411658752 |
|---|---|
| author | Tontodimamma, Alice Ignazzi, Elisa Anzani, Stefano Stranisci, Marco Basile, Valerio FONTANELLA, Lara |
| author_browse | Anzani, Stefano Basile, Valerio FONTANELLA, Lara Ignazzi, Elisa Stranisci, Marco Tontodimamma, Alice |
| author_facet | Tontodimamma, Alice Ignazzi, Elisa Anzani, Stefano Stranisci, Marco Basile, Valerio FONTANELLA, Lara |
| author_sort | Tontodimamma, Alice |
| collection | Directory of Open Access Books |
| description | In recent years, hatred directed against women has spread exponentially, especially in online social media. Although this alarming phenomenon has given rise to many studies both from the viewpoint of computational linguistics and from that of machine learning, less effort has been devoted to analysing whether models for the detection of misogyny are affected by bias. An emerging topic that challenges traditional approaches for the creation of corpora is the presence of social bias in natural language processing (NLP). Many NLP tasks are subjective, in the sense that a variety of valid beliefs exist about what the correct data labels should be; some tasks, for example misogyny detection, are highly subjective, as different people have very different views about what should or should not be labelled as misogynous. An increasing number of scholars have proposed strategies for assessing the subjectivity of annotators, in order to reduce bias both in computational resources and in NLP models. In this work, we present two corpora: a corpus of messages posted on Twitter after the liberation of Silvia Romano on the 9th of May, 2020 and corpus of comments constructed starting from posts on Facebook that contained misogyny, developed through an experimental annotation task, to explore annotators’ subjectivity. For a given comment, the annotation procedure consists in selecting one or more chunk from each text that is regarded as misogynistic and establishing whether a gender stereotype is present. Each comment is annotated by at least three annotators in order to better analyse their subjectivity. The annotation process was carried by trainees who are engaged in an internship program. We propose a qualitative-quantitative analysis of the resulting corpus, which may include non-harmonised annotations. |
| format | Online |
| id | doab-20.500.12854ir-112317 |
| institution | Directory of Open Access Books |
| language | eng |
| publishDate | 2023 |
| publishDateRange | 2023 |
| publishDateSort | 2023 |
| publisher | Firenze University Press, Genova University Press |
| publisherStr | Firenze University Press, Genova University Press |
| record_format | ojs |
| spelling | doab-20.500.12854ir-1123172025-07-17T09:59:59Z Chapter An experimental annotation task to investigate annotators’ subjectivity in a Misogyny dataset Tontodimamma, Alice Ignazzi, Elisa Anzani, Stefano Stranisci, Marco Basile, Valerio FONTANELLA, Lara subjectivity misogyny disagreement social bias thema EDItEUR::J Society and Social Sciences thema EDItEUR::J Society and Social Sciences In recent years, hatred directed against women has spread exponentially, especially in online social media. Although this alarming phenomenon has given rise to many studies both from the viewpoint of computational linguistics and from that of machine learning, less effort has been devoted to analysing whether models for the detection of misogyny are affected by bias. An emerging topic that challenges traditional approaches for the creation of corpora is the presence of social bias in natural language processing (NLP). Many NLP tasks are subjective, in the sense that a variety of valid beliefs exist about what the correct data labels should be; some tasks, for example misogyny detection, are highly subjective, as different people have very different views about what should or should not be labelled as misogynous. An increasing number of scholars have proposed strategies for assessing the subjectivity of annotators, in order to reduce bias both in computational resources and in NLP models. In this work, we present two corpora: a corpus of messages posted on Twitter after the liberation of Silvia Romano on the 9th of May, 2020 and corpus of comments constructed starting from posts on Facebook that contained misogyny, developed through an experimental annotation task, to explore annotators’ subjectivity. For a given comment, the annotation procedure consists in selecting one or more chunk from each text that is regarded as misogynistic and establishing whether a gender stereotype is present. Each comment is annotated by at least three annotators in order to better analyse their subjectivity. The annotation process was carried by trainees who are engaged in an internship program. We propose a qualitative-quantitative analysis of the resulting corpus, which may include non-harmonised annotations. 2023-08-08T07:26:18Z 2023-08-08T07:26:18Z 2023-08-03T15:06:52Z 2023 chapter ONIX_20230803_9791221501063_117 2704-5846 https://library.oapen.org/handle/20.500.12657/74921 9791221501063 https://directory.doabooks.org/handle/20.500.12854/112317 eng Proceedings e report open access image/png image/jpeg Attribution 4.0 International Attribution 4.0 International https://library.oapen.org/bitstream/20.500.12657/74921/1/9791221501063-49.pdf https://library.oapen.org/bitstream/20.500.12657/74921/1/9791221501063-49.pdf Firenze University Press, Genova University Press 10.36253/979-12-215-0106-3.49 10.36253/979-12-215-0106-3.49 74113d79-2268-4658-88bb-6e8757c543b0 ASA 2022 Data-Driven Decision Making 9791221501063 6 Florence open access |
| spellingShingle | subjectivity misogyny disagreement social bias thema EDItEUR::J Society and Social Sciences thema EDItEUR::J Society and Social Sciences Tontodimamma, Alice Ignazzi, Elisa Anzani, Stefano Stranisci, Marco Basile, Valerio FONTANELLA, Lara Chapter An experimental annotation task to investigate annotators’ subjectivity in a Misogyny dataset |
| title | Chapter An experimental annotation task to investigate annotators’ subjectivity in a Misogyny dataset |
| title_full | Chapter An experimental annotation task to investigate annotators’ subjectivity in a Misogyny dataset |
| title_fullStr | Chapter An experimental annotation task to investigate annotators’ subjectivity in a Misogyny dataset |
| title_full_unstemmed | Chapter An experimental annotation task to investigate annotators’ subjectivity in a Misogyny dataset |
| title_short | Chapter An experimental annotation task to investigate annotators’ subjectivity in a Misogyny dataset |
| title_sort | chapter an experimental annotation task to investigate annotators subjectivity in a misogyny dataset |
| topic | subjectivity misogyny disagreement social bias thema EDItEUR::J Society and Social Sciences thema EDItEUR::J Society and Social Sciences |
| topic_facet | subjectivity misogyny disagreement social bias thema EDItEUR::J Society and Social Sciences thema EDItEUR::J Society and Social Sciences |
| url | ONIX_20230803_9791221501063_117 |
| work_keys_str_mv | AT tontodimammaalice chapteranexperimentalannotationtasktoinvestigateannotatorssubjectivityinamisogynydataset AT ignazzielisa chapteranexperimentalannotationtasktoinvestigateannotatorssubjectivityinamisogynydataset AT anzanistefano chapteranexperimentalannotationtasktoinvestigateannotatorssubjectivityinamisogynydataset AT straniscimarco chapteranexperimentalannotationtasktoinvestigateannotatorssubjectivityinamisogynydataset AT basilevalerio chapteranexperimentalannotationtasktoinvestigateannotatorssubjectivityinamisogynydataset AT fontanellalara chapteranexperimentalannotationtasktoinvestigateannotatorssubjectivityinamisogynydataset |