Datasaur
Search…
Inter-annotator Agreement (IAA)
Data scientists have long used inter-annotator agreement to measure how well multiple annotators can make the same annotation decision for a certain label category or class. Now, this information can be found directly through Datasaur's dashboards. This handy calculation can help determine the clarity and consistent reproducibility of your results.
We use Cohen's Kappa to calculate the agreement while taking into account the possibility of chance agreement. The detailed simulation can be found in this page.
We apply Krippendorff's Alpha the interpretation scales of Kappa.
    Discard will be presented in red.
    Tentative will be presented in orange.
    Good will be presented in green.
✍ Inter-Annotator Agreement only applies to the completed projects. If you have projects in progress in your team dashboard, they will not be considered for the IAA equation until marked as completed.
You can view the Inter-annotator Agreement page by navigating to Analytics under the left sidebar. Please note that only Team Admins can access this page.
Furthermore, you will also be able to view the IAA for specific projects.
In the image above, the Label Set field will show All Label Sets as the default filter. When filtering by label set, the IAA calculation will take into consideration all label sets with the same label set items.

Example

Let's say there are five completed projects in your dashboard. Each project has different label sets.
Project
Label Set Items
NER for News
PER, LOC, GEO
Top 5 Stories
PER, LOC, GEO
Sentiment Analysis
Positive, Negative, Neutral
English POS
NNO, NNP, VB, VBT, VBI, JJ
Basic POS for Beginner
NNO, NNP, VB, VBT, VBI, JJ
Some projects have label sets with the exact same label set items (ex: NER for News and Top 5 Stories). This will determine how many label sets that are used for Inter-Annotator Agreement.
We will combine all projects with the same label set items into one label set. For this sample case, the label sets below will be shown in the Label Sets field.
    NER for News Label Set
    Sentiment Analysis Label Set
    English POS Label Set
Now, we can filter by label set.
Last modified 3mo ago
Copy link
Contents
Example