Review

Overview

The “Review Extension” is designed to streamline your labeling review process, providing you with a dedicated tool to ensure the accuracy and consistency of your labeled data. Whether you're working on a small dataset or a large-scale project, this feature ensures that no labeling discrepancies go unnoticed.

How to enable it?

You just have to enable it by clicking the Manage Extension button in the right sidebar.

How it works?

This feature helps you review your labeling process. It will show how the labeler works and show the conflicting labels.

Broadly, this extension consists of two main sections:

1. Conflicts

2. Member

Conflicts

Within this section, you will find three key aspects:

1. Document allows you to view the list of uploaded documents for the project and the conflict information.

2. Show work from allows to select a labeler to see their specific work.\

   
3. Conflicts allows you to identify all conflicting answers provided by different labelers.

   1. For projects with multiple files, make sure to specify the file by selecting it from the Document dropdown.

   2. The displayed information varies depending on the project type.

Members

In this section, you can easily access the list of labelers assigned to the project and their labeling status. Each project type has different displayed information.

Span Based Labeling

For the detailed how reviewing process, please refer to this Reviewer Mode page.

Conflicts

The Labeling Conflicts in Span Based labeling will be shown under the show work from dropdown.

It will be separated into three sections:

1. Contents: This will show you the conflicted words that were changed by the labeler, for example:

   1. The original sentence is “the quick brown fox.”

   2. Labeler 1: edit the sentence into “the quick black fox.”

   3. Labeler 2: edit the sentence into “the quick fox.”

2. Spans: This will show you the conflicted spans that were labeled by the labelers.

3. Arrows: This will show you the conflicted arrows that were labeled by the labelers.

Accept or Reject All Conflicted Labels

Previously, managing a large number of unresolved or conflicting labels was time-consuming for Reviewers, as they had to manually accept or reject each label individually. This capability streamlines the process, allowing Reviewers to accept or reject all remaining labels with a single action.

How to: Accept or Reject All

This action can be applied to both Span and Arrow conflicts.

Navigate to the Review extension and then click the "View All" button.

Click "Accept All" or "Reject All" depending on the action you would like to perform.

Important Notes

• Both actions cannot be undone and only apply to labels that are currently visible in the Reviewer’s interface.

• Accepting or rejecting labels will not affect labels associated with a span or arrow that is part of a sentence in conflict. The correct sentence must be selected first by the Reviewer.

Accept All Conflicted Labels

• When the project setting “Spans should have at most one label” is unchecked, all conflicted labels will be accepted.

• When the project setting “Spans should have at most one label” is checked, then:

  • If a conflicted label belongs to a span/arrow that has another different label applied to it, then the label will remain conflicted. These require manual resolution by the Reviewer.

  • If a conflicted label belongs to a span/arrow that has no other different label applied to it, then the label will be accepted.

Reject All Conflicted Labels

All conflicted labels will be rejected.

Members

In the members tab, you can see your labeler’s progress. It will also show how many files they have already labeled.

Row Based Labeling

For the detailed how reviewing process, please refer to this Reviewer Mode page

Conflicts

The Labeling Conflicts in Row Based labeling will be shown under the show work from dropdown.

You can click the View All button to see the Conflicted Answers that occur in the labeling process. And it will show you the list of conflicting labels.

You can also filter the conflicting answers based on the labeler. To do that, you have to select which labeler you want to review in the Show Work From the dropdown.

This will show you the conflicting answer caused by the labeler.

Members

In the members tab, you can see your labeler’s progress.

Document Labeling

For the detailed how reviewing process, please refer to this Reviewer Mode page

Conflicts

To resolve conflicts in Doc Based Labeling you have to go to Document Labeling extension to resolve conflicted answers.

You can see all the labeler’s answers. And you can also resolve the conflicts by choosing the best answer and submitting it.

Members

In the members tab, you can see your labeler’s progress. It will also show how many files they have already labeled.