Dynamic Review Capabilities

Allows you to assign another person to review your project automatically

Dynamic review is a feature that allows you to assign your team member as reviewer automatically when the labelers have conflicts in a project. This feature is best suited to small teams that only have a few of labelers available. The individual that you assign as the dynamic reviewer will receive an email notification if a conflict does occur in the project.

How do we enable Dynamic Review?

Dynamic review is available on Step 4: Assignment. Assign at least 2 labelers with the consensus 2. Why at least 2 labelers with a consensus of 2? This way there is a chance of conflict occurring for the dynamic reviewer to QA.

Check “Allow dynamic review assignment” option. Choose one of the options to assign the reviewer.

• Any other team member

  • It will select one of your team members, except those who have been assigned as labelers.

• Any reviewer

  • It will select one of the reviewers in your team.

• Dedicated member

  • Select a particular member as a reviewer.

Two Possible Outcomes:

No conflicts

1. Admin assign at least 2 labelers to the project.

2. The labelers start labeling.

3. The labelers have completed the project (Mark as Completed).

4. There are no conflicts available.

5. The project status will be updated automatically as completed.

Conflicts

1. Admin assign at least 2 labelers to the project.

2. The labelers start labeling.

3. The labelers have completed the project (Mark as Completed).

4. There is at least 1 conflict.

5. The project status will update to “Ready for review” and Datasaur will notify the assigned dynamic reviewer.