Workforce Management

Workforce management is a critical piece of the data labeling flow. We've interviewed 100+ ML teams around the world and captured best practices for managing a labeling team. Datasaur's management platform allows you to create teams, invite team members, assign projects, and review the work.

Creating a Team (Video Tutorial)

After signing in, you will automatically be directed to your personal workspace. On the top-right hand corner, please click Create a New Team. You can type in a team name and add a logo.

Note: the maximum logo size that can be uploaded is 500 KB.

You can also rename the team and change the logo by hovering to the left sidebar and clicking Settings.

Invite Members (Video tutorial)

Once you have reached your team workspace, invite your team members by navigating to the Team Members page on the left sidebar and clicking on Invite new member.

You can save time by entering multiple emails at once. The maximum number of emails that can be sent at once is 20 emails.

New members will automatically have a role as Members/Labelers. You can promote them to be Reviewers or Admins by clicking the triple-dot menus next to each labeler.

You can also filter members by name and role.

Creating a Project

  • Creating a project in a team workspace is the same as creating a project in your personal workspace, with one key difference. When creating a custom project as an admin, there is an important 4th step. Please follow the steps outlined in Creating a Custom Project until you reach step 4.

Assign Labelers (Video Tutorial)

A unique step for admins creating projects is to properly set project settings and to assign labelers.

  • A project can be assigned to one or more labelers. For projects with multiple files, you can click on Auto-distribute Files to manually distribute files to each of the labelers.

💡 Best practice: a common practice is to assign multiple labelers to the same project to get multiple perspectives and eliminate human bias. This can increase accuracy, but will increase costs and time.

  • Labels locked ensures that the label set cannot be changed by labelers.

  • Text locked ensures that the document text itself cannot be changed by labelers.

  • The Peer Review Consensus slider allows you to determine the threshold at which labels will be automatically accepted. For highly sensitive projects where there is no room for error, you may want to ensure unanimity from all assigned labelers. For less sensitive projects where efficiency and cost are more important than accuracy, a majority vote may be sufficient. Any label where the threshold is not met will need to be manually reviewed by you, the project creator / reviewer.

Hide Labelers' Names

As an admin, you can also hide labelers' names from the reviewers. This feature can be useful to avoid bias while reviewing the project. You can click on Hide Labeler Names during Review toggle to turn this feature on. The Review extension will look like the screenshot below to the Reviewer.

QA/Review (Video Tutorial)

After clicking the Launch Project button, you will be taken to the newly created project.

Note that as the project creator you start off in Reviewer Mode. If you would like to start labeling, you will need to assign yourself, then click on the Reviewer Mode button to switch to Labeler Mode. Any other assigned labelers will automatically start in Labeler Mode.

Reviewer Mode has a Review extension on the right side. This contains a Conflicts section and a Labelers section.

  • The Conflicts section shows any parts of the document where assigned labelers disagreed on the correct label. (The disagreement threshold is set by the Peer Review Consensus setting above). For example:

    • One labeler has labeled "Cover A".

    • The second labeler has labeled "Cover B".

    • Since the two labelers disagreed, this will be shown as a conflict.

    • Click one to resolve the conflict and set the final label that will be exported.

  • The Labelers section shows who has been assigned to this project, along with their current status.

Note: if you are interested in knowing more about QA/Review, please take a look at this video tutorial. After switching the page to Reviewer Mode, don't forget to refresh the page.

Overview

The Overview contains a high-level overview of your team's projects and progress. This dashboard is only available and visible to Team Admins.

Overall Projects

The pie chart shows the percentage of projects that are Not started, still In progress and Completed.

Timeline

Timeline shows the most recent activity on the team.

Projects

Projects contains an overview of the projects created on this team along with relevant statistics. This shows a subset of all the team's projects, ordered by most recent activity. You can access the complete set of projects by clicking on the Projects tab in the left sidebar.

Members

Members contains information about labeler productivity, including time spent and accuracy.

Manage Tags in Project Team

In addition to being able to manage tags from the Projects Dashboard, you can also manage tags for a project team from the Team Settings page. You can navigate to Settings on the left-side bar and click on Tags.

Note: only admins can manage tags.

Manage Export for Each Team Role

This feature allows you to set the limitations for each role in exporting the project.

For example, you want to disable Export feature on labelers' accounts due to security reasons. You can simply unchecked all the methods on Labeler section, then click Save.

After that, Export feature will not be shown on labeler's account like the example below.