Workforce Management
Workforce management is a critical piece of the data labeling flow. We've interviewed 100+ ML teams around the world and captured best practices for managing a labeling team. Datasaur's management platform allows you to create teams, invite team members, assign projects, and review the work.
Creating a Team (Video Tutorial)
After signing in, you will automatically be directed to your personal workspace. On the top-right hand corner, please click Create a New Team. You can type in a team name and add a logo.
Note: the maximum logo size that can be uploaded is 500 KB.
You can also rename the team and change the logo by hovering to the left sidebar and clicking Settings.
Invite Members (Video tutorial)
Once you have reached your team workspace, invite your team members by navigating to the Team Members page on the left sidebar and clicking on Invite new member.
You can save time by entering multiple emails at once. The maximum number of emails that can be sent at once is 20 emails.
New members will automatically have a role as Members/Labelers. You can promote them to be Reviewers or Admins by clicking the triple-dot menus next to each labeler.
You can also filter members by name and role.
Creating a project in a team workspace is the same as creating a project in your personal workspace, with one key difference. When creating a custom project as an admin, there is an important 4th step. Please follow the steps outlined in Creating a Custom Project until you reach step 4.
Assign Labelers (Video Tutorial)
A unique step for admins creating projects is to properly set project settings and to assign labelers.
A project can be assigned to one or more labelers.
Labels locked ensures that the label set cannot be changed by labelers.
Text locked ensures that the document text itself cannot be changed by labelers.
The Peer Review Consensus slider allows you to determine the threshold at which labels will be automatically accepted. For highly sensitive projects where there is no room for error, you may want to ensure unanimity from all assigned labelers. For less sensitive projects where efficiency and cost are more important than accuracy, a majority vote may be sufficient. Any label where the threshold is not met will need to be manually reviewed by you, the project creator / reviewer.
Allow dynamic review assignment allows you assign your team member as reviewer automatically when the labelers have conflicts in a project. The detailed information can be found here.
Advanced setting allows you to distribute files to each of the labelers and split the files into multiple parts.
If you have a project with multiple files and multiple labelers, using the Auto file distribution might save your time. Our system will distribute the files according to the consensus review that has been set previously.
💡 Best practice: a common practice is to assign multiple labelers to the same project to get multiple perspectives and eliminate human bias. This can increase accuracy, but will increase costs and time.
As an admin, you can also hide labelers' names from the reviewers. This feature can be useful to avoid bias while reviewing the project. You can click on Hide Labeler Names during Review toggle to turn this feature on. The Review extension will look like the screenshot below to the Reviewer.
QA/Review (Video Tutorial)
After clicking the Launch Project button, you will be taken to the newly created project.
âš Note that as the project creator you start off in Reviewer Mode. If you would like to start labeling, you will need to assign yourself, then click on the Reviewer Mode button to switch to Labeler Mode. Any other assigned labelers will automatically start in Labeler Mode.
Reviewer Mode has a Review extension on the right side. This contains a Conflicts section and a Labelers section.
The Conflicts section shows any parts of the document where assigned labelers disagreed on the correct label. (The disagreement threshold is set by the Peer Review Consensus setting above). For example:
One labeler has labeled "Cover A".
The second labeler has labeled "Cover B".
Since the two labelers disagreed, this will be shown as a conflict.
Click one to resolve the conflict and set the final label that will be exported.
The Labelers section shows who has been assigned to this project, along with their current status.
Note: if you are interested in knowing more about QA/Review, please take a look at this video tutorial. After switching the page to Reviewer Mode, don't forget to refresh the page.
The Overview contains a high-level overview of your team's projects and progress. This dashboard is only available and visible to Team Admins.
The pie chart shows the percentage of projects that are Created, still In progress, Ready for review, In review, and Completed.
Timeline shows the most recent activity on the team.
Project Over Time contains the detailed information about all projects created in the team.
Projects contains an overview of the projects created on this team along with relevant statistics. This shows a subset of all the team's projects, ordered by most recent activity. You can access the complete set of projects by clicking on the Projects tab in the left sidebar.
Members contains information about labeler productivity, including total time spent.
In addition to being able to manage tags from the Projects Dashboard, you can also manage tags for a project team from the Team Settings page. You can navigate to Settings on the left-side bar and click on Tags.
Note: only admins can manage tags.
This feature allows you to set the limitations for each role in exporting the project.
For example, you want to disable Export feature on labelers' accounts due to security reasons. You can simply unchecked all the methods on Labeler section, then click Save.
After that, Export feature will not be shown on labeler's account like the example below.
As an admin, you want to track your labeler's progress by seeing the time spent. Currently, we have three terms to differentiate the time spent.
Total time spent will record all projects assigned to the member since the project is created until project completed. If labelers away (e.g. lunch), this will be recorded as well (as long as they don't click Mark as Complete).
You can find total time spent on the Members page.
Effective total time spent will record the active time labeler open the workspace - even though they don't do the labeling. If labelers away or open another tab in their browsers, this will not be recorded.
If you want to see the effective total time spent, you can navigate to the Project section in the Overview page or download the detailed overview. Unzip the files and choose Projects.csv.
