Workforce Management
Workforce management is a critical piece of the data labeling flow. We've interviewed 100+ ML teams around the world and captured best practices for managing a labeling team. Datasaur's workforce management platform allows you to create teams, invite team members, assign projects, and review the work.
After signing in, you will automatically be directed to your personal workspace. On the top-right hand corner, please click Switch Workspace -> Create a New Workspace.. . You can type in a team name and add a logo.
Note: the maximum logo size that can be uploaded is 500 KB.
Datasaur Project List Page
You can also rename the team and change the logo by hovering to the left sidebar and clicking Settings.
Datasaur Setting Page
Once you have reached your team workspace, invite _y_our team members by navigating to the Team Members page on the left sidebar and clicking on Invite new member.
You can save time by entering multiple emails at once. The maximum number of emails that can be sent at once is 20 emails.
New members will automatically have a role as Members/Labelers. You can promote them to be Reviewers or Admins by clicking the triple-dot menus next to each labeler.
You can also filter members by name and role.
- Creating a project in a team workspace can only be done by an admin. It is the same as creating a project in your personal workspace, with one key difference. When creating a custom project as an admin, there are additional 4th and 5th steps. Please follow the steps outlined in Creating a Custom Project until you reach step 4.
The project creation step 4 is to properly set the assignment to labelers and reviewers.
Project Creation Wizard Step 4
- A project can be assigned to one or more labelers.
- Conflict Resolution:
- The Peer Review Consensus slider allows you to determine the threshold at which labels will be automatically accepted. For highly sensitive projects where there is no room for error, you may want to ensure unanimity from all assigned labelers. For less sensitive projects where efficiency and cost are more important than accuracy, a majority vote may be sufficient. Any label where the threshold is not met will need to be manually reviewed by you, the project creator / reviewer.
- If you check No Consensus, all of labelers label will be treated as conflicting labels
- Allow dynamic review assignment allows you assign your team member as a reviewer automatically when the labelers have conflicts in a project. The detailed information can be found here.
Advanced setting allows you to distribute files to each of the labelers and split the files into multiple parts.
Advanced setting in Step 4
If you have a project with multiple files and multiple labelers, using the Auto file distribution might save your time. Our system will distribute the files according to the consensus review that has been set previously.
💡 Best practice: a common practice is to assign multiple labelers to the same project to get multiple perspectives and eliminate human bias. This can increase accuracy, but will increase costs and time.
In step 5, you can modify the project settings to match with your labeling workflow. These settings are also available in the Project Settings once the project is created.
Project Creation Wizard Step 4
- Labels locked ensures that the label set cannot be changed by labelers.
- Text locked ensures that the document text itself cannot be changed by labelers.
- Confirm unapplied label classes when marking document as complete ensures that the assignees can review all label classes that are not applied in the document when marking the document as complete.
- Show rejected labels in review mode ensures that the reviewers can see all the labels they rejected.
- Hide labels from inactive label set during review ensures that labels from inactive label set will be hidden during review.
- Show original sentences in review mode ensures the the reviewers can see all original sentences.
You can set the Mask PII setting to anonymized the selected information attributes in the project. After you turn the Mask PII setting on, you can define the masking data method and the attributes to mask by clicking the Data Masking Settings.
Masking Data Method is how Datasaur masks the information attributes that you have defined. There are two ways to mask the information attributes:
- Random Character: Replace the personal information with random character.
- Example: May 23rd, 2022 → Pgh 78ax, 9928
- Asterisk: Replace the personal information with asterisk.
- Example: John Doe -> **** ***
To select the information attributes to mask, you can check/uncheck the PII fields on the table. You can also set the regular expression rule to define what entities to mask.
As an admin, you can also hide labelers' names from the reviewers. This feature can be useful to avoid bias while reviewing the project. You can click on Hide Labeler Names during Review toggle to turn this feature on. The Review extension will look like the screenshot below to the Reviewer.
After clicking the Launch Project button, you will be taken to the newly created project.
⚠ Note that as the project creator you will start in Reviewer Mode. If you would like to start labeling, you will need to assign yourself, click the profile dropdown on the navigation bar, then click Switch to Labeler Mode. Any other assigned labelers will automatically start in Labeler Mode.
Reviewer Mode has a Review extension on the right side. This contains a Conflicts section and a Members section.
- The Conflicts section shows any parts of the document where assigned labelers disagreed on the correct label. (The disagreement threshold is set by the Peer Review Consensus setting above). For example:
- One labeler has labeled "Positive".
- The second labeler has labeled "Neutral".
- Since the two labelers disagreed, this will be shown as a conflict.
- Click one to resolve the conflict and set the final label that will be exported.
- The Members section shows who has been assigned to this project, along with their current status.
Note: if you are interested in knowing more about QA/Review, please take a look at this video tutorial. After switching the page to Reviewer Mode, don't forget to refresh the page.
Overview contains a high-level overview of your team's projects and progress. This dashboard is only available and visible to Team Admins.
Datasaur has 10 different analytic measurement in this page. The detailed information about each analytic can be seen here.
Projects contains an overview of the projects created on this team along with relevant statistics. This shows a subset of all the team's projects, ordered by most recent activity. You can access the complete set of projects by clicking on the Projects tab in the left sidebar.
Beside the analytics for all projects in Overview page, Datasaur also provide the detailed analytics per project. You can find the page by click the triple dots on each project, then choose View Project Details.
In addition to being able to manage tags from the Projects Dashboard, you can also manage tags for a project team from the Team Settings page. You can navigate to Settings on the left-side bar and click on Tags.
Note: only admins can manage tags.
This feature allows you to set the limitations for each role in exporting the project.
For example, you want to disable Export feature on labelers' accounts due to security reasons. You can simply uncheck all the methods on Labeler section, then click Save.
After that, Export feature will not be shown on labeler's account like the example below.
Removing team members can be done by navigating to the left sidebar and select Members page. You can click the triple dots and click Remove.
Don't worry! Removing team members from the workspace will remain all their labeling work.
Last modified 12d ago