A Test Project allows you to create a test for your labelers. This will be useful to know how well the labelers understand the projects' guidelines.
The key concept in a Test Project is that all pre-applied labels or labels applied by reviewers are considered as Ground Truth Labels. In order for the labelers want to pass the test, their labels should match the Ground Truth Labels.
Create a Test Project
The Test Project template is only available on team project, and has 4 steps. Any pre-labeled file conforming to Datasaur's supported formats can be uploaded in step 1.
Assign the appropriate labeler
After assigning the project to the appropriate labelers, click Next and launch the project.
Note: most settings will be locked in Step 4 and Step 5, as they are not applicable.
Start labeling the tokens
After launching the project as the project creator, you will start in Reviewer Mode by default. If you uploaded an unlabeled file, you can start labeling the tokens. These labels will be the Ground Truth Labels.
Labeler will get a project with [TEST] prefix.
Uploading a pre-labeled file, as in the example below, will save you time.
Labeler has finished the project
Once the labelers click Mark as Complete, please navigate to or refresh the Reviewer Mode page. You will see conflicts when the labelers' labels and the Ground Truth labels do not match.
You can see labelers' work by clicking on the dropdown in the Test Project extension. There are three different label colors.
The blue labels are labels applied by labelers.
The purple labels are ground truth labels.
The gray labels mean that labelers' labels and ground truth labels are matched.
Three different colors
You can also see the detailed information about matched labels, incorrect labels, and total labeling time in the extension.
What types of projects does Test Project support?
All types! You can create token-based, row-based, and document-based projects.
Upload Pre-labeled Files
Besides uploading unlabeled files, a Test Project allows you to upload pre-labeled files. You can upload pre-labeled files with the format below.