A Test Project allows you to create a test for your labelers. This will be useful to know how well the labelers understand the projects' guidelines.
The key concept in a Test Project is that all pre-applied labels or labels applied by reviewers are considered as Ground Truth Labels. In order for the labelers want to pass the test, their labels should match the Ground Truth Labels.
The Test Project template is only available on team project, and has 4 steps. Any pre-labeled file conforming to Datasaur's supported formats can be uploaded in step 1.
After assigning the project to the appropriate labelers, click Next and launch the project.
Note: most settings will be locked in Step 4 and Step 5, as they are not applicable.
After launching the project as the project creator, you will start in Reviewer Mode by default. If you uploaded an unlabeled file, you can start labeling the tokens. These labels will be the Ground Truth Labels.
Uploading a pre-labeled file, as in the example below, will save you time.
Once the labelers click Mark as Complete, please navigate to or refresh the Reviewer Mode page. You will see conflicts when the labelers' labels and the Ground Truth labels do not match.
You can see labelers' work by clicking on the dropdown in the Test Project extension. There are three different label colors.
The blue labels are labels applied by labelers.
The purple labels are ground truth labels.
The gray labels mean that labelers' labels and ground truth labels are matched.
You can also see the detailed information about matched labels, incorrect labels, and total labeling time in the extension.
All types! You can create token-based, row-based, and document-based projects.
Besides uploading unlabeled files, a Test Project allows you to upload pre-labeled files. You can upload pre-labeled files with the format below.