Test Project
Set a test for your labelers!
A Test Project enables you to create a test for your labelers, which can be useful to assess how well they understand the project's guidelines.
The key concept in a Test Project is that all pre-applied labels or labels applied by reviewers are considered as Ground Truth Labels. To pass the test, labelers must ensure that their labels match the Ground Truth Labels.
Create a Test Project
The Test Project template is only available on team project, and has 4 steps. Any pre-labeled file conforming to Datasaur's supported formats can be uploaded in step 1. All settings in Step 5, as they are not applicable.
Assign the appropriate labeler
After assigning the project to the appropriate labelers, click Launch Project.
Start labeling the spans
As the project creator, you will begin in Reviewer Mode by default after launching the project. If you have uploaded an unlabeled file, you can start labeling the spans, which will become the Ground Truth Labels. Uploading a pre-labeled file will save you time.
If you upload a prelabeled file for TEST project, the labels applied will not be displayed in labelers account.
Labeler has finished the project
Once the labelers click Mark as Complete, please navigate to or refresh the Reviewer Mode page. You will see conflicts when the labelers' labels and the Ground Truth labels do not match.
The color-coding convention for All Labelers View is as follows:
Gray: You upload a prelabeled file in TEST project
Violet: all the Ground Truth (GT) labels, i.e.: the labels applied by the reviewer on review mode
Red: all the labels applied by the labelers.
You can see labelers' work by clicking on the dropdown in the Test Project extension.
The color-coding convention for Individual Labelers View is as follows:
Gray: all the labels applied by the labelers which match the GT labels.
Dark Blue: all the labels applied by the labelers which do not match the GT labels.
Violet: all the GT labels are not answered by labelers.
To help assess the labelers' performance, you can see the detailed information about matched labels, incorrect labels, and total labeling time in the extension. These calculations can be downloaded by clicking the triple dots and selecting Export Summary in csv.
What types of projects does Test Project support?
All types! You can create span-based, row-based, and document-based projects.
Upload Pre-labeled Files
Besides uploading unlabeled files, a Test Project allows you to upload pre-labeled files. You can upload pre-labeled files with the format below.
| Task type | Formats |
|---|---|
Span-based | |
Span-based with arrows | |
Span-based with character based labeling | |
Row-based | |
Document-based |
Last updated
