Collaborate on Projects
  • 20 Dec 2022
  • 8 Minutes to read
  • Dark
    Light
  • PDF
  This documentation version is deprecated, please click here for the latest version.

Collaborate on Projects

  • Dark
    Light
  • PDF

Article summary

Overview

LandingLens offers different ways to collaborate on a Project:

Note:
You can only send Labeling Tasks to Members who have been invited to your Project.

Project Permissions

When you invite Members to your Project, you can choose how they can participate in your Project. This is managed with Permissions. These are the types of Permissions:

  • Upload Data: Users can upload images.
  • Train Model: Users can train the Model.
  • Deploy Model: Users can deploy the Model.
  • Direct Label: Users can label images.

Invite Members to Projects

  1. Open the Project you want to invite a Member to.
  2. Click Settings in the left sidebar.
    Settings Button 
  3. Click Invite on the Permission tab.Invite Button
    Invite Button
  4. Select the Members you want to invite. (You may need to search for the Member to narrow down the list of users.)
  5. Select the Project Permissions you want to give the Member.
  6. Click Add Members to Projects. An email is sent to the user, inviting them to your Project.
Invite Members to Projects

Member Actions on the Permission Tab

The Permission tab on the Settings page allows you to manage Member access to Projects. These are the actions you can perform:

  • Manage Permissions: Edit the Permissions of the Member. This option is only available for Members that have the "Standard" role.
  • Change Role: Projects have three types of Roles—Owner, Labeler, and Standard. 
    • Owners are generally the creators of Projects. They can perform any action on the Project.
    • Labelers can only label images.
    • For Standard Members, you can define their access with Permissions.
  • Transfer Ownership: Make the selected user Owner of the Project.
  • Remove: Remove the Member from the Project.

Actions on the Permissions Tab

Note:
After making an edit, remember to click Done to save your updates.

Agreement-Based Labeling

LandingLens offers the ability to create and manage a Label Book. A Label Book is a living document to help guide users on how to properly label various categories of Classes. Now, you be asking:

  • How do we measure the consistency of labels across multiple users?
  • How do we ensure that only clean, accurate labels will be added to images?

The answer is Agreement-Based Labeling. Agreement-Based Labeling is the concept of allowing multiple users to label the same set of images to analyze the consistency of their labels. This method allows you to identify the areas where users may need more direction with labeling.

For example, let's say you have these Classes for Labels to choose from:

  • Scratch
  • Rub mark
  • Crack

Now take a look at the image below. Some Labelers may choose to label the identified area as a scratch, while others may select rub mark or crack. To help clarify this ambiguity, you can use Agreement-Based Labeling, then update your Labeling Book accordingly.

Ambiguous Mark to Be Labeled

Agreement-Based Labeling Workflow

This is how Agreement-Based Labeling works:

  1. Send a Labeling Task to two or three users. This will allow multiple users to label the same sets of images. 
  2. After the images have been labeled,LandingLens will:
    • Calculate the similarities between the labels. This calculation is called the Agreement Score.
    • Rank the images by Agreement Score, from lowest (least agreed and most confusion) to highest (most agreed and least confusion).
  3. You can then review the labels and accept or reject them. When you reject a label, you can provide feedback to the Labeler. It is also recommended to update the Label Book to prevent these types of occurrences in the future.
    Achieve Accurate Labeling with Agreement-Based Labeling 

Labeling Tasks

Let's say you labeled some images, and you want to compare your findings with another user. To do this, you can send a Labeling Task to that user. 

LandingLens sends images in batches of 20 until all images have been reviewed by the required number of Labelers. However, if you send a Labeling Task that has fewer than 20 images, LandingLens will automatically divide those images into smaller batches. Let's explore how LandingLens sends images works in an example.

Let's say you created a Labeling Task with these settings:

  • The Labeling Task has 30 images.
  • Two Labelers are required to label each image.

You then send the Labeling Task to these three Labelers: Anna, Cameron, and Eliza. The following steps will occur:

  1. LandingLens will take the 30 images and divide them into two batches:
    • Batch One: Includes the first 20 images 
    • Batch Two: Includes the remaining 10 images
  2. LandingLens will send Batch One to two of the Labelers. (Let's say, Anna and Cameron.) 
  3. Since two Labelers are required to label each image, the first person to complete their labels will be sent Batch Two. For example, if Cameron finishes labeling Batch One before Anna, then he will be sent Batch Two.
  4. LandingLens will send Batch Two to the third Labeler. (In this case, Eliza.)

LandingLens Sends Images in Batches Until All Images Have Been Reviewed by the Required Number of Labelers 
Notes:
  • You can only send Labeling Tasks to Members who have been invited to your Project.
  • One set of images can only be added to one Labeling Task at a time; you cannot have the same set of images in multiple Labeling Tasks.

Send Labeling Tasks

  1. Open the Project you want to send a Labeling Task for.
  2. Select the images you want to send.
  3. Click Create Labeling Task in the action bar.
    Create Labeling Task 
  4. Enter a brief, descriptive name for the Labeling Task in the Task Name field.
  5. Select how many Labelers you want to label the images from the drop-down menu. For example, if you want two Labelers to label images, select 2.
  6. If you've already labeled some images, you can choose whether to display (carry over) those existing labels. Select No to remove the labels, or select Yes to display the labels. 
  7. Select the Labelers you want to send a Labeling Task to from the Assign Labelers drop-down menu. To add multiple users, click the + icon.
  8. Click Create Task. An email is sent to the Labelers, notifying them of the task.
    Create a Labeling Task 

Events That Occur After Labeling Tasks Are Sent

After a Labeling Task is sent, these events occur:

  • The New Labeling Task email is sent to the Labeler. This email includes a link to the requested images.
  • Images in a Labeling Task will have the status In Task.
    The "In Task" Status Will Display for Images in a Labeling Task
  • Images in a Labeling Task can only be labeled by the requested Labelers. (The requested Labelers can only label images using the link from the New Labeling Task email.)
    Images Can Only Be Labeled by the Requested Labeler  

Assign Additional Users to Labeling Tasks

  1. Open the Project you want to assign users to.
  2. Click Tasks in the left side panel.
  3. On the Reviewing tab, click the vertical ellipses button and select Assign To.
    Assign To 
  4. Click the icon.
    Add a User
  5. Enter the name of the user you want to assign the Labeling Task to. (Remember, you can only invite users who have been invited to your Project.)
  6. Click Assign.
    Assign Another User to a Labeling Task 

    Close Labeling Tasks

    You can close a Labeling Task before it has been completed. For example, if an assigned Labeler is unable to complete a Labeling Task, you can close it.

    To close a Labeling Task:

    1. Open the Project that has the Labeling Task you want to close.
    2. Click Tasks in the left side panel.
    3. On the Reviewing tab, click the vertical ellipses button and select Assign To.
      Close Task
    4. Click Yes in the pop-up window to confirm your action.

View and Submit Labeling Tasks

If a user has sent you a Labeling Task, you will receive the New Labeling Task email. 

To view an assigned Labeling Task:

  1. Click the link in the New Labeling Task email.
  2. The My Tasks window displays. This window the details of the Task, including the overall progress. Click Go to Task to see the images you need to label.
    Go to Task 
  3. Label the images.
  4. After you've labeled all images, click Submit to share your labels.
    Submit a Labeling Task 

Review Submitted Labeling Tasks

After all the Labelers have submitted their Labeling Task, you will receive the Task [#] Has Labels Ready for Your Review email. This email includes a link to the labeled images. Click this link to compare the labels. You can also click the Review button on the Tasks page to compare the labels.

Review a Labeling Task 

The Review Task page with the labeled images will display. This page includes several settings to help you review the labels. The table below describes these settings. 

Review Task Settings 
NumberSettingsDescription
1Review Metrics
  • In Review: The number of labeled images that have yet to be reviewed.
  • Accepted: The number of images that have been manually accepted. 
  • Rejected: The number of images that have been auto-rejected or manually rejected. Images are auto-rejected based on the Agreement Threshold. (See number 3.)
2Hot Keys / Label Book / Submit
  • Hot Keys: LandingLens offers a variety of hot keys (or keyboard shortcuts) to help speed up the process of reviewing tasks. This button displays those hot keys.
  • Label Book: Opens the Label Book. This is useful so you can update the entries. 
  • Submit: After you've reviewed all the images, you can submit them to complete the review.
3Agreement ThresholdBy default, LandingLens calculates the similarities between all labels. This calculation is called the Agreement Score. The platform determines the Agreement Score based on the overlap of the Labelers' labels. For example, if the labels overlap by 50%, then the Agreement Score will be .50.

Note: If there is only one Labeler assigned to a task, you will not see the Agreement Threshold. 
 
You can adjust the required Agreement Score that images must have by changing the Agreement Threshold. For example, let's say the Agreement Threshold is set to .80. This means that the Labelers' labels must overlap by 80% to be approved.
 
Note: A high Agreement Score does not necessarily mean that the labels are correct. It is still recommended to review these labels.
4Image AgreementThe Agreement Score for the opened image.
5SettingsThe Settings icon allows you to adjust the brightness and contrast of the image. This allows you to brighten images that were taken at night, for example.
6Labelers' LabelsBy default, the labels of all Labelers display on the image. You can select a Labeler to see their label only. This is useful if you want to accept the labels of a single Labeler.
7Agreement SortBy default, images are sortedby Agreement Score, from lowest (least agreed and most confusion) to highest (most agreed and least confusion). You can click the Agreement arrow to sort from the highest to lowest Agreement Score instead.
8Action buttons
  • Add Image to Label Book: There may be times when a Labeler is unsure how to label an image because the objects to identify overlap or are close together. You can add these types of images to the Label Book to help users in these rare cases.
  • Add Metadata: Tag images by adding metadata. Tagging images can help users locate specific images. Example tags: rare-case, ambiguous-case.
  • Review Notes: Leave feedback for the Labelers to help them understand why you rejected an image.
  • Accept / Reject: Accept or reject the labels of an image.
  • Edit: Update the labels.




Was this article helpful?