Collaborate on Projects

Overview

LandingLens offers different ways to collaborate on a Project:

• Invite Members to Projects: Allow users to perform different tasks on your Project.
• Send Labeling Tasks: Invite users to label a set of images.

Project Permissions

When you invite Members to your Project, you can choose how they can participate in your Project. This is managed with Permissions. These are the types of Permissions:

• Upload Data: Users can upload images.
• Train Model: Users can train the Model.
• Deploy Model: Users can deploy the Model.
• Direct Label: Users can label images.

Invite Members to Projects

1. Open the Project you want to invite a Member to.
2. Click Settings in the left sidebar.
   Settings Button 
3. Click Invite on the Permission tab.
   Invite Button
4. Select the Members you want to invite. (You may need to search for the Member to narrow down the list of users.)
5. Select the Project Permissions you want to give the Member.
6. Click Add Members to Projects. An email is sent to the user, inviting them to your Project.

Invite Members to Projects

Member Actions on the Permission Tab

The Permission tab on the Settings page allows you to manage Member access to Projects. These are the actions you can perform:

• Manage Permissions: Edit the Permissions of the Member. This option is only available for Members that have the "Standard" role.
• Change Role: Projects have three types of Roles—Owner, Labeler, and Standard. 
  • Owners are generally the creators of Projects. They can perform any action on the Project.
  • Labelers can only label images.
  • For Standard Members, you can define their access with Permissions.
• Transfer Ownership: Make the selected user Owner of the Project.
• Remove: Remove the Member from the Project.

Actions on the Permissions Tab

Agreement-Based Labeling

LandingLens offers the ability to create and manage a Label Book. A Label Book is a living document to help guide users on how to properly label various categories of Classes. Now, you be asking:

• How do we measure the consistency of labels across multiple users?
• How do we ensure that only clean, accurate labels will be added to images?

The answer is Agreement-Based Labeling. Agreement-Based Labeling is the concept of allowing multiple users to label the same set of images to analyze the consistency of their labels. This method allows you to identify the areas where users may need more direction with labeling.

For example, let's say you have these Classes for Labels to choose from:

• Scratch
• Rub mark
• Crack

Now take a look at the image below. Some Labelers may choose to label the identified area as a scratch, while others may select rub mark or crack. To help clarify this ambiguity, you can use Agreement-Based Labeling, then update your Labeling Book accordingly.

Ambiguous Mark to Be Labeled

Agreement-Based Labeling Workflow

This is how Agreement-Based Labeling works:

1. Send a Labeling Task to two or three users. This will allow multiple users to label the same sets of images. 
2. After the images have been labeled,LandingLens will:
   • Calculate the similarities between the labels. This calculation is called the Agreement Score.
   • Rank the images by Agreement Score, from lowest (least agreed and most confusion) to highest (most agreed and least confusion).
3. You can then review the labels and accept or reject them. When you reject a label, you can provide feedback to the Labeler. It is also recommended to update the Label Book to prevent these types of occurrences in the future.
   

   Achieve Accurate Labeling with Agreement-Based Labeling 

Labeling Tasks

Let's say you labeled some images, and you want to compare your findings with another user. To do this, you can send a Labeling Task to that user. 

LandingLens sends images in batches of 20 until all images have been reviewed by the required number of Labelers. However, if you send a Labeling Task that has fewer than 20 images, LandingLens will automatically divide those images into smaller batches. Let's explore how LandingLens sends images works in an example.

Let's say you created a Labeling Task with these settings:

• The Labeling Task has 30 images.
• Two Labelers are required to label each image.

You then send the Labeling Task to these three Labelers: Anna, Cameron, and Eliza. The following steps will occur:

1. LandingLens will take the 30 images and divide them into two batches:
   • Batch One: Includes the first 20 images 
   • Batch Two: Includes the remaining 10 images
2. LandingLens will send Batch One to two of the Labelers. (Let's say, Anna and Cameron.) 
3. Since two Labelers are required to label each image, the first person to complete their labels will be sent Batch Two. For example, if Cameron finishes labeling Batch One before Anna, then he will be sent Batch Two.
4. LandingLens will send Batch Two to the third Labeler. (In this case, Eliza.)

LandingLens Sends Images in Batches Until All Images Have Been Reviewed by the Required Number of Labelers 

Send Labeling Tasks

1. Open the Project you want to send a Labeling Task for.
2. Select the images you want to send.
3. Click Create Labeling Task in the action bar.
   Create Labeling Task 
4. Enter a brief, descriptive name for the Labeling Task in the Task Name field.
5. Select how many Labelers you want to label the images from the drop-down menu. For example, if you want two Labelers to label images, select 2.
6. If you've already labeled some images, you can choose whether to display (carry over) those existing labels. Select No to remove the labels, or select Yes to display the labels. 
7. Select the Labelers you want to send a Labeling Task to from the Assign Labelers drop-down menu. To add multiple users, click the + icon.
8. Click Create Task. An email is sent to the Labelers, notifying them of the task.
   Create a Labeling Task 

Events That Occur After Labeling Tasks Are Sent

After a Labeling Task is sent, these events occur:

• The New Labeling Task email is sent to the Labeler. This email includes a link to the requested images.
• Images in a Labeling Task will have the status In Task.
  The "In Task" Status Will Display for Images in a Labeling Task
• Images in a Labeling Task can only be labeled by the requested Labelers. (The requested Labelers can only label images using the link from the New Labeling Task email.)
  Images Can Only Be Labeled by the Requested Labeler  

Assign Additional Users to Labeling Tasks

1. Open the Project you want to assign users to.
2. Click Tasks in the left side panel.
3. On the Reviewing tab, click the vertical ellipses button and select Assign To.
   Assign To 
4. Click the + icon.
   Add a User
5. Enter the name of the user you want to assign the Labeling Task to. (Remember, you can only invite users who have been invited to your Project.)
6. Click Assign.
   Assign Another User to a Labeling Task 

   Close Labeling Tasks

   You can close a Labeling Task before it has been completed. For example, if an assigned Labeler is unable to complete a Labeling Task, you can close it.

   To close a Labeling Task:

   1. Open the Project that has the Labeling Task you want to close.
   2. Click Tasks in the left side panel.
   3. On the Reviewing tab, click the vertical ellipses button and select Assign To.
      
      Close Task
   4. Click Yes in the pop-up window to confirm your action.

View and Submit Labeling Tasks

If a user has sent you a Labeling Task, you will receive the New Labeling Task email. 

To view an assigned Labeling Task:

1. Click the link in the New Labeling Task email.
2. The My Tasks window displays. This window the details of the Task, including the overall progress. Click Go to Task to see the images you need to label.
   
   Go to Task 
3. Label the images.
4. After you've labeled all images, click Submit to share your labels.
   
   

   Submit a Labeling Task 

Review Submitted Labeling Tasks

After all the Labelers have submitted their Labeling Task, you will receive the Task [#] Has Labels Ready for Your Review email. This email includes a link to the labeled images. Click this link to compare the labels. You can also click the Review button on the Tasks page to compare the labels.

Review a Labeling Task 

The Review Task page with the labeled images will display. This page includes several settings to help you review the labels. The table below describes these settings. 

Review Task Settings