Segmentation

Segmentation Overview

Segmentation is used to identify objects with great precision. This is done by using a "brush" or Smart Labeling to mark the exact location of an object you want your model to detect. 

Let’s say you want to identify cracks in the cement. You could use Object Detection. However, bounding boxes would include too much non-defective space (or space that you do not want to include in your label). Remember, when you include non-defective space in your label, you are telling the platform that this extra space is part of the defect (or object). Consider the screenshots below where the user is trying to identify a crack with Object Detection and Segmentation. Which would you choose?

Bounding Boxes to Detect Cracks in Cement

Segmentation to Detect Cracks in Cement

Labeling Tools for Segmentation

Segmentation offers the following tools to label objects. You can mix and match tools when labeling.

• Smart Labeling: Click an area to auto-detect objects. 
• Polygons: Create a shape to fill the object you want to detect. You can also use this tool to "erase" labeled regions.
• Brushes: "Paint" over the object you want to detect. You can also use this tool to "erase" labeled regions.
• Polylines: Create lines over the object you want to detect.

Label with Smart Labeling

Smart Labeling is a labeling tool powered by the Segment Anything Model (SAM) that helps you label objects in Segmentation projects faster than using traditional methods.

To use Smart Labeling, simply select the tool, and hover over and click the object you want to select. Smart Labeling automatically detects the shape of the object and adds a gray overlay to it. To confirm that Smart Labeling detected the correct object, press Return (Mac) or Enter (Windows) to confirm!

Smart Labeling

Label an Object with Smart Labeling

To label an object with Smart Labeling:

1. Click the Smart Labeling icon if it isn't selected, or press q.
2. Select the Class you want to label from the drop-down menu.
   
   Select Smart Labeling and the Class
3. Hover over the object you want to select. Smart Labeling adds a gray overlay over the object it "thinks" you want to select.
   Hover Over the Object
4. Click the object. Smart Labeling adds a green dot where you clicked. This, along with the gray overlay, help you track what regions you've selected. Clicking the object also adds a blue border to your frame, which indicates that you're using Smart Labeling.LandingLens Adds a Green Dot
5. If you want to add another region to the same object, hover over that region. The cursor turns into a + (plus).
   The Cursor Turns into a + (Plus)
6. Click the object. Smart Labeling adds a gray overlay to the object and adds a green dot where you clicked.
   LandingLens Adds an Overlay and a Dot
7. You can continue to add regions using this method.
   Click All the Regions You Want to Add
8. Once you've labeled all regions of the object, either click Save Labels (Return), or press Return (Mac) or Enter (Windows).
   
   Save the Labels
9. Smart Labeling labels the object with the selected Class.
   The Object Is Labeled

Remove a Region from a Selection

Let's say a region gets accidentally added to your selection. While Smart Labeling is selected, you can hover over that region until the cursor turns into a − (minus). Then click that region. Smart Labeling removes the selection. When you're done, press Return (Mac) or Enter (Windows). 

To remove a region from an object (assumes that Smart Labeling is enabled, and you just labeled an object):

1. Hover over and click the labeled region. Smart Labeling removes the overlay and adds a red dot where you clicked, which helps you track what regions you've removed.
   Remove Regions
2. Continue to click selected regions until you've removed all the selections you want.
3. When you're done removing regions, press Return (Mac) or Enter (Windows).

Tips for Labeling Multiple Objects in an Image

Let's say you're developing a model that detects tractors in construction sites. When using Smart Labeling, we recommend that you label each tractor one at a time, instead of labeling them all at once. In other words, label one tractor, and then press Return (Mac) or Enter (Windows). Then label the second tractor, and so on.

Label Each Object Individually

We don't recommend labeling all tractors at once, and then clicking Enter/Return. Doing that makes Smart Labeling think that these areas are the same object, and not the same type of object. If you do this, you might notice additional areas added to the object that aren't actually part of that object.

Label with Polygons

1. Open the Project you want to edit.
2. Open the image you want to label.
3. Click the Label tab.
4. Select the Class you want to label from the drop-down menu.
   Class on the Label Tab
5. Click the Polygon icon. (You can also press P on your keyboard.)
6. The Draw icon is auto-selected. Keep this setting.
   Polygon Icon
7. Click where you want to start your shape. A blue circle displays, indicating the start of the shape.
8. Click to continue creating your shape.
9. Complete your shape by clicking the blue circle. The shape automatically fills with the color of the Class.
   Create Shapes with the Polygon Tool

Label with Brushes

1. Open the Project you want to edit.
2. Open the image you want to label.
3. Click the Label tab.
4. Select the Class you want to label from the drop-down menu.
   
   Class on the Label Tab
5. Click the Brush icon. (You can also press B on your keyboard.)
6. By default, the Draw icon is auto-selected. Keep this setting.
7. Adjust the size of the line width accordingly.
   Configure the Brush Settings
8. Click and drag your cursor over the area you want to identify.
9. Release the mouse to complete your edits.
   Label with the Brush Tool

Remove Excess Labels with Brushes

To remove excess labels with a Brush:

1. Open the Project you want to edit.
2. Open the image you want to label.
3. Click the Label tab.
4. Click the Brush icon. (You can also press B on your keyboard.)
5. Click the Erase icon. (You can also press E on your keyboard.)
6. Adjust the size of the eraser accordingly.
   Configure the Eraser Settings
7. Click and drag your cursor over the area you want to remove.
8. Release the mouse to complete your edits.
   Erase Excess Lines

Label with Polylines

1. Open the Project you want to edit.
2. Open the image you want to label.
3. Click the Label tab.
4. Select the Class you want to label from the drop-down menu.
   
   Class on the Label Tab
5. Click the Polyline icon. (You can also press L on your keyboard.)
6. Adjust the size of the line width accordingly.
   Configure the Polyline Settings
7. Click where you want to start your line.
8. Click to change the direction of your line.
9. Press Enteron your keyboard to complete your edits.
   Create Lines with the Polyline Tool