Extending Faster R-CNN: A Deep Dive into Mask R-CNN for Segmentation
Written on
Chapter 1: Understanding Image Segmentation
In the realm of computer vision, image segmentation refers to the process of classifying each pixel within an image. This technique essentially divides an image into distinct sections for easier analysis and understanding.
Previously, we delved into various object detection architectures, including YOLO and SSD. Among these, Faster R-CNN has emerged as a leading model, achieving superior accuracy. In this article, we will discuss how researchers have expanded the capabilities of Faster R-CNN to not only recognize objects but also to generate pixel-level masks for each detected object, thus enabling effective image segmentation.
Section 1.1: Categories of Image Segmentation
While many readers may already be acquainted with image segmentation, it's important to distinguish between two primary types: semantic segmentation and instance segmentation.
Semantic Segmentation
This method classifies each pixel into predefined categories, primarily distinguishing between foreground and background elements. However, it does not differentiate individual objects within similar categories. To illustrate this, consider the following image example:
Instance Segmentation
Conversely, instance segmentation identifies each distinct object within the same class. This approach encompasses:
- Object detection, which recognizes individual items within the frame.
- Object localization, providing bounding box coordinates for each identified object.
- Object classification, determining the types of objects present.
Instance segmentation thus extends the concepts of semantic segmentation by offering a clearer separation between individual objects rather than merely distinguishing between foreground and background.
Section 1.2: The Architecture of Mask R-CNN
Mask R-CNN builds upon the foundation of Faster R-CNN to perform instance segmentation in computer vision tasks. Let's review the components of Faster R-CNN before comparing them with Mask R-CNN.
Faster R-CNN Overview
To summarize, the object detection pipeline of Faster R-CNN involves five key steps:
- Input the image/frame into a backbone network (typically ResNet).
- Extract the feature map using a Feature Pyramid Network (FPN).
- Pass the feature map to the Region Proposal Network (RPN).
- Generate Regions of Interest (RoIs) and return fixed-size feature maps through pooling or other operations.
- Forward these feature maps to the R-CNN to obtain class labels and bounding box coordinates.
This state-of-the-art (SOTA) object detection algorithm produces two outputs:
- A class label for each candidate object.
- A bounding box offset for each candidate object.
Mask R-CNN introduces a third output, enabling instance segmentation: an object mask for each identified object. This necessitates an additional branch in the architecture's final layers, which acts as a binary classifier to generate masks for each category. The Mask head, composed of two convolutional layers, is responsible for producing masks for each Region of Interest (RoI).
To improve accuracy, the RoI pooling module is replaced with a more precise RoI Align module. Future articles will cover the distinctions among various RoI operations.
Additional Note:
The original Mask R-CNN sometimes yields masks with less precise boundaries. Enhancements, such as Point-based Rendering (PointRend), aim to rectify this issue, producing high-resolution predictions over finer grids.
Chapter 2: Training Mask R-CNN for Custom Applications
Having understood Mask R-CNN's operation atop Faster R-CNN, we can explore how to train our own model for custom object detection down to the pixel level. Similar to our previous discussions, we will outline several straightforward steps that you can follow along in Google Colab. A complete notebook for reference is available here.
In this notebook, my objective is to identify two distinct soft drink brands—Pepsi and Mountain Dew—as part of a university project focused on differentiating aluminum cans from plastic bottles. Although there are less expensive alternatives, this solution suffices for demonstration purposes.
- Acquiring the Mask R-CNN Model
I opted to download the entire Mask R-CNN folder for ongoing use, though cloning it from the Matterport GitHub repository is the recommended approach. Ensure your working directory is set to the Mask_RCNN folder.
- Installing Compatible Packages
While various versions may function adequately, these are the specific versions I've tested successfully.
- Setting Up the Environment
Specify the path where you intend to save logs and model checkpoints.
- Configuration Adjustments
- Preparing Custom Dataset and Annotations
A helper function is included for simplified operations.
- Loading Weights and Verifying Configuration
- Training the Model
This step is straightforward, as it directly follows the guidelines from the Mask R-CNN codebase.
- Validating Model Performance
Ensure you update the weights path to utilize the most current weights during inference. You will observe images from your validation set being detected alongside class probabilities, bounding boxes, and corresponding masks.
- Single Image Inference
To run inference on a single image, follow these procedures.
Results from the model are impressive:
- A Pepsi bottle detected with 94% accuracy.
- A Mountain Dew bottle detected with 96.6% accuracy.
You may be curious about the dataset utilized for this project. I encourage you to curate your own dataset by following my article on "Computer Vision: How to curate a simple image dataset for image classification (No code!)". Annotations can be facilitated using tools such as VGG Image Annotator or labelImg.
In summary, this article has illustrated how Mask R-CNN serves as an enhancement of Faster R-CNN with some key modifications. Future discussions will delve into these modifications in greater detail. We also outlined nine straightforward steps to train our own Mask R-CNN model for detecting targeted objects. Upcoming articles will explore how to enhance general Mask R-CNN models by adjusting either the dataset or available hyperparameters—most improvements will likely stem from the images or annotations themselves. Be diligent when annotating your images!
Chapter 3: Practical Implementation and Resources
The first video, titled "Instance Segmentation Using Mask R-CNN on Custom Dataset," offers a detailed explanation of the process of implementing Mask R-CNN for instance segmentation using a custom dataset.
The second video, "Train Mask R-CNN for Image Segmentation (online free GPU)," showcases how to train a Mask R-CNN model for image segmentation using free online GPU resources.
References
- RoI Operations