Projects | Rahil Bhowal

Attention-guided Off-Road Semantic Segmentation for Autonomous Navigation

Sun, 01 Dec 2024 00:00:00 +0000

Introduction

Semantic segmentation is a critical task in computer vision, enabling pixel-level understanding of images for applications ranging from autonomous driving to environmental monitoring. While signifi- cant advancements have been made with architectures like U-Net and DeepLabv3+, most research has focused on structured environments with well-defined boundaries. Off-road environments, however, present unique challenges such as diverse terrain types, imbalanced class distributions, and complex navigability conditions. This project addresses these challenges by proposing a semantic segmentation framework tailored for unstructured off-road environments, evaluated on the RUGD dataset.

Method

A novel model architecture that integrates a ResNet-50 encoder, Atrous Spatial Pyramid Pooling (ASPP) for multiscale context aggregation, and Squeeze-and-Excitation (SE) blocks for adaptive attention. A class rebalancing approach inspired by prior work to mitigate dataset imbalances by grouping the 24 RUGD classes into 4 broader categories relevant to off-road navigation. Comprehen- sive evaluation of different model architectures, including SegFormer and ResNetUNetASPP variants, across coarse-grained (4-class), intermediate (6-class), and fine-grained (24-class) classification levels. Our results demonstrate that combining hierarchical feature extraction with attention mechanisms significantly improves segmentation performance in off-road scenarios. Additionally, we identify key limitations and propose avenues for future research, including lightweight models for edge deployment and sensor fusion strategies for enhanced robustness. The implemented model combines a ResNet-50 encoder, an Atrous Spatial Pyramid Pooling (ASPP) module, and Squeeze-and-Excitation (SE) blocks for attention-based refinement. This design leverages hierarchical feature extraction, multiscale context handling, and adaptive feature emphasis, ensuring robust semantic segmentation capabilities.

Experiments

Our experimental testbed consists of the RUGD dataset, a real-world dataset specifically curated for semantic segmentation tasks in outdoor environments. This dataset features diverse terrain types, obstacles, and environmental conditions, making it a robust benchmark for evaluating segmentation model. We evaluated our model’s performance across three levels of classification complexity: 4-class classification: Coarse-grained segmentation to test general terrain distinction. 6-class clas- sification: Intermediate granularity for assessing the model’s ability to differentiate more nuanced categories. 24-class classification: Fine-grained segmentation to push the model’s capacity for detailed understanding and feature discrimination During the start of the project, we considered the following model architectures, which includes.

The architecture of the ResNetUNetASPP model, which combines elements from ResNet, U-Net, and Atrous Spatial Pyramid Pooling (ASPP). This architecture involved segmentation for all 24 classes present in the RUGD dataset.
Next we consider the ResNetUNetASPPWithAttention model which is a combination of the following model arhitectures ResNet50 Encoder, Atrous Spatial Pyramid Pooling (ASPP) module, U- Net-style Decoder and Squeeze-and-Excitation (SE) Attention blocks. We use a ResNet50 backbone for the model pretrained on the ImageNet.
Next we focused on changing out ResNetUNetASPPWithAttention model for 4 classes. Dividing our 24 classes into 4 categories. The model architecture is subtly changed with a focus on modularity and attention-driven refinement.

Addressing Data Imbalance There is a class imbalance in the dataset. So inorder to have a higher accuracy segmenting the off-road environment. We took help from the Ga-Nav paper which addressed the class imbalance by broadly classifying the 24 classes into 6 categories which includes Smooth Region(includes terrains like concrete and asphalt, which are highly navigable and suitable for autonomous navigation), Rough Region(Covers terrains such as gravel, grass, dirt, and sand, which are navigable but may introduce moderate friction or difficulty), Bumpy Region(includes terrains with small rocks or rock beds that are more challenging to traverse, especially for smaller or less capable robots), Forbidden Region(includes unsafe terrains like water, bushes, or tall vegetation that the robot must avoid to prevent damage or failure), Obstacle(Includes physical barriers like trees, poles, or logs that block the robot’s path and must be navigated around), Background(covers irrelevant classes such as sky, signs, or void regions, which are not part of the navigable environment). These classes were specifically picked adhering to the requirements of off-road navigation and simplifying the segmentation task. Our approach inspired by this paper divided the 24 classes into 4 categories which includes Back- ground:(includes element that are not part of the environment like Sky, Sign), Smooth region (consists of region that are easily navigable like Concrete, Asphalt), Rough Navigable (includes classes like Gravel, Grass, Dirt, Sand, Mulch) and Obstacles/Bumpy Navigable (includes region where the robot might get stuck while navigating. Includes classes like Rock, Rock-Bed, Tree, Pole, Vehicle, Container/Generic Object, Building, Log, Bicycle, Person, Fence, Bush, Picnic Table, Bridge).

Implementation Details Besides categorising the 24 classes into 4 categories. We also computed the class weights based on the frequency of 4 classes in the dataset. Balance weights were calculated to ensure greater representation of underrepresented groups. These weights were incorporated in the CrossEntropy class ensuring the model learns features for all classes. The below table shows the parameters for training on the RUGD Dataset.

Table 1: Comparison of Training Parameters for Different Models

$$ \begin{array}{|l|c|c|c|c|} \hline \textbf{Method} & \textbf{Epochs} & \textbf{Learning Rate} & \textbf{Optimizer} & \textbf{Batch Size} \\ \hline \text{ResNetUNetASPP} & 16 & 0.001 & \text{Adam} & 18 \\ \hline \text{ResNetUNetASPPWithAttention} & 31 & 0.001 & \text{Adam} & 16 \\ \hline \text{Segformer} & 25 & 0.00006 & \text{AdamW} & 32 \\ \hline \text{ResNetUNetASPPWithAttention*} & 13 & 0.001 & \text{Adam} & 32 \\ \hline \end{array} $$

Evaluation and Metrics

The 4 models were evaluated on the RUGD dataset, where the ResNetUNetASPP, ResNetUNetASP- PWithAttention, Segformer were evaluated on all 24 classes, the ResNetUNetASPPWithAttention* was evaluated dividing the 24 classes into 4 categories.

Table 2: Performance Metrics on RUGD

$$ \begin{array}{|l|c|c|} \hline \textbf{Method} & \textbf{mIoU} & \textbf{Mean Pixel Accuracy} \\ \hline \text{ResNetUNetASPP} & 0.1303 & 0.7449 \\ \hline \text{ResNetUNetASPPWithAttention} & 0.1401 & 0.7221 \\ \hline \text{Segformer} & 0.2732 & 0.3331 \\ \hline \text{ResNetUNetASPPWithAttention*} & 0.6138 & 0.7411 \\ \hline \end{array} $$

Output

Output on RUGD Dataset

3D Reconstruction Using Structure From Motion (SfM)

Sun, 24 Nov 2024 00:00:00 +0000

Objective

To reconstruct a 3D scene and camera trajectory from a sequence of images using Structure-from-Motion (SfM). The pipeline includes feature detection, pose estimation, 3D point triangulation, and bundle adjustment for optimization.

Dataset

Source: A custom dataset of images depicting a static scene (“buddha_images”). Details: The dataset consists of 25 images captured from different perspectives, simulating real-world camera motion around the object.

Original Dataset

Preprocessing: Images are normalized to ensure uniform pixel intensity. CLAHE (Contrast Limited Adaptive Histogram Equalization) is applied for better feature detection in low-contrast regions.

Preprocessed dataset

Problem Statement

The goal is to accurately reconstruct the 3D structure of the scene and estimate the camera trajectory by:

Detecting and matching features across the images.
Estimating relative camera poses and 3D points.
Refining the reconstruction with bundle adjustment to minimize reprojection errors.

Workflow

1. Image Preprocessing

Images are loaded and enhanced to improve feature detection.
Conversion to grayscale and CLAHE is applied for contrast improvement.

2. Feature Detection and Matching

SIFT (Scale-Invariant Feature Transform) is used to detect robust keypoints in each image.
Keypoints are matched between consecutive images using FLANN-based matching.
RANSAC is applied to eliminate outliers and ensure reliable matches.

3. Pose and Structure Estimation

Fundamental Matrix: Estimated using RANSAC for epipolar geometry constraints.
Essential Matrix: Extracted from the fundamental matrix to estimate relative camera poses.
3D Point Triangulation: Feature correspondences are triangulated to estimate initial 3D points.

Epipolar lines **4. Bundle Adjustment**

Framework: GTSAM is used to build a factor graph that models the relationship between camera poses and 3D points.
Optimization:
Camera poses and 3D points are refined by minimizing reprojection errors.
The Levenberg-Marquardt algorithm is employed for nonlinear optimization.

5. Visualization The 3D reconstruction is visualized as: A point cloud representing the reconstructed scene. A trajectory showing the path of the camera during image capture.

Output

Without Bundle Adjustment

With Bundle Adjustment

Output image

Underwater Image Mosaicing

Sun, 01 Sep 2024 00:00:00 +0000

Introduction

In this project I worked on the understanding of image registration by working on a low contrast, somewhat challenging dataset.

The paper describing the work for photomosaicking is attached !Pizarro2003joe.pdf. I wanted to read this paper and see if we can duplicate some of the work in it while using some of the techniques I have learned in the class so far.

Output

There are 2 set of images I worked with 2 set of images each containing 6 images and 128 images.

Steps involved are normalising the image, applying histogram equalization techniques like CLAHE, feature extraction and trying image registration. Next step I tried to use GTSAM and build a factor graph to get a better fit.

Normalising and applying CLAHE to all the images

Normalised Images

CLAHE Images

SIFT Images

Factor graph before Optimization for 6 image set

Factor graph after Optimization for 6 Image set

Output image after Optimization for 6 Image set

Output Image before Optimization for 128 Image set

Output Image after Optimization for 128 Image set

Real-Time Social Navigation Strategy using Deep Reinforcement Learning

Wed, 01 May 2024 00:00:00 +0000

Introduction

Our approach to local navigation for autonomous mobile robots in crowded environments, leveraging deep reinforcement learning (DRL) techniques and 2D laser scans. Traditional navigation methods struggle in densely populated areas, often leading to constant replanning and inefficiency. To address this challenge, recent research has turned to DRL-based solutions, which have shown promising results in mapless navigation using 2D laser scans. In our proposed approach, we integrate the perception of risk posed by the moving crowd into the observation space of our DRL system. By focusing on the K most hazardous obstacles, identified based on collision probability, the robot can better assess risk even in unfamiliar crowd behaviors. This targeted attention enables effective navigation in high-density crowds while maintaining scala- bility. Furthermore, we enhance learning efficiency by incorporating local waypoints into the reward function, aiding in global path planning. Through extensive evaluation across various crowd scenar- ios, we compare our approach with a recent state-of-the-art DRL-based method, demonstrating its effectiveness under similar test conditions.

Experiment Setup

The virtual environment for this project was designed to simulate real-world scenarios and train reinforcement learning models effectively. The setup involved:

Simulation Platform:

Gazebo 11: Used in combination with ROS2 Foxy to create a stable simulation environment. Custom Worlds: Developed to replicate realistic and dynamic environments, including static and moving obstacles. Custom SDF Models: Simulated human walking patterns using randomized cylinder animations to mimic crowd movement. Training Worlds:

Arena World: Optimized for training in mixed static and dynamic obstacle environments. Corridor World: Simulated a realistic, constrained environment for validation and testing. Room World: Designed for dense crowd scenarios, primarily dynamic obstacle avoidance. Robot Models:

Training started with Turtlebot3 Waffle Pi, transitioning to Turtlebot4 Lite for deployment due to its dimensional resemblance to the final robot. Custom URDF files were created for compatibility with Gazebo 11 and ROS2 Foxy. Simulation Features:

Environments included randomized dynamic obstacles and realistic terrain features. The LIDAR sensor was integral to robot perception and state estimation. Deployment Interface:

ROS2-based Communication: A unique ROS-DOMAIN-ID facilitated real-time updates and commands between the host PC and onboard Raspberry Pi 4. Host PC Requirements: Ubuntu 22.04 with ROS2 Humble for advanced computing tasks, ensuring better performance than onboard hardware. This environment setup ensured robust training, testing, and deployment of the socially aware navigation system, allowing the robot to navigate crowded, dynamic environments effectively.

Approach

Our project utilizes Deep Reinforcement Learning (DRL) to enable socially aware navigation for mobile robots in crowded environments. The approach prioritizes safe, efficient, and dynamic movement in real-world scenarios, leveraging modern reinforcement learning algorithms like Twin Delayed Deep Deterministic Policy Gradient (TD3).

Problem Formulation:

The robot navigates using 2D laser scans, incorporating both static and dynamic obstacles into the observation space. A novel perceived risk framework focuses on the K most hazardous obstacles in the environment, based on collision probability. Observations include: Static environment data from LIDAR. Goal-relative metrics like distance and heading. Obstacle data including positions and velocities of the most hazardous objects. Action and Reward Spaces:

Action Space: Continuous linear and angular velocity commands for smooth and adaptive movement. Reward Structure: Designed to guide behavior, it penalizes collisions, incentivizes reaching goals, and introduces waypoint-based rewards for efficient pathfinding. Reinforcement Learning Algorithm:

TD3 Enhancements: Dual Critic Networks to mitigate overestimation bias. Delayed actor updates for stability. Action noise for better exploration. The model learns an optimal policy by interacting with custom dynamic environments during training. Simulation and Deployment:

Training was conducted in Gazebo 11 with ROS2 Foxy, using custom environments tailored to social navigation challenges. Deployment on Turtlebot4 Lite in real-world settings validated the system, highlighting its robustness and areas for improvement.

Videos

Read more about it on !Report

Search and Rescue Mission Using Turtlebot

Wed, 01 May 2024 00:00:00 +0000

Introduction

In response to the critical need for effective reconnaissance in disaster scenarios, this research article presents the design and implementation of a complete autonomous robot system using the Turtlebot3 Burger (TB3) with Robot Operating System (ROS) Noetic. Upon deployment in closed, initially unknown environments, the system aims to generate a comprehensive map and identify any present ’victims’ using AprilTags as stand-ins. We discuss our solution for search and rescue missions, while additionally exploring more advanced algorithms to improve search and rescue functionalities. We introduce a Cubature Kalman Filter (CKF) to help reduce the Mean Squared Error (MSE) [m] for AprilTag localization and an information-theoretic exploration algorithm to expedite exploration in unknown environments.

This project we addresses the need for an autonomous robotic system that can both map unknown, closed environments and identify “victims(AprilTags)” in these settings. Our goal is to design and implement a robust system using the Robot Operating System (ROS) Noetic, which can:

• Generate a complete occupancy grid map of an unknown environment, providing essential data for navigation and strategic planning.

• Locate and accurately estimate the poses of “victims,” represented by AprilTags, to simulate the identification and location of individuals in need of rescue.

Method

Frontier Detection1 (background):

Detection of new frontier cells (and erase old ones).

Form frontiers from frontier cells.

Compute Exploration Goal2,3 (service request):

Sample free points around each frontier’s centroid.

Compute gain for each point (optimizes yaw)

Return highest gain pose as exploration goal.

Videos

Demo Video Trial 1

Demo Video Trial 2

Final Map and Tag Location

Read more about it on !Report

Loam-Series Evaluation

Fri, 24 Nov 2023 00:00:00 +0000

Introduction

In this project, we systematically examined and compared various algorithms within the LOAM series, specifically ALOAM, LEGO-LOAM, and SC-LEGO-LOAM. Our primary focus was on meticulously analyzing the developmental trajectory from a singular lidar odometry approach (the original LOAM) to the comprehensive SLAM systems, namely Lego- LOAM and SC-LEGO-LOAM. This comprehensive analysis involved intricate comparisons between algorithms, elucidation of improvements made, and in-depth assessments. Ultimately, we conducted thorough testing and comparison of these three algorithms on two extensive long-distance datasets

Evaluation and Metrics

Table 1: ATE of LeGO-LOAM and SC-LeGO-LOAM in Kaist sequence {{< math >}}

$$ \begin{array}{|l|c|c|} \hline \textbf{Metric} & \textbf{SC-LeGO-LOAM (\%)} & \textbf{LeGO-LOAM (\%)} \\ \hline \text{max} & 1.60 & 2.55 \\ \hline \text{mean} & 0.84 & 1.03 \\ \hline \text{median} & 0.82 & 1.01 \\ \hline \text{min} & 0.08 & 1.35 \\ \hline \text{rmse} & 0.90 & 1.13 \\ \hline \end{array} $$

Table 2: ATE of LeGO-LOAM and SC-LeGO-LOAM in Riverside sequence {{< math >}}

$$ \begin{array}{|l|c|c|} \hline \textbf{Metric} & \textbf{SC-LeGO-LOAM (\%)} & \textbf{LeGO-LOAM (\%)} \\ \hline \text{max} & 2.89 & 8.55 \\ \hline \text{mean} & 1.35 & 2.82 \\ \hline \text{median} & 1.36 & 3.34 \\ \hline \text{min} & 0.20 & 0.27 \\ \hline \text{rmse} & 1.46 & 3.34 \\ \hline \end{array} $$

Fig 1. Point Cloud Comparison on RIVERSIDE 03 Dataset

Fig 2. Point Cloud Comparison on KAIST 01 Dataset

The mapping results of three algorithms on the Kaist dataset are illustrated in Figure 1 and Figure 2.

Fig 3. LOAM on RIVERSIDE Dataset

Fig 4. LeGO-LOAM on RIVERSIDE Dataset

Fig 5. SC-LeGO-LOAM on RIVERSIDE Dataset

Output

Map Here are the map formed by the LOAM Algorithm on the MulRAN Dataset(KAIST track):

Autonomous Underwater Vehicle

Sun, 24 Nov 2019 00:00:00 +0000

Introduction

A student run team made AUV during undergrad. This project is made with collaboration over 16 members across Software Domain, Electronics, Mechanical and Corporate. I later had the chance to lead the team.

Projects | Rahil Bhowal

Attention-guided Off-Road Semantic Segmentation for Autonomous Navigation

Introduction

Method

Experiments

Evaluation and Metrics

Output

3D Reconstruction Using Structure From Motion (SfM)

Objective

Dataset

Problem Statement

Workflow

Output

Did you find this page helpful? Consider sharing it 🙌

Underwater Image Mosaicing

Introduction

Output

Real-Time Social Navigation Strategy using Deep Reinforcement Learning

Introduction

Experiment Setup

Approach

Videos

Search and Rescue Mission Using Turtlebot

Introduction

Method

Videos

Loam-Series Evaluation

Introduction

Evaluation and Metrics

Output

MORE VIDEOS WILL BE ADDED SOON

Did you find this page helpful? Consider sharing it 🙌

Autonomous Underwater Vehicle

Introduction

Video