greenhouse/README.md

greenhouse

This repository contains code for detecting heat pipes in the greenhouse as well as estimating the pose of the pipes.

Platform: ROS 2, Humble, Ubuntu 22.04

How to build the workspace?

How to install dependencies??

This section explains what each module is responsible for.

pipe_msgs

This module contains ros msgs for storing information about the detected object's bounding box.

$ cd ros2_ws/
$ colcon build --packages-select pipe_msgs
$ . install/setup.bash

To check if msgs are built properly, run following command

$ ros2 interface show pipe_msgs/msg/BoundingBox

The output will be:
float64 probability
int64 xmin
int64 ymin
int64 xmax
int64 ymax
int16 id
string class_id

find-pose

This ROS module is responsible for determining the position of the detected objects. The following input/ros topics are needed:

• /rgb_img: RGB Image topic
• /camera_info: Camera calibration parameters topic
• /depth_img: Aligned depth image topic (aligned with rgb image)
• /bboxes: Bounding box of each detected object. (Comes from yolov3 detection module) Output:
• TF: Transform between camera_link and detected_object frame. How to build and run? This package is dependent on custom pcl_conversion and pcl_ros module. Make sure you have built those before building this package.

$ colcon build --packages-select find-pose
$ ros2 launch find-pose find-pose-node.launch.py

All the topics can be remapped in the launch file.

yolov3_ros

This ROS module is responsible for detecting the pipes from rgb image topic and also syncing the depth, rgb and camera_info topics. The following input/ros topics are needed:

• /camera/color/image_raw: RGB Image topic
• /camera/aligned_depth_to_color/image_raw: Aligned depth image topic (aligned with rgb image)
• /camera/color/camera_info: Camera calibration parameters topic

ROS Paramater Input:

• best_weights: String that is the path to the best weights file of yolov3 detection Defaults: 'src/pipe_weights.pt'

The following are the output topics:

• /detection_image: The RGB Image topic with bounding box drawn on it for visualization and debugging purpose
• /bboxes: Bounding box of each detected object.
• /rgb_img: Time Synced RGB Image topic
• /camera_info: Time Synced Camera calibration parameters topic
• /depth_img: Time Synced Aligned depth image topic (aligned with rgb image)

How to build and run?

$ colcon build --packages-select yolov3_ros
$ ros2 launch yolov3_ros pipe_detection.launch.py

All the topics can be remapped in the launch file. The path to best_weights can also be changed inside the launch file. Launch file is stored in yolov3_ros/launch/.

yolov3

This module contains code for yolov3. This method is being used to train the model to detect pipes in the greenhouse.

• Ros bag was converted from ros 1 to ros 2 using rosbags module

• Ros 2 bag was played, convert_2_img was used to subscribe to images and all the images were saved as .jpeg in a folder.

• Images were labeled used labelImg.

• Create a custom-yolov3.yaml file that has information about number of classes, location for labels,images.

• Used google colab to run the yolov3 training.

• Upload the label and images to drive and Mount the google drive in python notebook

from google.colab import drive
drive.mount("/content/gdrive")

• Upload the yolov3 code and cd into the location of code

%cd /content/gdrive/MyDrive/yolov3

• Run the training script

!python train.py --img 1280 --batch 16 --epochs 300 --data data/custom-yolov3.yaml --weights '' --cfg yolov3.yaml

• Important to note that we are not using any pre-trained weights.
• Once training is finished, a run folder is created with exp number that stores the best weights (a .pt file).
• This file is then used by detection to node to detect on new data. Update the image path and weights path to run detection.

!python detect.py --img 1280 --source ../pipe-dataset/validate/images/stereo_image103.jpeg --weights runs/train/exp14/weights/best.pt

• Once detect.py is finished, it create a new folder called detect inside runs folder that store the image with bounding box of detected object. More details: https://github.com/ultralytics/yolov3

convert_2_img

This ROS module converts the rosbag data to images for yolo training purpose etc.

• Make sure you have this module inside a ros workspace.
• Create folder called stereo inside convert_2_img module.
• Run following command to launch the node. Currently, this node listens for /camera/color/image_raw topic.

$ cd ros2_ws/src/convert_2_img
$ python3 convert_2_img/convert_to_img.py

• Play the rosbag in another terminal

$ ros2 bag play bag/bag.db

• Once bag has finished playing, the images will be stored inside stereo folder.

labelImg

This module is used to label images for yolo. The pre-defined custom classes file was changed to use new labels. This file is stored in cd labelImg/data/predefined_classes.txt To launch the gui, run

$ cd labelImg
$ python3 labelImg.py 

More details: https://github.com/heartexlabs/labelImg

rosbags

This module convert rosbags from ros 1 to ros 2.

git clone https://gitlab.com/ternaris/rosbags.git
cd rosbags
python -m venv venv
. venv/bin/activate

pip install -r requirements-dev.txt
pip install -e .

rosbags-convert ../single_depth_color_640x480.bag

perception_pcl

pcl_conversions

pcl_ros

flann_based

Module that uses 3D model of the pipe to estimate pose. This method was not successful.

yolov7

This module contains code for yolov7. This method was too heavy and didn't produce great results for detection.

darknet_ros2

This module was provides interface to run neural network as rosnodes. The purpose was to use yolo model as ros node but this method was not successful

darknet_vendor

This module was needed to build darknet_ros2.