# greenhouse This repository contains code for detecting heat pipes in the greenhouse as well as estimating the pose of the pipes. Platform: ROS 2, Humble, Ubuntu 22.04 How to build the workspace? How to install dependencies?? This section explains what each module is responsible for. ## pipe_msgs This module contains ros msgs for storing information about the detected object's bounding box. ``` $ cd ros2_ws/ $ colcon build --packages-select pipe_msgs $ . install/setup.bash ``` To check if msgs are built properly, run following command ``` $ ros2 interface show pipe_msgs/msg/BoundingBox The output will be: float64 probability int64 xmin int64 ymin int64 xmax int64 ymax int16 id string class_id ``` ## find-pose This ROS module is responsible for determining the position of the detected objects. The following input/ros topics are needed: - /rgb_img: RGB Image topic - /camera_info: Camera calibration parameters topic - /depth_img: Aligned depth image topic (aligned with rgb image) - /bboxes: Bounding box of each detected object. (Comes from yolov3 detection module) Output: - TF: Transform between camera_link and detected_object frame. How to build and run? This package is dependent on custom `pcl_conversion` and `pcl_ros` module. Make sure you have built those before building this package. ``` $ colcon build --packages-select find-pose $ ros2 launch find-pose find-pose-node.launch.py ``` All the topics can be remapped in the launch file. ## yolov3_ros This ROS module is responsible for detecting the pipes from rgb image topic and also syncing the depth, rgb and camera_info topics. The following input/ros topics are needed: - /camera/color/image_raw: RGB Image topic - /camera/aligned_depth_to_color/image_raw: Aligned depth image topic (aligned with rgb image) - /camera/color/camera_info: Camera calibration parameters topic ROS Paramater Input: - best_weights: String that is the path to the best weights file of yolov3 detection Defaults: `'src/pipe_weights.pt'` The following are the output topics: - /detection_image: The RGB Image topic with bounding box drawn on it for visualization and debugging purpose - /bboxes: Bounding box of each detected object. - /rgb_img: Time Synced RGB Image topic - /camera_info: Time Synced Camera calibration parameters topic - /depth_img: Time Synced Aligned depth image topic (aligned with rgb image) How to build and run? ``` $ colcon build --packages-select yolov3_ros $ ros2 launch yolov3_ros pipe_detection.launch.py ``` All the topics can be remapped in the launch file. The path to best_weights can also be changed inside the launch file. Launch file is stored in `yolov3_ros/launch/`. ## yolov3 This module contains code for yolov3. This method is being used to train the model to detect pipes in the greenhouse. - Ros bag was converted from ros 1 to ros 2 using rosbags module - Ros 2 bag was played, convert_2_img was used to subscribe to images and all the images were saved as .jpeg in a folder. - Images were labeled used labelImg. - Create a `custom-yolov3.yaml` file that has information about number of classes, location for labels,images. - Used google colab to run the yolov3 training. - Upload the label and images to drive and Mount the google drive in python notebook ``` from google.colab import drive drive.mount("/content/gdrive") ``` - Upload the yolov3 code and cd into the location of code ``` %cd /content/gdrive/MyDrive/yolov3 ``` - Run the training script ``` !python train.py --img 1280 --batch 16 --epochs 300 --data data/custom-yolov3.yaml --weights '' --cfg yolov3.yaml ``` - Important to note that we are not using any pre-trained weights. - Once training is finished, a `run` folder is created with exp number that stores the best weights (a .pt file). - This file is then used by detection to node to detect on new data. Update the image path and weights path to run detection. ``` !python detect.py --img 1280 --source ../pipe-dataset/validate/images/stereo_image103.jpeg --weights runs/train/exp14/weights/best.pt ``` - Once detect.py is finished, it create a new folder called `detect` inside `runs` folder that store the image with bounding box of detected object. More details: https://github.com/ultralytics/yolov3 ## convert_2_img This ROS module converts the rosbag data to images for yolo training purpose etc. - Make sure you have this module inside a ros workspace. - Create folder called `stereo` inside convert_2_img module. - Run following command to launch the node. Currently, this node listens for `/camera/color/image_raw` topic. ``` $ cd ros2_ws/src/convert_2_img $ python3 convert_2_img/convert_to_img.py ``` - Play the rosbag in another terminal ``` $ ros2 bag play bag/bag.db ``` - Once bag has finished playing, the images will be stored inside `stereo` folder. ## labelImg This module is used to label images for yolo. The pre-defined custom classes file was changed to use new labels. This file is stored in `cd labelImg/data/predefined_classes.txt` To launch the gui, run ``` $ cd labelImg $ python3 labelImg.py ``` More details: https://github.com/heartexlabs/labelImg ## rosbags This module convert rosbags from ros 1 to ros 2. ``` git clone https://gitlab.com/ternaris/rosbags.git cd rosbags python -m venv venv . venv/bin/activate pip install -r requirements-dev.txt pip install -e . rosbags-convert ../single_depth_color_640x480.bag ``` ## perception_pcl ### pcl_conversions ### pcl_ros ## flann_based Module that uses 3D model of the pipe to estimate pose. This method was not successful. ## yolov7 This module contains code for yolov7. This method was too heavy and didn't produce great results for detection. ## darknet_ros2 This module was provides interface to run neural network as rosnodes. The purpose was to use yolo model as ros node but this method was not successful ## darknet_vendor This module was needed to build darknet_ros2.