169 lines
5.8 KiB
Markdown
169 lines
5.8 KiB
Markdown
# greenhouse
|
|
|
|
This repository contains code for detecting heat pipes in the greenhouse as well as estimating the pose of the pipes.
|
|
|
|
Platform: ROS 2, Humble, Ubuntu 22.04
|
|
|
|
How to build the workspace?
|
|
|
|
How to install dependencies??
|
|
|
|
|
|
This section explains what each module is responsible for.
|
|
|
|
## pipe_msgs
|
|
This module contains ros msgs for storing information about the detected object's bounding box.
|
|
```
|
|
$ cd ros2_ws/
|
|
$ colcon build --packages-select pipe_msgs
|
|
$ . install/setup.bash
|
|
```
|
|
To check if msgs are built properly, run following command
|
|
```
|
|
$ ros2 interface show pipe_msgs/msg/BoundingBox
|
|
|
|
The output will be:
|
|
float64 probability
|
|
int64 xmin
|
|
int64 ymin
|
|
int64 xmax
|
|
int64 ymax
|
|
int16 id
|
|
string class_id
|
|
```
|
|
|
|
## find-pose
|
|
This ROS module is responsible for determining the position of the detected objects.
|
|
The following input/ros topics are needed:
|
|
- /rgb_img: RGB Image topic
|
|
- /camera_info: Camera calibration parameters topic
|
|
- /depth_img: Aligned depth image topic (aligned with rgb image)
|
|
- /bboxes: Bounding box of each detected object. (Comes from yolov3 detection module)
|
|
Output:
|
|
- TF: Transform between camera_link and detected_object frame.
|
|
How to build and run?
|
|
This package is dependent on custom `pcl_conversion` and `pcl_ros` module. Make sure you have built those before building this package.
|
|
```
|
|
$ colcon build --packages-select find-pose
|
|
$ ros2 launch find-pose find-pose-node.launch.py
|
|
```
|
|
All the topics can be remapped in the launch file.
|
|
|
|
## yolov3_ros
|
|
This ROS module is responsible for detecting the pipes from rgb image topic and also syncing the depth, rgb and camera_info topics.
|
|
The following input/ros topics are needed:
|
|
- /camera/color/image_raw: RGB Image topic
|
|
- /camera/aligned_depth_to_color/image_raw: Aligned depth image topic (aligned with rgb image)
|
|
- /camera/color/camera_info: Camera calibration parameters topic
|
|
|
|
ROS Paramater Input:
|
|
- best_weights: String that is the path to the best weights file of yolov3 detection
|
|
Defaults: `'src/pipe_weights.pt'`
|
|
|
|
The following are the output topics:
|
|
- /detection_image: The RGB Image topic with bounding box drawn on it for visualization and debugging purpose
|
|
- /bboxes: Bounding box of each detected object.
|
|
- /rgb_img: Time Synced RGB Image topic
|
|
- /camera_info: Time Synced Camera calibration parameters topic
|
|
- /depth_img: Time Synced Aligned depth image topic (aligned with rgb image)
|
|
|
|
How to build and run?
|
|
```
|
|
$ colcon build --packages-select yolov3_ros
|
|
$ ros2 launch yolov3_ros pipe_detection.launch.py
|
|
```
|
|
All the topics can be remapped in the launch file. The path to best_weights can also be changed inside the launch file.
|
|
Launch file is stored in `yolov3_ros/launch/`.
|
|
|
|
## yolov3
|
|
This module contains code for yolov3. This method is being used to train the model to detect pipes in the greenhouse.
|
|
- Ros bag was converted from ros 1 to ros 2 using rosbags module
|
|
- Ros 2 bag was played, convert_2_img was used to subscribe to images and all the images were saved as .jpeg in a folder.
|
|
- Images were labeled used labelImg.
|
|
- Create a `custom-yolov3.yaml` file that has information about number of classes, location for labels,images.
|
|
|
|
- Used google colab to run the yolov3 training.
|
|
- Upload the label and images to drive and Mount the google drive in python notebook
|
|
```
|
|
from google.colab import drive
|
|
drive.mount("/content/gdrive")
|
|
```
|
|
- Upload the yolov3 code and cd into the location of code
|
|
```
|
|
%cd /content/gdrive/MyDrive/yolov3
|
|
```
|
|
- Run the training script
|
|
```
|
|
!python train.py --img 1280 --batch 16 --epochs 300 --data data/custom-yolov3.yaml --weights '' --cfg yolov3.yaml
|
|
```
|
|
- Important to note that we are not using any pre-trained weights.
|
|
- Once training is finished, a `run` folder is created with exp number that stores the best weights (a .pt file).
|
|
- This file is then used by detection to node to detect on new data. Update the image path and weights path to run detection.
|
|
```
|
|
!python detect.py --img 1280 --source ../pipe-dataset/validate/images/stereo_image103.jpeg --weights runs/train/exp14/weights/best.pt
|
|
```
|
|
- Once detect.py is finished, it create a new folder called `detect` inside `runs` folder that store the image with bounding box of detected object.
|
|
More details: https://github.com/ultralytics/yolov3
|
|
|
|
## convert_2_img
|
|
This ROS module converts the rosbag data to images for yolo training purpose etc.
|
|
- Make sure you have this module inside a ros workspace.
|
|
- Create folder called `stereo` inside convert_2_img module.
|
|
- Run following command to launch the node. Currently, this node listens for `/camera/color/image_raw` topic.
|
|
```
|
|
$ cd ros2_ws/src/convert_2_img
|
|
$ python3 convert_2_img/convert_to_img.py
|
|
```
|
|
- Play the rosbag in another terminal
|
|
```
|
|
$ ros2 bag play bag/bag.db
|
|
```
|
|
- Once bag has finished playing, the images will be stored inside `stereo` folder.
|
|
|
|
## labelImg
|
|
This module is used to label images for yolo. The pre-defined custom classes file was changed to use new labels. This file is stored in `cd labelImg/data/predefined_classes.txt`
|
|
To launch the gui, run
|
|
```
|
|
$ cd labelImg
|
|
$ python3 labelImg.py
|
|
```
|
|
More details: https://github.com/heartexlabs/labelImg
|
|
|
|
## rosbags
|
|
This module convert rosbags from ros 1 to ros 2.
|
|
```
|
|
git clone https://gitlab.com/ternaris/rosbags.git
|
|
cd rosbags
|
|
python -m venv venv
|
|
. venv/bin/activate
|
|
|
|
pip install -r requirements-dev.txt
|
|
pip install -e .
|
|
|
|
rosbags-convert ../single_depth_color_640x480.bag
|
|
```
|
|
|
|
## perception_pcl
|
|
|
|
|
|
### pcl_conversions
|
|
|
|
### pcl_ros
|
|
|
|
|
|
## flann_based
|
|
Module that uses 3D model of the pipe to estimate pose. This method was not successful.
|
|
|
|
## yolov7
|
|
This module contains code for yolov7. This method was too heavy and didn't produce great results for detection.
|
|
|
|
## darknet_ros2
|
|
This module was provides interface to run neural network as rosnodes. The purpose was to use yolo model as ros node but this method was not successful
|
|
|
|
## darknet_vendor
|
|
This module was needed to build darknet_ros2.
|
|
|
|
|
|
|
|
|