greenhouse/README.md

# greenhouse

This repository contains code for detecting heat pipes in the greenhouse as well as estimating the pose of the pipes.

Platform: ROS 2, Humble, Ubuntu 22.04

## How to install dependencies??
- Install `git`
```
sudo apt install git
```
- Install `ROS 2 Humble` on `Ubuntu 22.04` by following https://docs.ros.org/en/foxy/Installation/Ubuntu-Install-Debians.html
- Install `colcon build` by following https://docs.ros.org/en/foxy/Tutorials/Beginner-CLI-Tools/Configuring-ROS2-Environment.html
```
sudo apt install python3-colcon-common-extensions
```
- Instal `pip` for python packages
```
sudo apt install python3-pip
```
- Now clone the repository
```
git clone https://tea.der-space.de/apoorva/greenhouse.git
```
- Install `ultralytics` for yolov3 package
```
pip install ultralytics
```
- For `yolov3_ros`, there are a bunch of other requirements. Go to `yolov3` folder and install using following commands:
```
cd ~/greenhouse/yolov3
pip install -r requirements.txt
```
- Go to `ros2_ws` inside `greenhouse`. Make sure you only have `src` folder.
```
cd ~/greenhouse/ros2_ws
ls
src
```
- Inside `ros2_ws` folder, start building individual packages in the below sequence to avoid errors.
```
colcon build --packages-select pipe_msgs
colcon build --packages-select pcl_ros
colcon build --allow-overriding pcl_ros
colcon build --packages-select pcl_conversions
colcon build --allow-overriding pcl_conversions
colcon build --packages-select find-pose
colcon build --packages-select yolov3_ros
. install/setup.bash
```
- The code should be ready to launch as explained in [How to run Live Detection?](#how-to-run-live-detection)
This section explains what each module is responsible for.
## perception_pcl
This module is responsible for providing `pcl_conversions` and `pcl_ros` modules in `ros 2`.
To build, run the following command:
```
cd ros2_ws/
colcon build --packages-select perception_pcl
. install/setup.bash
```

## pipe_msgs
This module contains ros msgs for storing information about the detected object's bounding box.
```
cd ros2_ws/
colcon build --packages-select pipe_msgs
. install/setup.bash
```
To check if msgs are built properly, run following command
```
ros2 interface show pipe_msgs/msg/BoundingBox
```
The output will be:
float64 probability
int64 xmin
int64 ymin
int64 xmax
int64 ymax
int16 id
string class_id


## find-pose
This ROS module is responsible for determining the position of the detected objects.
The following input/ros topics are needed:
- /rgb_img: RGB Image topic
- /camera_info: Camera calibration parameters topic
- /depth_img: Aligned depth image topic (aligned with rgb image)
- /bboxes: Bounding box of each detected object. (Comes from yolov3 detection module)
Output:
- TF: Transform between camera_link and detected_object frame.
How to build and run?
This package is dependent on custom `pcl_conversion` and `pcl_ros` module. Make sure you have built those before building this package.
```
colcon build --packages-select find-pose
. install/setup.bash
ros2 launch find-pose find-pose-node.launch.py
```
All the topics can be remapped in the launch file.

## yolov3_ros
This ROS module is responsible for detecting the pipes from rgb image topic and also syncing the depth, rgb and camera_info topics.
The following input/ros topics are needed:
- /camera/color/image_raw: RGB Image topic
- /camera/aligned_depth_to_color/image_raw: Aligned depth image topic (aligned with rgb image)
- /camera/color/camera_info: Camera calibration parameters topic

ROS Paramater Input:
- best_weights: String that is the path to the best weights file of yolov3 detection
Defaults: `'src/pipe_weights.pt'` inside `ros2_ws` folder

The following are the output topics:
- /detection_image: The RGB Image topic with bounding box drawn on it for visualization and debugging purpose
- /bboxes: Bounding box of each detected object.
- /rgb_img: Time Synced RGB Image topic
- /camera_info: Time Synced Camera calibration parameters topic
- /depth_img: Time Synced Aligned depth image topic (aligned with rgb image)

How to build and run?
```
colcon build --packages-select yolov3_ros
. install/setup.bash
cd greenhouse/ros2_ws/
ros2 launch yolov3_ros pipe_detection.launch.py
```
All the topics can be remapped in the launch file. The path to best_weights can also be changed inside the launch file.
Launch file is stored in `yolov3_ros/launch/`.

## yolov3
This module contains code for yolov3. This method is being used to train the model to detect pipes in the greenhouse.
- Ros bag was converted from ros 1 to ros 2 using rosbags module
- Ros 2 bag was played, convert_2_img was used to subscribe to images and all the images were saved as .jpeg in a folder.
- Images were labeled used labelImg.
- Create a `custom-yolov3.yaml` file that has information about number of classes, location for labels,images.

- Used google colab to run the yolov3 training.
- Upload the label and images to drive and Mount the google drive in python notebook
```
from google.colab import drive
drive.mount("/content/gdrive")
```
- Upload the yolov3 code and cd into the location of code
```
cd /content/gdrive/MyDrive/yolov3
```
- Run the training script
```
!python train.py --img 1280 --batch 16 --epochs 300 --data data/custom-yolov3.yaml --weights '' --cfg yolov3.yaml
```
- Important to note that we are not using any pre-trained weights.
- Once training is finished, a `run` folder is created with exp number that stores the best weights (a .pt file).
- This file is then used by detection to node to detect on new data. Update the image path and weights path to run detection.
```
!python detect.py --img 1280 --source ../pipe-dataset/validate/images/stereo_image103.jpeg --weights runs/train/exp14/weights/best.pt
```
- Once detect.py is finished, it create a new folder called `detect` inside `runs` folder that store the image with bounding box of detected object.
More details: https://github.com/ultralytics/yolov3

## convert_2_img
This ROS module converts the rosbag data to images for yolo training purpose etc.
- Make sure you have this module inside a ros workspace.
- Create folder called `stereo` inside convert_2_img module.
- Run following command to launch the node. Currently, this node listens for `/camera/color/image_raw` topic.
```
cd ros2_ws/src/convert_2_img
python3 convert_2_img/convert_to_img.py
```
- Play the rosbag in another terminal
```
ros2 bag play bag/bag.db
```
- Once bag has finished playing, the images will be stored inside `stereo` folder.

## labelImg
This module is used to label images for yolo. The pre-defined custom classes file was changed to use new labels. This file is stored in `cd labelImg/data/predefined_classes.txt`
To launch the gui, run
```
cd labelImg
python3 labelImg.py 
```
More details: https://github.com/heartexlabs/labelImg

## rosbags
This module convert rosbags from ros 1 to ros 2. 
```
git clone https://gitlab.com/ternaris/rosbags.git
cd rosbags
python -m venv venv
. venv/bin/activate

pip install -r requirements-dev.txt
pip install -e .

rosbags-convert ../single_depth_color_640x480.bag
```

## flann_based
Module that uses 3D model of the pipe to estimate pose. This method was not successful.

## yolov7
This module contains code for yolov7. This method was too heavy and didn't produce great results for detection.

## darknet_ros2
This module was provides interface to run neural network as rosnodes. The purpose was to use yolo model as ros node but this method was not successful

## darknet_vendor
This module was needed to build darknet_ros2.

# How to run live detection?
Once you have succefully built the ROS Modules specifically, `yolov3_ros` and `find-pose` along with dependencies like `pipe_msgs` and `perception_ros`, `perception_pcl`,you can do the following: 
Assumption:
You have the following required data in forms of ros topic:
- /camera/color/image_raw: RGB Image topic
- /camera/aligned_depth_to_color/image_raw: Aligned depth image topic (aligned with rgb image)
- /camera/color/camera_info: Camera calibration parameters topic
How do you get this data?
- This data can come from a pre recorded Rosbag. If this is the case, do following:
```
ros2 bag play bag-folder/bag-name.db3
```
- This data can come directly from camera's (D455/ZED2I) ROS Node: Launch your node in a terminal. 
- If your camera topics have different names than the default topic names mentioned above, update/remap ONLY the `pipe_detection.launch.py` script stored in `ros2_ws/src/yolov3_ros/launch` folder.
- Once you have updated the launch file, build the code again using `colcon build --packages-select yolov3_ros` and proceedas mentioned below.

Steps to run:
- Open a terminal.
- Run `$ cd greenhouse/ros2_ws/` and `$ . install/setup.bash`.
- Launch node for object detection:
```
ros2 launch yolov3_ros pipe_detection.launch.py
```
- This node will output two topics: `/bboxes` and `/detection_image`.
- Make sure that you run this launch file from `greenhouse/ros2_ws/` folder since the path of weights is relative (`/src/pipe_weights.pt`) inside the launch file.
- Launch node for pose estimation:
```
ros2 launch find-pose find-pose-node.launch.py
```
- This node will output TF topics between `/camera_link` and `/${detected_object_name}`.
- Open RVIZ by running `$ rviz2`. Change `fixed frame` from `map` to `camera_link`.
    - Go to `Add` button. Under `By Display type`, select `tf`. Once tf is added, select required frames like `camera_link` and `l_trail` to see the tfs.
    - To see the current image with detected object, Go to `Add` Button. Under `By Topic`, select topci called `detection_image`.
    -You can add other topics as per the need and topic names. 
- You can open launch files to update/remap topic name if different camera is being used. 
- You can also update ros parameter from launch file. Currently, the `pipe_weights.pt` file is the one used. This file can be changed and you can update the parameter name `best_weights` inside the `pipe_detection.launch.py` file.