Real-time voxel based 3D semantic mapping with a hand held RGB-D camera [SLAM Project]

# Project - [Real-time voxel based 3D semantic mapping with a hand held RGB-D camera](https://drive.google.com/file/d/1Uk5YLMyoyMGMkE8FtJCNrZp_brGh5z3M/view) # Source Code - Official github: [floatlazer/semantic_slam](https://github.com/floatlazer/semantic_slam) - My customized version: [yubaoliu/semantic_slam_floatlazer](https://github.com/yubaoliu/semantic_slam_floatlazer)  # Demo # Tutorials (in Chinese Language) - [语义建图学习（1-4）floatlazer-semantic-slam项目工程简介及代码分析](https://www.bilibili.com/video/BV1WU4y1s7Rx/) # Compile ## Get source code - clone project into ROS workspace, e.g. catkin_ws/src ## compile ORB_SLAM2 ```sh cd semantic_slam_floatlazer cd ORB_SLAM2 ../build.sh ``` ## Comple ROS wrapper ```sh catkin_make ``` ## Install Python dependencies ```sh setuptools>=41.0.0 numpy>=1.15 scipy Pillow cython opencv-python==3.3.1.11 matplotlib scikit-image tensorflow-gpu==1.13.1 keras==2.0.8 h5py imageio==2.6.1 imgaug pandas future torch torchvision protobuf IPython[all] ``` Note: - you can remove keras related packages - Install them using "pip install -r [FILE_NAME].txt" # How to run - Modify parames ```sh cd /semantic_slam_floatlazer/semantic_slam/params vim semantic_cloud.yaml ``` You have to set model_path. E.g. ```sh model_path: "/home/.../Dataset/semantic_slam/pspnet_50_ade20k.pth" ``` - Get CNN models - [Model trained on ade20k dataset](https://drive.google.com/file/d/1u_BEWdVIYiDnpVmAxwME1z3rnWWkjxm5/view?usp=sharing) - [Model fine tuned on SUNRGBD dataset](https://drive.google.com/file/d/1t26t2VHNOzmjH-0lDTdYzXBACOV_4-eL/view?usp=sharing) - Run launch file ```sh roslaunch floatlazer_semantic_slam semantic_mapping.launch ``` For TUM dataset (TUM bagfile), run ```sh roslaunch floatlazer_semantic_slam semantic_mapping_tum.launch ``` - Run bag file For example: ```sh rosbag play --clock demo.bag ``` [download demo.bag](https://drive.google.com/file/d/1j12c_Fruu-ylO1FHYC4sbmlG9IutYJQg/view?usp=sharing) # Possible Errors ## ColorOccupancyGrid Issue: ```sh [ERROR] [1595235776.415659314]: PluginlibFactory: The plugin for class 'octomap_rviz_plugin/ColorOccupancyGrid' failed to load. Error: According to the loaded plugin descriptions the class octomap_rviz_plugin/ColorOccupancyGrid with base class type rviz::Display does not exist. Declared types are rviz/Axes rviz/Camera rviz/DepthCloud rviz/Effort rviz/FluidPressure rviz/Grid rviz/GridCells rviz/Illuminance rviz/Image rviz/InteractiveMarkers rviz/LaserScan rviz/Map rviz/Marker rviz/MarkerArray rviz/Odometry rviz/Path rviz/PointCloud rviz/PointCloud2 rviz/PointStamped rviz/Polygon rviz/Pose rviz/PoseArray rviz/PoseWithCovariance rviz/Range rviz/RelativeHumidity rviz/RobotModel rviz/TF rviz/Temperature rviz/WrenchStamped rviz_plugin_tutorials/Imu ``` Solution: ```sh sudo apt install ros-kinetic-octomap-rviz-plugins ```