Real-time voxel based 3D semantic mapping with a hand held RGB-D camera [SLAM Project]

# Project - [Real-time voxel based 3D semantic mapping with a hand held RGB-D camera]( # Source Code - Official github: [floatlazer/semantic_slam]( - My customized version: [yubaoliu/semantic_slam_floatlazer]( semantic mapping ![Architechture]( # Demo # Tutorials (in Chinese Language) - [语义建图学习(1-4)floatlazer-semantic-slam项目工程简介及代码分析]( # Compile ## Get source code - clone project into ROS workspace, e.g. catkin_ws/src ## compile ORB_SLAM2 ```sh cd semantic_slam_floatlazer cd ORB_SLAM2 ../ ``` ## Comple ROS wrapper ```sh catkin_make ``` ## Install Python dependencies ```sh setuptools>=41.0.0 numpy>=1.15 scipy Pillow cython opencv-python== matplotlib scikit-image tensorflow-gpu==1.13.1 keras==2.0.8 h5py imageio==2.6.1 imgaug pandas future torch torchvision protobuf IPython[all] ``` Note: - you can remove keras related packages - Install them using "pip install -r [FILE_NAME].txt" # How to run - Modify parames ```sh cd /semantic_slam_floatlazer/semantic_slam/params vim semantic_cloud.yaml ``` You have to set model_path. E.g. ```sh model_path: "/home/.../Dataset/semantic_slam/pspnet_50_ade20k.pth" ``` - Get CNN models - [Model trained on ade20k dataset]( - [Model fine tuned on SUNRGBD dataset]( - Run launch file ```sh roslaunch floatlazer_semantic_slam semantic_mapping.launch ``` For TUM dataset (TUM bagfile), run ```sh roslaunch floatlazer_semantic_slam semantic_mapping_tum.launch ``` - Run bag file For example: ```sh rosbag play --clock demo.bag ``` [download demo.bag]( # Possible Errors ## ColorOccupancyGrid Issue: ```sh [ERROR] [1595235776.415659314]: PluginlibFactory: The plugin for class 'octomap_rviz_plugin/ColorOccupancyGrid' failed to load. Error: According to the loaded plugin descriptions the class octomap_rviz_plugin/ColorOccupancyGrid with base class type rviz::Display does not exist. Declared types are rviz/Axes rviz/Camera rviz/DepthCloud rviz/Effort rviz/FluidPressure rviz/Grid rviz/GridCells rviz/Illuminance rviz/Image rviz/InteractiveMarkers rviz/LaserScan rviz/Map rviz/Marker rviz/MarkerArray rviz/Odometry rviz/Path rviz/PointCloud rviz/PointCloud2 rviz/PointStamped rviz/Polygon rviz/Pose rviz/PoseArray rviz/PoseWithCovariance rviz/Range rviz/RelativeHumidity rviz/RobotModel rviz/TF rviz/Temperature rviz/WrenchStamped rviz_plugin_tutorials/Imu ``` Solution: ```sh sudo apt install ros-kinetic-octomap-rviz-plugins ```

No comments