A seemingly straightforward idea for robot control involves using depth, and object detection, to form a rough model of the environment.
After failed attempts to create our own stereo camera using two monocular cameras, we eventually decided to buy a commercial product instead, The Intel depth camera, D455.
After a first round of running a COCO trained MobileSSDv2 object detection network in Tensorflow 2 Lite, on the colour images obtained from the realsense camera, on the Jetson Nano, the results were just barely acceptable (~2 FPS) for a localhost stream, and totally unacceptable (~0.25 FPS) served as JPEG over HTTP, to a browser on the network.
Looking at the options, the only feasible solution was to redo the network using TensorRT, the NVIDIA-specific, quantized (16 bit on the Nano, 8 bit on the NX/AGX) neural network framework. The other solution involved investigating options other than simple JPEG compression over HTTP, such as RTSP and WebRTC.
The difficult part was setting up the environment. We used the NVIDIA detectnet code, adapted to take the realsense camera images as input, and to display the distance to the objects. An outdated example was found at CAVEDU robotics blog/github. Fixed up below.
#!/usr/bin/python3
import jetson_inference
import jetson_utils
import argparse
import sys
import os
import cv2
import re
import numpy as np
import io
import time
import json
import random
import pyrealsense2 as rs
from jetson_inference import detectNet
from jetson_utils import videoSource, videoOutput, logUsage, cudaFromNumpy, cudaAllocMapped, cudaConvertColor
parser = argparse.ArgumentParser(description="Locate objects in a live camera stream using an object detection DNN.",
formatter_class=argparse.RawTextHelpFormatter, epilog=jetson_utils.logUsage())
parser.add_argument("--network", type=str, default="ssd-mobilenet-v2",
help="pre-trained model to load (see below for options)")
parser.add_argument("--threshold", type=float, default=0.5,
help="minimum detection threshold to use")
parser.add_argument("--width", type=int, default=640,
help="set width for image")
parser.add_argument("--height", type=int, default=480,
help="set height for image")
opt = parser.parse_known_args()[0]
# load the object detection network
net = detectNet(opt.network, sys.argv, opt.threshold)
# Configure depth and color streams
pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, opt.width, opt.height, rs.format.z16, 30)
config.enable_stream(rs.stream.color, opt.width, opt.height, rs.format.bgr8, 30)
# Start streaming
pipeline.start(config)
press_key = 0
while (press_key==0):
# Wait for a coherent pair of frames: depth and color
frames = pipeline.wait_for_frames()
depth_frame = frames.get_depth_frame()
color_frame = frames.get_color_frame()
if not depth_frame or not color_frame:
continue
# Convert images to numpy arrays
depth_image = np.asanyarray(depth_frame.get_data())
show_img = np.asanyarray(color_frame.get_data())
# convert to CUDA (cv2 images are numpy arrays, in BGR format)
bgr_img = cudaFromNumpy(show_img, isBGR=True)
# convert from BGR -> RGB
img = cudaAllocMapped(width=bgr_img.width,height=bgr_img.height,format='rgb8')
cudaConvertColor(bgr_img, img)
# detect objects in the image (with overlay)
detections = net.Detect(img)
for num in range(len(detections)) :
score = round(detections[num].Confidence,2)
box_top=int(detections[num].Top)
box_left=int(detections[num].Left)
box_bottom=int(detections[num].Bottom)
box_right=int(detections[num].Right)
box_center=detections[num].Center
label_name = net.GetClassDesc(detections[num].ClassID)
point_distance=0.0
for i in range (10):
point_distance = point_distance + depth_frame.get_distance(int(box_center[0]),int(box_center[1]))
point_distance = np.round(point_distance / 10, 3)
distance_text = str(point_distance) + 'm'
cv2.rectangle(show_img,(box_left,box_top),(box_right,box_bottom),(255,0,0),2)
cv2.line(show_img,
(int(box_center[0])-10, int(box_center[1])),
(int(box_center[0]+10), int(box_center[1])),
(0, 255, 255), 3)
cv2.line(show_img,
(int(box_center[0]), int(box_center[1]-10)),
(int(box_center[0]), int(box_center[1]+10)),
(0, 255, 255), 3)
cv2.putText(show_img,
label_name + ' ' + distance_text,
(box_left+5,box_top+20),cv2.FONT_HERSHEY_SIMPLEX,0.4,
(0,255,255),1,cv2.LINE_AA)
cv2.putText(show_img,
"{:.0f} FPS".format(net.GetNetworkFPS()),
(int(opt.width*0.8), int(opt.height*0.1)),
cv2.FONT_HERSHEY_SIMPLEX,1,
(0,255,255),2,cv2.LINE_AA)
display = cv2.resize(show_img,(int(opt.width*1.5),int(opt.height*1.5)))
cv2.imshow('Detecting...',display)
keyValue=cv2.waitKey(1)
if keyValue & 0xFF == ord('q'):
press_key=1
cv2.destroyAllWindows()
pipeline.stop()
Assuming you have a good cmake version and cuda is available (if nvcc doesn’t work, you need to configure linker paths, check this link)… (if you have a cmake version around ~ 3.22-3.24 or so, you need an older one)… prerequisite sudo apt-get install libssl-dev also required.
The hard part was actually setting up the Realsense python bindings.
Clone the repo…
git clone https://github.com/IntelRealSense/librealsense.git
The trick being, to request the python bindings, and cuda, during the cmake phase. Note that often, none of this works. Some tips include…
sudo apt-get install xorg-dev libglu1-mesa-dev
and changing PYTHON to Python
mkdir build
cd build
cmake ../ -DBUILD_PYTHON_BINDINGS:bool=true -DPYTHON_EXECUTABLE=/usr/bin/python3 -DCMAKE_BUILD_TYPE=release -DBUILD_EXAMPLES=true -DBUILD_GRAPHICAL_EXAMPLES=true -DBUILD_WITH_CUDA:bool=true
The above worked on Jetpack 4.6.1, while the below worked on Jetpack 5.0.2
cmake ../ -DBUILD_PYTHON_BINDINGS:bool=true -DPython_EXECUTABLE=/usr/bin/python3.8 -DCMAKE_BUILD_TYPE=release -DBUILD_EXAMPLES=true -DBUILD_GRAPHICAL_EXAMPLES=true -DBUILD_WITH_CUDA:bool=true -DPYTHON_INCLUDE_DIRS=/usr/include/python3.8 -DPython_LIBRARIES=/usr/lib/aarch64-linux-gnu/libpython3.8.so
(and sudo make install)
Update the python path
export PYTHONPATH=$PYTHONPATH:/usr/local/lib
(or a specific python if you have more than one)
if installed in /usr/lib, change accordingly
Check that the folder is in the correct location (it isn't, after following official instructions).
./usr/local/lib/python3.6/dist-packages/pyrealsense2/
Check that the shared object files (.so) are in the right place:
chicken@chicken:/usr/local/lib$ ls
cmake libjetson-inference.so librealsense2-gl.so.2.50 librealsense2.so.2.50 pkgconfig
libfw.a libjetson-utils.so librealsense2-gl.so.2.50.0 librealsense2.so.2.50.0 python2.7
libglfw3.a librealsense2-gl.so librealsense2.so librealsense-file.a python3.6
If it can't find 'pipeline', it means you need to copy the missing __init__.py file.
sudo cp ./home/chicken/librealsense/wrappers/python/pyrealsense2/__init__.py ./usr/local/lib/python3.6/dist-packages/pyrealsense2/
Some extra things to do,
sudo cp 99-realsense-libusb.rules /etc/udev/rules.d/
Eventually, I am able to run the inference on the realsense camera, at an apparent 25 FPS, on the localhost, drawing to an OpenGL window.
I also developed a Dockerfile, for the purpose, which benefits from an updated Pytorch version, but various issues were encountered, making a bare-metal install far simpler, ultimately. Note that building jetson-inference, and Realsense SDK on the Nano require increasing your swap size, beyond the 2GB standard. Otherwise, the Jetson freezes once memory paging leads to swap death.
Anyway, since the objective is remote human viewing, (while providing depth information for the robot to use), the next step will require some more tests, to find a suitable option.
The main blocker is the power usage limitations on the Jetson Nano. I can’t seem to run Wifi and the camera at the same time. According to the tegrastats utility, the POM_5V_IN usage goes over the provided 4A, under basic usage. There are notes saying that 3A can be provided to 2 of the 5V GPIO pins, in order to get 6A total input. That might end up being necessary.
Initial investigation into serving RTSP resulted in inferior, and compressed results compared to a simple python server streaming image by image. The next investigation will be into WebRTC options, which are supposedly the current state of the art, for browser based video streaming. I tried aiortc, and momo, so far, both failed on the Nano.
I’ve decided to try on the Xavier NX, too, just to replicate the experiment, and see how things change. The Xavier has some higher wattage settings, and the wifi is internal, so worth a try. Also, upgraded to Jetpack 5.0.2, which was a gamble. Thought surely it would be better than upgrading to a 5.0.1 dev preview, but none of their official products support 5.0.2 yet, so there will likely be much pain involved. On the plus side, python 3.8 is standard, so some libraries are back on the menu.
On the Xavier, we’re getting 80 FPS, compared to 25 FPS on the Nano. Quite an upgrade. Also, able to run wifi and realsense at the same time.
Looks like a success. Getting multiple frames per second with about a second of lag over the network.
