This is turning into one of the most oddly complicated sub-tasks of making a robot.
Turns out you need an intuition for epipolar geometry, to understand what it’s trying to do.
I’ve been battling this for weeks or months, on-and-off, and I must get it working soon. This week. But I know a week won’t be enough. This shit is hairy. Here’s eleven tips on StackOverflow. And here’s twelve tips from another guru.
The only articles that look like they got it working, have printed out these A2 sized chessboards. It’s ridiculous. Children in Africa don’t have A2 sized chessboard printouts!
Side notes: I’d like to mention that there also seem to be some interesting developments, in the direction of not needing perfect gigantic chessboards to calibrate your cameras, which in turn led down a rabbit hole, into a galaxy of projects packaged together with their own robotics software philosophies, and so on. Specifically, I found this OpenHSML project. They packaged their project using PID framework. This apparently standardises the build process, in some way. Clicking on the links to officially released frameworks using PID, leads to RPC, ethercatcpp, hardio, RKCL, and a whole world of sensor fusion topics, including a recent IEEE conference on MultiSensor Fusion and Integration (MFI2021.org).
So, back to chessboards. It’s clearly more of an art than a science.
Let’s, for the sake of argument, use the code at this OpenCV URL on the topic.
Let’s fire up the engines. I really want to close some R&D tabs, but we’re down to about 30. I’ll try reduce, to 20, while we go through the journey, together, dear internet reader, or future intelligence. Step One. Install OpenCV. I’m using the ISAAC ROS common docker with various modifications. (Eg. installing matplotlib, Jupyter Lab)
cd ~/your_ws/src/isaac_ros_common ./scripts/run_dev.sh ~/your_ws/
python3
print(cv2.version)
4.5.0
So this version of the docs should work, surely. Let’s start up Jupyter because it looks like it wants it.
python3 example_1.py
example_1.py:61: UserWarning: Matplotlib is currently using agg, which is a non-GUI backend, so cannot show the figure.
plt.show()
Firing up Jupyter, inside the docker and check out http://chicken:8888/lab
admin@chicken:/workspaces$ jupyter lab --ip 0.0.0.0 --port 8888 --allow-root & Let's run the code... let's work out how to make the images bigger, ok, plt.rcParams["figure.figsize"] = (30,20)
It’s definitely doing something. I see the epilines, they look the same. That looks great. Ok next tutorial.
So, StereoBM. “Depth Map from Stereo Images”
so, the top one is the disparity, viewed with StereoBM. Interesting. BM apparently means Block Matching. Interesting to see that the second run of the epiline code earlier, is changing behaviour. Different parallax estimation?
Changing the max disparities, and block size changes the image, kinda like playing with a photoshop filter. Anyway we’re here to calibrate. What next?
I think the winner for nicest website on the topic goes to Andreas Jakl here, professor.
“Here, we’ll use the traditional SIFT algorithm. Its patent expired in March 2020, and the algorithm got included in the main OpenCV implementation.”
Ok first things first, I’m tightening the screws into the camera holder plastic. We’re only doing this a few times.
Let’s run the capture program one more time for fun.
~/your_ws/jetson-stereo-depth/calib$ python3 capture-stereo.py
I edited the code to make the images 1280×720, which was one of the outputs of the camera. It’s apparently 16:9.
I took a bunch of left and right images. Ran it through the jetson-stereo-depth/calib/01_intrinsics_lens_dist.ipynb code, and the only chessboards it found were the smaller boards .
So let’s put back in the resizing. Yeah no. Ok. Got like 1 match out of 35.
Ok I’m giving up for now. But we’re printing, boys and girls. We’ll try A4 first, and see how it goes. Nope. A3. Professional print.
Nope. Nicely printed doesn’t matter.
Ran another 4 rounds of my own calibration code. Hundreds of pictures. You only need one match.
Nope. You can read more about messing around with failed calibration attempts in this section.
Asked NVIDIA – is it possible? Two monocular cameras into a stereo camera? – no reply. Two months later, NVIDIA replied. “problem will be time synchronizing their frames. You’ll also need to introduce the baseline into the camera_info of the right camera to make it work”
Time is running out. So I threw money at this problem.
Bought the Intel Realsense D455. At first glance, the programs that come with it are very cool, ‘realsense-viewer’. A bit freaky.
Installation is based off these instructions, because these instructions for the Jetson... don't quite work.