Categories
control envs form Gripper Hardware hardware_ Locomotion power robots

Exhibition Robots

For the MFRU exhibition, we presented a variety of robots. The following is some documentation, on the specifications, and setup instructions. We are leaving the robots with konS.

All Robots

Li-Po batteries need to be stored at 3.8V per cell. For exhibition, they can be charged to 4.15A per cell, and run with a battery level monitor until they display 3.7V, at which point they should be swapped out. Future iterations of robotic projects will make use of splitter cables to allow hot swapping batteries, for zero downtime.

We leave our ISDT D2 Mark 2 charger, for maintaining and charging Li-Po batteries.

At setup time, in a new location, Raspberry Pi SD cards need to be updated to connect to the new Wi-fi network. Simplest method is to physically place the SD card in a laptop, and transfer a wpa_supplicant.conf file with the below changed to the new credentials and locale, and a blank file called ssh, to allow remote login.

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=si

network={
    ssid="Galerija"
    psk="galerija"
    key_mgmt=WPA-PSK
}

Then following startup with the updated SD card, robot IP addresses need to be determined, typically using `nmap -sP 192.168.xxx.xxx`, (or a windows client like ZenMap).

Usernames and passwords used are:

LiDARbot – pi/raspberry

Vacuumbot – pi/raspberry and chicken/chicken

Pinkbot – pi/raspberry

Gripperbot – pi/raspberry

Birdbot – daniel/daniel

Nipplebot – just arduino

Lightswitchbot – just arduino and analog timer

For now, it is advised to shut down robots by connecting to their IP address, and typing sudo shutdown -H now and waiting for the lights to turn off, before unplugging. It’s not 100% necessary, but it reduces the chances that the apt cache becomes corrupted, and you need to reflash the SD card and start from scratch.

Starting from scratch involves reflashing the SD card using Raspberry Pi Imager, cloning the git repository, running pi_boot.sh and pip3 install -y requirements.txt, configuring config.py, and running create_service.sh to automate the startup.

LiDARbot

Raspberry Pi Zero W x 1
PCA9685 PWM controller x 1
RPLidar A1M8 x 1
FT5835M servo x 4

Powered by: Standard 5V Power bank [10Ah and 20Ah]

Startup Instructions:
– Plug in USB cables.
– Wait for service startup and go to URL.
– If Lidar chart is displaying, click ‘Turn on Brain’

LiDARbot has a lidar, laser head spinning around, collecting distance updates from the light that bounces back, allowing it to update a 2d (top-down) map of its surroundings.
It is able to not bump into things.

Vacuumbot

Raspberry Pi 3b x 1
LM2596 stepdown converter x 1
RDS60 servo x 4

Powered by: 7.4V 4Ah Li-Po battery

NVIDIA Jetson NX x 1
Realsense D455 depth camera x 1

Powered by: 11.1V 4Ah Li-Po battery

Instructions:
– Plug Jetson assembly connector into 11.4V, and RPi assembly connector into 7.4V
– Connect to Jetson:

cd ~/jetson-server
python3 inference_server.py

– Go to the Jetson URL to view depth and object detection.
– Wait for Rpi service to start up.
– Connect to RPi URL, and click ‘Turn on Brain’

It can scratch around, like the chickens.
Vacuumbot has a depth camera, so it can update a 3d map of its surroundings, and it runs an object detection neural network, so it can interact with its environment. It uses 2 servos per leg, 1 for swivelling its hips in and out, and 1 for the leg rotation.

Pinkbot

Raspberry Pi Zero W x 1
PCA9685 PWM controller x 1
LM2596 stepdown converter x 1
RDS60 servo x 8
Ultrasonic sensors x 3

Powered by: 7.4V 6.8Ah Li-Po battery

Instructions:
– Plug in to Li-Po battery
– Wait for Rpi service to start up.
– Connect to RPi URL, and click ‘Turn On Brain’

Pinkbot has 3 ultrasonic distance sensors, so it has a basic “left” “forward” “right” sense of its surroundings. It uses 8 x 60kg-cm servos, (2 per leg), and 2 x 35kg-cm servos for the head. The servos are powerful, so it can walk, and even jump around.

Gripperbot

Gripperbot has 4 x 60kg-cm servos and 1 x 35kg-cm continuous rotation servo with a worm gear, to open and close the gripper. It uses two spring switches which let it know when the hand is closed. A metal version would be cool.

Raspberry Pi Zero W x 1
150W stepdown converter (to 7.4V) x 1
LM2596 stepdown converter (to 5V) x 1
RDS60 servo x 4
MGGR996 servo x 1

Powered by: 12V 60W power supply

Instructions:
– Plug in to wall
– Wait for Rpi service to start up.
– Connect to RPi URL, and click ‘Fidget to the Waves’

Birdbot

Raspberry Pi Zero W x 1
FT SM-85CL-C001 servo x 4
FE-URT-1 serial controller x 1
12V input step-down converter (to 5V) x 1
Ultrasonic sensor x 1
RPi camera v2.1 x 1

Powered by: 12V 60W power supply

Instructions:
– Plug in to wall
– Wait for Rpi service to start up.
– Connect to RPi URL, and click ‘Fidget to the Waves’

Birdbot is based on the Max Planck Institute BirdBot, and uses some nice 12V servos. It has a camera and distance sensor, and can take pictures when chickens pass by. We didn’t implement the force sensor central pattern generator of the original paper, however. Each leg uses 5 strings held in tension, making it possible, with one servo moving the leg, and the other servo moving the string, to lift and place the leg with a more sophisticated, natural, birdlike motion.

Lightswitchbot

Turns on the light, in the morning
Categories
dev envs hardware_ Locomotion robots

Feetech SCServo

I got the Feetech Smart Bus servos running on the RPi. Using them for the birdbot.

Some gotchas:

  • Need to wire TX to TX, RX to RX.
  • Despite claiming 1000000 baudrate, 115200 was required, or it says ‘There is no status packet!”
  • After only one servo working, for a while, I found their FAQ #5, and installed their debugging software, and plugged each servo in individually, and changed their IDs to 1/2/3/4. It was only running the first one because all of their IDs were still 1.
  • For python, you need to pip3 install pyserial, and then import serial.

Categories
control dev envs hardware_ robots UI Vision

Slamtec RPLidar

I got the RPLidar A1M8-R6 with firmware 1.29, and at first, it was just plastic spinning around, and none of the libraries worked.

But I got it working on Windows, as a sanity check, so it wasn’t broken. So I started again on getting it working on the Raspberry Pi Zero W.

Tried the Adafruit python libs, but v1.29 had some insurmountable issue, and I couldn’t downgrade to v1.27.

So needed to compile the Slamtec SDK.

A helpful post pointed out how to fix the compile error and I was able to compile.

I soldered on some extra wires to the motor + and -, to power the motor separately.

Wasn’t getting any luck, but it turned out to be the MicroUSB cable (The OTG cable was ok). After swapping it out, I was able to run the simple_grabber app and confirm that data was coming out.

pi@raspberrypi:~/rplidar_sdk/output/Linux/Release $ ./simple_grabber --channel --serial /dev/ttyUSB0 115200
theta: 59.23 Dist: 00160.00
theta: 59.50 Dist: 00161.00
theta: 59.77 Dist: 00162.00
theta: 59.98 Dist: 00164.00
theta: 60.29 Dist: 00165.00
theta: 61.11 Dist: 00168.00

I debugged the Adafruit v1.29 issue too. So now I’m able to get the data in python, which will probably be nicer to work with, as I haven’t done proper C++ in like 20 years. But this Slamtec code would be the cleanest example to work with.

So I added in some C socket code and recompiled, so now the demo app takes a TCP connection and starts dumping data.

./ultra_simple --channel --serial /dev/ttyUSB0 115200

It was actually A LOT faster than the python libraries. But I started getting ECONNREFUSED errors, which I thought might be because the Pi Zero W only has a single CPU, and the Python WSGI worker engine was eventlet, which only handles 1 worker, for flask-socketio, and running a socket server, client, and socket-io, on a single CPU, was creating some sort of resource contention. But I couldn’t solve it.

I found a C++ python-wrapped project but it was compiled for 64 bit, and the software, SWIG, which I needed to recompile for 32 bit, seemed a bit complicated.

So, back to Python.

Actually, back to javascript, to get some visuals in a browser. The Adafruit example is for pygame, but we’re over a network, so that won’t work. Rendering Matplotlib graphs is going to be too slow. Need to stream data, and render it on the front end.

Detour #1: NPM

Ok… so, need to install Node.js to install this one, which for Raspberry Pi Zero W, is ARM6.

This is the most recent ARM6 nodejs tarball:

wget https://nodejs.org/dist/latest-v11.x/node-v11.15.0-linux-armv6l.tar.gz

tar xzvf node-v11.15.0-linux-armv6l.tar.gz
cd node-v11.15.0-linux-armv6l
sudo cp -R * /usr/local/
sudo ldconfig
npm install --global yarn
sudo npm install --global yarn

npm install rplidar

npm ERR! serialport@4.0.1 install: `node-pre-gyp install --fallback-to-build`
 
Ok...  never mind javascript for now.

Detour #2: Dash/Plotly

Let’s try this python code. https://github.com/Hyun-je/pyrplidar

Ok well it looks like it works maybe, but where is s/he getting that nice plot from? Not in the code. I want the plot.

So, theta and distance are just polar coordinates. So I need to plot polar coordinates.

PolarToCartesian.

Convert a polar coordinate (r,θ) to cartesian (x,y): x = r cos(θ), y = r sin(θ)

Ok that is easy, right? So here’s a javascript library with a polar coordinate plotter

So, plan is, set up a flask route, read RPLidar data, publish to a front end, which plots it in javascript

Ok after some googling, Dash / Plotly looks like a decent option.

Found this code. Cool project! And though this guy used a different Lidar, it’s pretty much what I’m trying to do, and he’s using plotly.

pip3 install pandas
pip3 install dash

k let's try...
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 48 from C header, got 40 from PyObject

ok
pip3 install --upgrade numpy     
(if your numpy version is < 1.20.0)

ok now  bad marshal data (unknown type code)
sheesh, what garbage.  
Posting issue to their github and going back to the plan.

Reply from Plotly devs: pip3 won’t work, will need to try conda install, for ARM6

Ok let’s see if we can install plotly again….

Going to try miniconda – they have a arm6 file here

Damn. 2014. Python 2. Nope. Ok Plotly is not an option for RPi Zero W. I could swap to another RPi, but I don’t think the 1A output of the power bank can handle it, plus camera, plus lidar motor, and laser. (I am using the 2.1A output for the servos).

Solution #1: D3.js

Ok, Just noting this link, as it looks useful for the lidar robot, later.

So, let’s install socket io and websockets

pip3 install flask_socketio
pip3 install simple-websocket
pip3 install flask-executor

(looking at this link) for flask and socket-io, and this link for d3 polar chart

app isn’t starting though, since adding socket-io So, hmm. Ok, this issue. Right, needs 0.0.0.0.

socketio.run(app, debug=True, host='0.0.0.0')

Back to it…

K. Let’s carry on with Flask/d3.js though.

I think if we’re doing threading, I need to use a WSGI server.

pip install waitress

ok that won’t work with flask-socketio. Needs gevent or eventlet.

eventlet is the best performant option, with support for long-polling and WebSocket transports.”

apparently needs redis for message queueing…

pip install eventlet
pip install redis

Ok, and we need gunicorn, because eventlet is just for workers...

pip3 install gunicorn

gunicorn --worker-class eventlet -w 1 module:app

k, that throws an error.
I need to downgrade eventlet, or do some complicated thing.
pip install eventlet==0.30.2

gunicorn --bind 0.0.0.0 --worker-class eventlet -w 1 kmp8servo:app
(my service is called kmp8servo.py)


ok so do i need redis?
sudo apt-get install redis
ok it's already running now, 
at /usr/bin/redis-server 127.0.0.1:6379
no, i don't really need redis.  Could use sqlite, too. But let's use it anyway.

Ok amazing, gunicorn works.  It's running on port 8000

Ok, after some work,  socket-io is also working.

Received #0: Connected
Received #1: I'm connected!
Received #2: Server generated event
Received #3: Server generated event
Received #4: Server generated event
Received #5: Server generated event
Received #6: Server generated event
Received #7: Server generated event

So, I’m going to go with d3.js instead of P5js, just cause it’s got a zillion more users, and there’s plenty of polar coordinate code to look at, too.

Got it drawing the polar background… but I gotta change the scale a bit. The code uses a linear scale from 0 to 1, so I need to get my distances down to something between 0 and 1. Also need radians, instead of the degrees that the lidar is putting out.

ok finally. what an ordeal.

But now we still need to get python lidar code working though, or switch back to the C socket code I got working.

Ok, well, so I added D3 update code with transitions, and the javascript looks great.

But the C Slamtec SDK, and the Python RP Lidar wrappers are a source of pain.

I had the C sockets working briefly, but it stopped working, seemingly while I added more Python code between each socket read. I got frustrated and gave up.

The Adafruit library, with the fixes I made, seem to work now, but it’s in a very precarious state, where looking at it funny causes a bad descriptor field, or checksum error.

But I managed to get the brain turning on, with the lidar. I’m using Redis to track the variables, using the memory.py code from this K9 repo. Thanks.

I will come back to trying to fix the remaining python library issues, but for now, the robot is running, so, on to the next.

Categories
AI/ML CNNs dev Locomotion OpenCV robots UI Vision

Realsense Depth and TensorRT object detection

A seemingly straightforward idea for robot control involves using depth, and object detection, to form a rough model of the environment.

After failed attempts to create our own stereo camera using two monocular cameras, we eventually decided to buy a commercial product instead, The Intel depth camera, D455.

After a first round of running a COCO trained MobileSSDv2 object detection network in Tensorflow 2 Lite, on the colour images obtained from the realsense camera, on the Jetson Nano, the results were just barely acceptable (~2 FPS) for a localhost stream, and totally unacceptable (~0.25 FPS) served as JPEG over HTTP, to a browser on the network.

Looking at the options, the only feasible solution was to redo the network using TensorRT, the NVIDIA-specific, quantized (16 bit on the Nano, 8 bit on the NX/AGX) neural network framework. The other solution involved investigating options other than simple JPEG compression over HTTP, such as RTSP and WebRTC.

The difficult part was setting up the environment. We used the NVIDIA detectnet code, adapted to take the realsense camera images as input, and to display the distance to the objects. An outdated example was found at CAVEDU robotics blog/github. Fixed up below.

#!/usr/bin/python3



import jetson_inference
import jetson_utils
import argparse
import sys
import os
import cv2
import re
import numpy as np
import io
import time
import json
import random
import pyrealsense2 as rs
from jetson_inference import detectNet
from jetson_utils import videoSource, videoOutput, logUsage, cudaFromNumpy, cudaAllocMapped, cudaConvertColor

parser = argparse.ArgumentParser(description="Locate objects in a live camera stream using an object detection DNN.",
formatter_class=argparse.RawTextHelpFormatter, epilog=jetson_utils.logUsage())
parser.add_argument("--network", type=str, default="ssd-mobilenet-v2",
help="pre-trained model to load (see below for options)")
parser.add_argument("--threshold", type=float, default=0.5,
help="minimum detection threshold to use")
parser.add_argument("--width", type=int, default=640,
help="set width for image")
parser.add_argument("--height", type=int, default=480,
help="set height for image")
opt = parser.parse_known_args()[0]

# load the object detection network
net = detectNet(opt.network, sys.argv, opt.threshold)

# Configure depth and color streams
pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, opt.width, opt.height, rs.format.z16, 30)
config.enable_stream(rs.stream.color, opt.width, opt.height, rs.format.bgr8, 30)
# Start streaming
pipeline.start(config)


press_key = 0
while (press_key==0):
	# Wait for a coherent pair of frames: depth and color
	frames = pipeline.wait_for_frames()
	depth_frame = frames.get_depth_frame()
	color_frame = frames.get_color_frame()
	if not depth_frame or not color_frame:
		continue
	# Convert images to numpy arrays
	depth_image = np.asanyarray(depth_frame.get_data())
	show_img = np.asanyarray(color_frame.get_data())
	
	# convert to CUDA (cv2 images are numpy arrays, in BGR format)
	bgr_img = cudaFromNumpy(show_img, isBGR=True)
	# convert from BGR -> RGB
	img = cudaAllocMapped(width=bgr_img.width,height=bgr_img.height,format='rgb8')
	cudaConvertColor(bgr_img, img)

	# detect objects in the image (with overlay)
	detections = net.Detect(img)

	for num in range(len(detections)) :
		score = round(detections[num].Confidence,2)
		box_top=int(detections[num].Top)
		box_left=int(detections[num].Left)
		box_bottom=int(detections[num].Bottom)
		box_right=int(detections[num].Right)
		box_center=detections[num].Center
		label_name = net.GetClassDesc(detections[num].ClassID)

		point_distance=0.0
		for i in range (10):
			point_distance = point_distance + depth_frame.get_distance(int(box_center[0]),int(box_center[1]))

		point_distance = np.round(point_distance / 10, 3)
		distance_text = str(point_distance) + 'm'
		cv2.rectangle(show_img,(box_left,box_top),(box_right,box_bottom),(255,0,0),2)
		cv2.line(show_img,
			(int(box_center[0])-10, int(box_center[1])),
			(int(box_center[0]+10), int(box_center[1])),
			(0, 255, 255), 3)
		cv2.line(show_img,
			(int(box_center[0]), int(box_center[1]-10)),
			(int(box_center[0]), int(box_center[1]+10)),
			(0, 255, 255), 3)
		cv2.putText(show_img,
			label_name + ' ' + distance_text,
			(box_left+5,box_top+20),cv2.FONT_HERSHEY_SIMPLEX,0.4,
			(0,255,255),1,cv2.LINE_AA)

	cv2.putText(show_img,
		"{:.0f} FPS".format(net.GetNetworkFPS()),
		(int(opt.width*0.8), int(opt.height*0.1)),
		cv2.FONT_HERSHEY_SIMPLEX,1,
		(0,255,255),2,cv2.LINE_AA)


	display = cv2.resize(show_img,(int(opt.width*1.5),int(opt.height*1.5)))
	cv2.imshow('Detecting...',display)
	keyValue=cv2.waitKey(1)
	if keyValue & 0xFF == ord('q'):
		press_key=1


cv2.destroyAllWindows()
pipeline.stop()

Assuming you have a good cmake version and cuda is available (if nvcc doesn’t work, you need to configure linker paths, check this link)… (if you have a cmake version around ~ 3.22-3.24 or so, you need an older one)… prerequisite sudo apt-get install libssl-dev also required.

The hard part was actually setting up the Realsense python bindings.

Clone the repo…

git clone https://github.com/IntelRealSense/librealsense.git

The trick being, to request the python bindings, and cuda, during the cmake phase. Note that often, none of this works. Some tips include…

sudo apt-get install xorg-dev libglu1-mesa-dev

and changing PYTHON to Python

mkdir build
cd build
cmake ../ -DBUILD_PYTHON_BINDINGS:bool=true -DPYTHON_EXECUTABLE=/usr/bin/python3 -DCMAKE_BUILD_TYPE=release -DBUILD_EXAMPLES=true -DBUILD_GRAPHICAL_EXAMPLES=true -DBUILD_WITH_CUDA:bool=true

The above worked on Jetpack 4.6.1, while the below worked on Jetpack 5.0.2

cmake ../ -DBUILD_PYTHON_BINDINGS:bool=true -DPython_EXECUTABLE=/usr/bin/python3.8 -DCMAKE_BUILD_TYPE=release -DBUILD_EXAMPLES=true -DBUILD_GRAPHICAL_EXAMPLES=true -DBUILD_WITH_CUDA:bool=true -DPYTHON_INCLUDE_DIRS=/usr/include/python3.8 -DPython_LIBRARIES=/usr/lib/aarch64-linux-gnu/libpython3.8.so

(and sudo make install)
Update the python path

export PYTHONPATH=$PYTHONPATH:/usr/local/lib
(or a specific python if you have more than one)
if installed in /usr/lib, change accordingly

Check that the folder is in the correct location (it isn't, after following official instructions).

./usr/local/lib/python3.6/dist-packages/pyrealsense2/

Check that the shared object files (.so) are in the right place: 

chicken@chicken:/usr/local/lib$ ls
cmake       libjetson-inference.so  librealsense2-gl.so.2.50    librealsense2.so.2.50    pkgconfig
libfw.a     libjetson-utils.so      librealsense2-gl.so.2.50.0  librealsense2.so.2.50.0  python2.7
libglfw3.a  librealsense2-gl.so     librealsense2.so            librealsense-file.a      python3.6


If it can't find 'pipeline', it means you need to copy the missing __init__.py file.

sudo cp ./home/chicken/librealsense/wrappers/python/pyrealsense2/__init__.py ./usr/local/lib/python3.6/dist-packages/pyrealsense2/

Some extra things to do, 
sudo cp 99-realsense-libusb.rules  /etc/udev/rules.d/

Eventually, I am able to run the inference on the realsense camera, at an apparent 25 FPS, on the localhost, drawing to an OpenGL window.

I also developed a Dockerfile, for the purpose, which benefits from an updated Pytorch version, but various issues were encountered, making a bare-metal install far simpler, ultimately. Note that building jetson-inference, and Realsense SDK on the Nano require increasing your swap size, beyond the 2GB standard. Otherwise, the Jetson freezes once memory paging leads to swap death.

Anyway, since the objective is remote human viewing, (while providing depth information for the robot to use), the next step will require some more tests, to find a suitable option.

The main blocker is the power usage limitations on the Jetson Nano. I can’t seem to run Wifi and the camera at the same time. According to the tegrastats utility, the POM_5V_IN usage goes over the provided 4A, under basic usage. There are notes saying that 3A can be provided to 2 of the 5V GPIO pins, in order to get 6A total input. That might end up being necessary.

Initial investigation into serving RTSP resulted in inferior, and compressed results compared to a simple python server streaming image by image. The next investigation will be into WebRTC options, which are supposedly the current state of the art, for browser based video streaming. I tried aiortc, and momo, so far, both failed on the Nano.

I’ve decided to try on the Xavier NX, too, just to replicate the experiment, and see how things change. The Xavier has some higher wattage settings, and the wifi is internal, so worth a try. Also, upgraded to Jetpack 5.0.2, which was a gamble. Thought surely it would be better than upgrading to a 5.0.1 dev preview, but none of their official products support 5.0.2 yet, so there will likely be much pain involved. On the plus side, python 3.8 is standard, so some libraries are back on the menu.

On the Xavier, we’re getting 80 FPS, compared to 25 FPS on the Nano. Quite an upgrade. Also, able to run wifi and realsense at the same time.

Looks like a success. Getting multiple frames per second with about a second of lag over the network.

Categories
control dev envs robots Vision

RTSP

This was simple to set up, and is meant to be fast.

https://github.com/aler9/rtsp-simple-server#configuration

Unfortunately the results of using the OpenCV/GStreamer example code to transmit over the network using H264 compression, were even worse than the JPEG over HTTP attempt I’m trying to improve on. Much worse. That was surprising. It could be this wifi dongle though, which is very disappointing on the Jetson Nano. It appears as though the Jetson Nano tries to keep total wattage around 10W, however plugging in the Realsense camera and a wifi dongle is pulling way more than that (All 4A @ 5W supplied by the barrel jack). It may mean that wireless robotics with the Realsense is not practical, on the Jetson.

Required apt install gstreamer1.0-rtsp to be installed.

Back to drawing board for getting the RealSense colour and depth transmitting to a different viewing machine, on the network (while still providing distance data for server side computation).

Categories
AI/ML CNNs deep dev ears robots

YAMNet

After changing the normalisation code, (because the resampled 16000Hz audio was too soft), we have recordings from the microphone being classified by yamnet on the jetson.

Pretty cool. It’s detecting chicken sounds. I had to renormalize the recording volume between -1 and 1, as everything was originally detected as ‘Silence’.

Currently I’m saving 5 second wav files and processing them in Jupyter. But it’s not really interactive in a real-time way, and it would need further training to detect distress, or other, more useful metrics.

Training instructions – however, there appears to be a tutorial.

We’re unlikely to have time to implement the transfer learning, to continue with the chicken stress vocalisation work, for this project, but it definitely looks like the way to go about it.

There are also some papers that used the VGG-11 architecture for this purpose, chopping up recordings into overlapping 1 second segments, for training, matching them to a label (stressed / not stressed). Note: If downloading the dataset, use the G-Drive link, not the figshare link, which is truncated.

Let’s try get YAMnet running in real time.

After following the installation procedure for my ‘Respeaker 2-Mic hat’, I’ve set up a dockerfile with TF2 and the audio libs, including librosa, in order to try out this real time version. Getting this right was a real pain, because of breaking changes in the ‘numba’ package.

FROM nvcr.io/nvidia/l4t-tensorflow:r32.6.1-tf2.5-py3

RUN apt-get update && apt-get install -y curl build-essential
RUN apt-get update && apt-get install -y libffi6 libffi-dev

RUN pip3 install -U Cython
RUN pip3 install -U pillow
RUN pip3 install -U numpy
RUN pip3 install -U scipy
RUN pip3 install -U matplotlib
RUN pip3 install -U PyWavelets
RUN pip3 install -U kiwisolver

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
            alsa-base \
            libasound2-dev \
            alsa-utils \
            portaudio19-dev \
            libsndfile1 \
            unzip \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean


RUN pip3 install soundfile pyaudio wave
RUN pip3 install tensorflow_hub
RUN pip3 install packaging
RUN pip3 install pyzmq==17.0.0
RUN pip3 install jupyterlab

RUN apt-get update && apt-get install -y libblas-dev \
				      liblapack-dev \
				      libatlas-base-dev \
				      gfortran \
				      protobuf-compiler \
				      libprotoc-dev \
		                      llvm-9 \
				      llvm-9-dev 

RUN export LLVM_CONFIG=/usr/lib/llvm-9/bin/llvm-config && pip3 install llvmlite==0.36.0
RUN pip3 install --upgrade pip
RUN python3 -m pip install --user -U numba==0.53.1
RUN python3 -m pip install --user -U librosa==0.9.2
#otherwise matplotlib can't draw to gui
RUN apt-get update && apt-get install -y python3-tk


RUN jupyter lab --generate-config

RUN python3 -c "from notebook.auth.security import set_password; set_password('nvidia', '/root/.jupyter/jupyter_notebook_config.json')"

EXPOSE 6006
EXPOSE 8888

CMD /bin/bash -c "jupyter lab --ip 0.0.0.0 --port 8888 --allow-root &> /var/log/jupyter.log" & \
        echo "allow 10 sec for JupyterLab to start @ http://$(hostname -I | cut -d' ' -f1):8888 (password nvidia)" && \
        echo "JupterLab logging location:  /var/log/jupyter.log  (inside the container)" && \
        /bin/bash


I'm running it with

sudo docker run -it --rm --runtime nvidia --network host --privileged=true -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v /home/chicken/:/home/chicken nano_tf2_yamnet

Categories
3D Research control dev Locomotion robots

Inertial Measurement Unit: IMU

Like a SLAM part 2.1,

Some topics here for general vibes, and a robotics site’s article on them.

Of course having sensors telling you if you’ve fallen over are useful. Of course.

The IMU just gives you the relative axes on the robot, and that can feed back to a control algorithm which uses that info to influence behaviour.

Here’s a useful Q that made me realise I must get an IMU running, perhaps. https://robotics.stackexchange.com/questions/21279/do-imu-have-value-in-slam

After reading up on IMUs, you get 3 axes: accelerometer, 6 axes: + gyroscope, 9 axes: + magnetometer. And some 10 axes ones, if it’s fancy, and has a thermometer to correct inaccuracies, etc.

6 axes gives you relative positions, 9 axes gives you absolute positions.

I happen to have a 6 axis one, from Aliexpress, from years ago. Never used it, but now I have a reason. It’s labelled GY-521. Here’s a video tutorial on putting it all together, with the tutorial link for reading.

“the 6-DoF, which was used to determine the linear velocity, angular velocity, position and orientation.” – this paper, “Pose Estimation of a Mobile Robot Based on Fusion of IMU Data and Vision Data Using an Extended Kalman Filter”

You need to take these folders from the github link.

and put them in your Arduino libs folder

The github also has some code for Raspberry Pi, which I might get to next. Badabing badaboom, it actually worked first time. ( I touched the USB cable though, and needed to restart it, but that seems like something that can be prevented).

Ok so accelerometer x/y/z, temperature nice, gyroscope x/y/z

I watched these numbers as I moved it around, and at 9600 baud, it’s really slow. It’s not going to help for real time decision making.

Maybe we’ll come back to IMUs later. A bit complicated to visualise and make sense of the data, but with a visualisation, it would make odometry SLAM methods more robust.

Categories
control dev Locomotion robots

Basic Planning

We need the robot to be somewhat intelligent, and that means some rules, or processes based on sensor input. Originally I was doing everything in PyBullet, and that leads to techniques like PyBullet-Planning. But as we’re getting closer to a deadline, simpler ideas are winning out.

I came across this paper, and figures 1 and 2 give an idea.

Figure 1 “Rule-Based Approach for Constrained Motion Control of a Teleoperated Robot Arm in a Dynamic Environment”

WordPress apparently won’t show this image in original resolution, but you can look up close in the PDF.

Here’s a fuzzy controller: FUZZY LOGIC CONTROL FOR ROBOT MAZE TRAVERSAL: AN UNDERGRADUATE CASE STUDY

I like that idea too. Fuzzifying concepts, defuzzifying into adjusting towards descriptions from rules

This is probably starting to get too many acronyms for me, but Goal Reasoning is a pretty fascinating topic. Now that I see it, of course NASA has been working on this stuff for ages.

Excerpted below (Doychev, 2021):

The ”C” Language Production System (CLIPS) is a portable, rule-based production system [Wyg89]. It was first developed for NASA as an expert system and uses forward chaining inference based on the Rete algorithm consisting of three building blocks[JCG]:
  • Fact List: The global memory of the agent. It is used as a container to
    store basic pieces of information about the world in the form of facts, which are usually of specific types. The fact list is constantly updated using the knowledge in the knowledge base.
  • Knowledge Base: It comprises heuristic knowledge in two forms:

    • Procedural Knowledge: An operation that leads to a certain effect.
    These can, for example, modify the fact base. Functions carry procedural knowledge and can also be implemented in C++. They are
    mainly used for the utilization of the agent, such as communication to
    a behavior engine. An example for procedural knowledge would be a
    function that calls a robot-arm driver to grasp at a target location, or a
    fact base update reflecting a sensor reading.

    • Rules: Rules play an important role in CLIPS. They can be compared
    to IF-THEN statements in procedural languages. They consist of several preconditions, that need to be satisfied by the current fact list for the rule to be activated, and effects in the form of procedural knowledge. When all its preconditions are satisfied, a rule is added to the agenda, which executes all the activated rules subsequently by firing
    the corresponding procedural knowledge.
  • Inference Engine: The main controlling block. It decides which rules
    should be executed and when. Based on the knowledge base and the fact base, it guides the execution of agenda and rules, and updates the fact base, if needed. This is performed until a stable state is reached, meaning, there are no more activated rules. The inference engine supports different conflict resolution strategies, as multiple rules can be active at a time. For example, rules are ordered by their salience, a numeric value where a higher value means higher priority. If rules with the same salience are active at a time, they are executed in the order of their activation.


  • CLIPS Executive
    The CLIPS Executive (CX) is a CLIPS-based production system which serves as a high-level controller, managing all the high-level decision making. Its main tasks involve goal formation, goal reasoning, on-demand planner invocation, goal execution and monitoring, world and agent memory (a shared database for multiple agents) information synchronization. In general, this is achieved by individual CLIPS structures (predefined facts, rules, etc.), that get added to the CLIPS environment.

Above is excerpted from paper.

So, basically it’s a rule engine.

It’s the Rete algorithm, so it’s a rule engine. It’s a cool algorithm. If you don’t know about rule engines, they are what you use when you start to have more than 50 ‘if’ statements.

Ok, that’s all I need to know. I’ve used KIE professionally. I don’t want to use Java in this project. There appear to be some simplified Python Rule Engines, and so I’ll check them out, when I have some sensor input.

I think I’m going to try this one. They snagged ‘rule-engine’ at PyPi, so they must be good.

But I’ve also looked at this python rule engine comparison, and might consider ‘Durable Rules’.

https://zerosteiner.github.io/rule-engine/getting_started.html

pip3 install rule-engine

Ok, I’ve set up an Arduino with three ultrasonic distance sensors, and it’s connected to the Raspberry Pi. I should do a write-up on that. So I can poll the Arduino and get ‘forward left’, ‘forward’ and ‘forward right’ ultrasonic sensor distance back, in JSON.

I think for now, a good start would be having consequences of, forward, backwards, left, right, and stand still.

These are categories of motions. Motions have names, so far, so we will just categorize motions by their name, by whether they contain one of these cardinal motions (forward or walk, back, left, right, still) in their name.

To keep things interesting, the robot can pick motions from these categories at random. I was thinking of making scripts, to join together motions, but I’m going to come back to that later. Scripts would just be sequences of motions, so it’s not strictly necessary, since we’re going to use a rule engine now.

Ok… after thinking about it, screw the rule engine. We only need like 10 rules, at the moment, and I just need to prioritize the rules, and reevaluate often. I’ll just write them out, with a basic prioritisation.

I also see an interesting hybrid ML / Rule option from sci-kit learn.

Anyway, too complicated for now. So this would all be in a loop.

TOO_CLOSE=30
priority = 0

# High Priority '0'
if F < TOO_CLOSE:
    runMotionFromCategory("BACK")
    priority = 1
if L < TOO_CLOSE:
    runMotionFromCategory("BACK")
    runMotionFromCategory("RIGHT")
    priority = 1
if R < TOO_CLOSE:
    runMotionFromCategory("BACK")
    runMotionFromCategory("LEFT")
    priority = 1

# Medium Priority '1'  (priority is still 0)
if priority == 0 and L < R and L < F:
    runMotionFromCategory("RIGHT")
    priority = 2
 
if priority == 0 and R < L and R < F:
    runMotionFromCategory("LEFT")
    priority = 2

# Low Priority '2'  (priority is still 0)
if priority == 0 and L < F and R < F:
    runMotionFromCategory("WALK")
    priority = 3

Good enough. So I still want this to be part of the UI though, and so the threading, and being able to exit the loop will be important.

Basically, the problem is, how to implement a stop button, in HTTP / Flask. Need a global variable, basically, which can be changed. But a global variable in a web app? Sounds like a bad idea. We have session variables, but the thread that’s running the motion is in the past, and probably evaluating a different session map. Maybe not though. Will need to research and try a few things.

Yep… “Flask provides you with a special object that ensures it is only valid for the active request and that will return different values for each request.”

Ok, Flask has ‘g’ …?

from flask import g
user = getattr(flask.g, 'user', None)
user = flask.g.get('user', None)

Hmm ok that’s not right. It’s not sharable between requests, apparently. There’s Flask-Cache…. that might work? Ok I think this is it maybe.

def task():
    if exit_event.is_set(): 
            print(f'end {threading.current_thread().name} ')
            return

exit_event = threading.Event()

@app.route("/", methods=["GET", "POST"])
def index():
    exit_event.clear()
    t=threading.Thread(target=task, args=(thread_return,), daemon=True)
    t.start()
    ...
    exit_event.set()

Ok, except I’m getting “RuntimeError: Working outside of request context.” So there’s some official info

Ok the official info looks odd, calling it a test. But I found this decorator elsewhere…

@copy_current_request_context
def runMotionTask(data):

RuntimeError: This decorator can only be used when a request context is active, such as within a view function.

Ok, so I need to put the thread inside the view…. ok…

Ha it worked. This is ugly though. How do I reuse code with this shitty thread-inside-view syntax?

@app.route('/runadjustedmotion', methods=['POST', 'GET'])
def runadjustedmotion():
    data = request.get_json()
    return runadjustedmotiondirect(data)

def runadjustedmotiondirect(data):
    @copy_current_request_context
    def runMotionTask(data):
        ...

    (start thread)

Ok so. Kill switch is working. Very good.

Now, to run the brain. I don’t know how to reuse code with these inner functions. So it’s copy-paste time. The brain will need to be a single thread. So something like

@app.route('/runBrain', methods=['POST', 'GET']) 
def runBrain():     
   @copy_current_request_context     
   def runBrainTask(data):
      @copy_current_request_context     
      def runMotionFromCategory(category):
         ...
      ...
      if x then runMotionFromCategory("BACK")
      if y then runMotionFromCategory("LEFT")
   (start runBrainTask thread)

Ok let’s try it…

Ok first need to fix the motions.

Alright. It’s working! Basic Planning working.

It can shuffle around now, without bumping into walls. Further testing will be needed. But a good basis for a robot.

Categories
bio control form Locomotion robots

Pantograph Legs

“They observed that many quadrupedal, mammalian animals feature a distinguished functional three-segment front leg and hind leg design, and proposed a “pantograph” leg abstraction for robotic research.”

1 DOF (degree of freedom). 1 motor. Miranda wants jointed legs, and I don’t want to work out inverse kinematics, so this looks ideal. Maybe a bit complicated still.

Biorobotics Laboratory, EPFL

The simpler force diagram:

Cheetah-cub leg mechanism, and leg compliance. A single leg is shown abstracted, detailed leg segment ratios are omitted for clarity, robot heading direction is to the left. (1) shows the three leg angles αprox, αmid, and αdist. Hip and knee RC servo motors are mounted proximally, the leg length actuation is transmitted by a cable mechanism. The pantograph structure was inspired by the work of Witte et al. (2003) and Fischer and Blickhan (2006). (2) The foot segment describes a simplified foot-locus, showing the leg in mid-swing. For ground clearance, the knee motor shortens the leg by pulling on the cable mechanism (green, Fcable). Fdiag is the major, diagonal leg spring. Its force extends the pantograph leg, against gravitational and dynamic forces. (3) The leg during mid-stance. (4) In case of an external translational perturbation, the leg will be compressed passively. (5) If an external perturbation torque applies e.g., through body pitching, the leg linkage will transmit it into a deflection of the parallel spring, not of the diagonal spring.
Cheetah-cub leg mechanism, and leg compliance. A single leg is shown abstracted, detailed leg segment ratios are omitted for clarity, robot heading direction is to the left. (1) shows the three leg angles αprox, αmid, and αdist. Hip and knee RC servo motors are mounted proximally, the leg length actuation is transmitted by a cable mechanism. The pantograph structure was inspired by the work of Witte et al. (2003) and Fischer and Blickhan (2006). (2) The foot segment describes a simplified foot-locus, showing the leg in mid-swing. For ground clearance, the knee motor shortens the leg by pulling on the cable mechanism (green, Fcable). Fdiag is the major, diagonal leg spring. Its force extends the pantograph leg, against gravitational and dynamic forces. (3) The leg during mid-stance. (4) In case of an external translational perturbation, the leg will be compressed passively. (5) If an external perturbation torque applies e.g., through body pitching, the leg linkage will transmit it into a deflection of the parallel spring, not of the diagonal spring.Kinematic primitives for walking and trotting gaits of a quadruped robot with compliant legs (Alexander Badri-Spröwitz et al, 2014)

Compliance is a feature, made possible by springs typically.

Biologically Inspired Robots - nitishpuri.github.io
https://nitishpuri.github.io/posts/robotics/biologically-inspired-robots/

A homemade attempt here with the Mojo robot of the Totally Not Evil Robot Army. Their robot only uses 9g servos, and can’t quite pick itself up.

I did an initial design with what I had around, and it turns out compliance is a delicate balance. Too much spring, and it just mangles itself up. Too little spring and it can’t lift off the ground.

Further iterations removed the springs, which were too tight by far, and used cable ties to straighten the legs, but the weight of the robot is a little bit too much for the knee joints.

I will likely leave it until I have a 3d printer, some better springs, and will give it another try with more tools and materials available. Maybe even hydraulics, some day,

Some more research required, too.

https://www.mdpi.com/1424-8220/20/17/4911/htm

Categories
3D Research AI/ML arxiv control form Locomotion robots

Kinematic Motion Primitives

This post follows the ‘Finding where we left off’ post, focused on locomotion sim2real. In that post I tried to generalise and smooth the leg angle servo movements in their -PI/2 to PI/2 range.

I will likely try extracting kMPs, before this is all over, which from a skim read, and look at the pictures, are like, just taking a single slice of the wave data, and repeating that. Or, taking consecutive periodic waves, and extracting the average / normalized movement from them.

https://becominghuman.ai/introduction-to-timeseries-analysis-using-python-numpy-only-3a7c980231af

Cheetah-cub leg mechanism, and leg compliance. A single leg is shown abstracted, detailed leg segment ratios are omitted for clarity, robot heading direction is to the left. (1) shows the three leg angles αprox, αmid, and αdist. Hip and knee RC servo motors are mounted proximally, the leg length actuation is transmitted by a cable mechanism. The pantograph structure was inspired by the work of Witte et al. (2003) and Fischer and Blickhan (2006). (2) The foot segment describes a simplified foot-locus, showing the leg in mid-swing. For ground clearance, the knee motor shortens the leg by pulling on the cable mechanism (green, Fcable). Fdiag is the major, diagonal leg spring. Its force extends the pantograph leg, against gravitational and dynamic forces. (3) The leg during mid-stance. (4) In case of an external translational perturbation, the leg will be compressed passively. (5) If an external perturbation torque applies e.g., through body pitching, the leg linkage will transmit it into a deflection of the parallel spring, not of the diagonal spring.
Kinematic primitives for walking and trotting gaits of a quadruped robot with compliant legs (Alexander Badri-Spröwitz et al, 2014)

It’s now December 6th 2021, as I continue here…

This paper is very relevant, “Realizing Learned Quadruped Locomotion Behaviors through Kinematic Motion Primitives”

Some Indian PhDs have summed up the process. Unfortunately I’m not quite on the exact same page. I understand the pictures, haha.

Here’s where this picture comes from, which is useful for explaining what I need to do: (Short paper)

In 2014, also, same thing, Kinematic primitives for walking and trotting gaits of a quadruped robot with compliant legs

They just used PCA. (Principal Component Analysis). That’s like a common ML toolkit thing.

Kinematic primitives for walking and trotting gaits of a quadruped robot with compliant legs (2014)

See now this is where they lose me: “The covariance matrix of the normalized dataset”. Come on guys. Throw us a bone.

I found this picture, which is worth 1000 words, in the discussion on stackexchange about PCA and SVD:

Rotating PCA animation

So, I’m not quite ready for PCA. That is two dimensions, anyway. Oh right, so I need to add a ‘time’ dimension. numpy’s expand_dims?

I played around with Codex, to assist with finding the peaks, and to find the period length.

And I separated them out to different plots… and got the peaks matching once I passed in ( , distance=80).

I had to install these, and restart the Jupyter kernel (and I think close and restart the Chrome tab.) in order to get some matplotlib widgets.

Error message:
Jupyter Lab: Error displaying widget: model not found



!pip3 install --upgrade jupyterlab ipympl
%matplotlib widget
The matplotlib slider example (image thereof)

I started on a slider widget to draw a vertical line on top of the leg data, but I need to fix the refresh issue. Anyhow, it’s not quite what i want. What do I want?

So, I want the kMPs. The kMPs are like, a gif of a basic action, e.g. robot taking a full step forward, on all legs, which we can run once, twice, etc.

We can ‘average’ or ‘normalise’ or ‘phase’ the waves, and assume that gives us a decent average step forward.

I think there’s enough variation in this silly simulation walk that we should start with just the simplest, best single wave.

But since they ran PCA, let’s run it to see what it does for the data. We have a single integer value, which is 1D. To make it 2D, so we can run PCA on it… we add a time dimension?

But also, so I measured the period a few programs up, to be

67 steps (front right),

40 steps (front left),

59 steps (back right),

42 steps (back left).

So, as a starting point, it would be nice to be as close to servos at 90 degrees as possible. If I iterate the values, and track the lowest sum diff, yeah… is that it? I’m looking at this link at SO.

Ideally I could visualise the options..

Repeating a slice. Averaging the slices.

Ok, so I need a start index, end index, to index a range.

After some investigation, the index where the legs are closest to 90 degrees, is at 1739

Computer Enhance

So that’s kinda close to our ideal kMPs, from about 1739 to about 1870 maybe, but clearly the data is messy. Could be tweaked. Wavetable editor, basically.

Alright, let’s make an app. We can try run a Flask server on the Pi, with Javascript front end using chart.js.

pip3 install flask

Save the test web app, kmpapp.py

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return 'Hello world'

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0') 

python3 kmpapp.py

Ok good start. We need to get the x and y data into JSON so Javascript can plot it, in chart.js

That’s looking good. Maybe too many points. Ok, so I want to edit, save, and run the KMPs on the robot.

Well it took a day but it’s working, and is pretty cool. Used smooth.js to allow smoother transitions. Took another day to add save and load features.

I’ll upload this to the project repo.

Many improvements added. Will update repo again closer to MFRU.