Categories
dev envs Linux OpenCV

Installing OpenCV on Jetson

I have the Jetson NX, and I tried the last couple of days to install OpenCV, and am still fighting it. But we’re going to give it a few rounds.

Let’s see, so I used DustyNV’s Dockerfile with an OpenCV setup for 4.4 or 4.5.

But the build dies, or is still missing libraries. There’s a bunch of them, and as I’m learning, everything is a linker issue. Everything. sudo ldconfig.

Here’s a 2019 quora answer to “What is sudo ldconfig in linux?”

“ldconfig updates the cache for the linker in a UNIX environment with libraries found in the paths specified in “/etc/ld.so.conf”. sudo executes it with superuser rights so that it can write to “/etc/ld.so.cache”.

You usually use this if you get errors about some dynamically linked libraries not being found when starting a program although they are actually present on the system. You might need to add their paths to “/etc/ld.so.conf” first, though.” – Marcel Noe

So taking my own advice, let’s see:

chicken@chicken:/etc/ld.so.conf.d$ find | xargs cat $1
cat: .: Is a directory
/opt/nvidia/vpi1/lib64
/usr/local/cuda-10.2/targets/aarch64-linux/lib
# Multiarch support
/usr/local/lib/aarch64-linux-gnu
/lib/aarch64-linux-gnu
/usr/lib/aarch64-linux-gnu
/usr/lib/aarch64-linux-gnu/libfakeroot
# libc default configuration
/usr/local/lib
/usr/lib/aarch64-linux-gnu/tegra
/usr/lib/aarch64-linux-gnu/fakechroot
/usr/lib/aarch64-linux-gnu/tegra-egl
/usr/lib/aarch64-linux-gnu/tegra

Ok. On our host (Jetson), let’s see if we can install it, or access it. It’s Jetpack 4.6.1 so it should have it installed already.

ImportError: libblas.so.3: cannot open shared object file: No such file or directory

cd /usr/lib/aarch64-linux-gnu/
ls -l liblas.so*

libblas.so -> /etc/alternatives/libblas.so-aarch64-linux-gnu

cd /etc/alternatives
ls -l liblas.so*

libblas.so.3-aarch64-linux-gnu -> /usr/lib/aarch64-linux-gnu/atlas/libblas.so.3
libblas.so-aarch64-linux-gnu -> /usr/lib/aarch64-linux-gnu/atlas/libblas.so

Let’s see that location…

chicken@chicken:/usr/lib/aarch64-linux-gnu/atlas$ ls -l
total 22652
libblas.a
libblas.so -> libblas.so.3.10.3
libblas.so.3 -> libblas.so.3.10.3
libblas.so.3.10.3
liblapack.a
liblapack.so -> liblapack.so.3.10.3
liblapack.so.3 -> liblapack.so.3.10.3
liblapack.so.3.10.3

And those are shared objects.

So why do we get ‘libblas.so.3: cannot open shared object file: No such file or directory?’

So let’s try 

sudo apt-get install libopenblas-dev liblapack-dev libatlas-base-dev gfortran



Sounds promising. Ha it worked.

chicken@chicken:/usr/lib/aarch64-linux-gnu/atlas$ python3
Python 3.6.9 (default, Dec  8 2021, 21:08:43) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>>

Now let’s try fix DustyNV’s Dockerfile. Oops right, it takes forever to build things, or even to download and install them. So try not to change things early on in the install. So besides, Dusty’s setup already has these being installed. So it’s not that it’s not there. It’s some linking issue.

Ok I start up the NV docker and try import cv2, but

admin@chicken:/workspaces/isaac_ros-dev$ python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/cv2/__init__.py", line 96, in <module>
    bootstrap()
  File "/usr/local/lib/python3.6/dist-packages/cv2/__init__.py", line 86, in bootstrap
    import cv2
ImportError: libtesseract.so.4: cannot open shared object file: No such file or directory

So where is that file?

admin@chicken:/usr/lib/aarch64-linux-gnu$ find | grep tess | xargs ls -l $1
-rw-r--r-- 1 root root 6892600 Apr  7  2018 ./libtesseract.a
lrwxrwxrwx 1 root root      21 Apr  7  2018 ./libtesseract.so -> libtesseract.so.4.0.0
-rw-r--r-- 1 root root     481 Apr  7  2018 ./pkgconfig/tesseract.pc

Well indeed, that plainly doesn’t exist?

admin@chicken:/$ sudo apt-get install libtesseract-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libarchive13 librhash0 libuv1
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libleptonica-dev
The following NEW packages will be installed:
  libleptonica-dev libtesseract-dev
0 upgraded, 2 newly installed, 0 to remove and 131 not upgraded.
Need to get 2,666 kB of archives.
After this operation, 14.1 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://ports.ubuntu.com/ubuntu-ports bionic/universe arm64 libleptonica-dev arm64 1.75.3-3 [1,251 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports bionic/universe arm64 libtesseract-dev arm64 4.00~git2288-10f4998a-2 [1,415 kB]
Fetched 2,666 kB in 3s (842 kB/s)           
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libleptonica-dev.
dpkg: warning: files list file for package 'libcufft-10-2' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'cuda-cudart-10-2' missing; assuming package has no files currently installed
(Reading database ... 97997 files and directories currently installed.)
Preparing to unpack .../libleptonica-dev_1.75.3-3_arm64.deb ...
Unpacking libleptonica-dev (1.75.3-3) ...
Selecting previously unselected package libtesseract-dev.
Preparing to unpack .../libtesseract-dev_4.00~git2288-10f4998a-2_arm64.deb ...
Unpacking libtesseract-dev (4.00~git2288-10f4998a-2) ...
Setting up libleptonica-dev (1.75.3-3) ...
Setting up libtesseract-dev (4.00~git2288-10f4998a-2) ...

ImportError: libtesseract.so.4: cannot open shared object file: No such file or directory

admin@chicken:/$ echo $LD_LIBRARY_PATH 
/opt/ros/foxy/install/opt/yaml_cpp_vendor/lib:/opt/ros/foxy/install/lib:/usr/lib/aarch64-linux-gnu/tegra-egl:/usr/local/cuda-10.2/targets/aarch64-linux/lib:/usr/lib/aarch64-linux-gnu/tegra:/opt/nvidia/vpi1/lib64:/usr/local/cuda-10.2/targets/aarch64-linux/lib::/opt/tritonserver/lib

So this sounds like a linker issue to me. We tell the linker where things are, it finds them.

admin@chicken:/$ export LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/atlas/:/usr/lib/aarch64-linux-gnu:/usr/lib/aarch64-linux-gnu/lapack:$LD_LIBRARY_PATH

admin@chicken:/$ sudo ldconfig

Ok, would be bad to have too much hope now. Let’s see… no, of course it didn’t work.

So let’s see what libleptonica and libtesseract-dev set up

./usr/lib/aarch64-linux-gnu/libtesseract.so
./usr/lib/aarch64-linux-gnu/libtesseract.a
./usr/lib/aarch64-linux-gnu/pkgconfig/tesseract.pc

And it wants 

admin@chicken:/$ ls -l ./usr/lib/aarch64-linux-gnu/libtesseract.so
lrwxrwxrwx 1 root root 21 Apr  7  2018 ./usr/lib/aarch64-linux-gnu/libtesseract.so -> libtesseract.so.4.0.0

and yeah it's installed.  

Start-Date: 2022-04-15  14:03:03
Commandline: apt-get install libtesseract-dev
Requested-By: admin (1000)
Install: libleptonica-dev:arm64 (1.75.3-3, automatic), libtesseract-dev:arm64 (4.00~git2288-10f4998a-2)
End-Date: 2022-04-15  14:03:06

This guy has a smart idea, to install them, which is pretty clever. But I tried that already, and tesseract’s build failed, of course. Then it complains about undefined references to jpeg,png,TIFF,zlib,etc. Hmm. All that shit is installed. 

/usr/lib/gcc/aarch64-linux-gnu/8/../../../aarch64-linux-gnu/liblept.a(libversions.o): In function `getImagelibVersions':
(.text+0x98): undefined reference to `jpeg_std_error'
(.text+0x158): undefined reference to `png_get_libpng_ver'
(.text+0x184): undefined reference to `TIFFGetVersion'
(.text+0x1f0): undefined reference to `zlibVersion'
(.text+0x21c): undefined reference to `WebPGetEncoderVersion'
(.text+0x26c): undefined reference to `opj_version'

But so here’s the evidence: cv2 is looking for libtesseract.so.4, which doesn’t exist at all. And even if we symlinked it to point to the libtesseract.so file, that just links to libtesseract.so.4.0.0 which is empty.

Ah. Ok I had to sudo apt-get install libtesseract-dev on the Jetson host, not inside the docker!!. Hmm. Right. Cause I’m sharing most of the libs on the host anyway. It’s gotta be on the host.

admin@chicken:/usr/lib/aarch64-linux-gnu$ ls -l *tess*
-rw-r--r-- 1 root root 6892600 Apr  7  2018 libtesseract.a
lrwxrwxrwx 1 root root      21 Apr  7  2018 libtesseract.so -> libtesseract.so.4.0.0
lrwxrwxrwx 1 root root      21 Apr  7  2018 libtesseract.so.4 -> libtesseract.so.4.0.0
-rw-r--r-- 1 root root 3083888 Apr  7  2018 libtesseract.so.4.0.0



admin@chicken:/usr/lib/aarch64-linux-gnu$ python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> exit()


Success.

So now back to earlier, we were trying to run jupyter lab, to try run the camera calibration code again. I added the installation to the dockerfile. So this command starts it up at http://chicken:8888/lab (or name of your computer).

jupyter lab --ip 0.0.0.0 --port 8888 --allow-root &
Needed matplotlib, so just did quick macro install:

%pip install matplotlib

ModuleNotFoundError: No module named 'matplotlib'

Note: you may need to restart the kernel to use updated packages.

K... restart kernel.  Kernel -> Restart.

Ok now I’m going to try calibrate the stereo cameras, since OpenCV is back.

Seems after some successes, the cameras are not creating capture sessions anymore, even from the host. Let’s reboot.

Alright, new topic. Calibrating stereo cameras.

Categories
chicken research chicken_research control sexing The Chicken Experience

DIY Incubator

Just a quick write-up on the incubator I made, for the fertilized eggs we picked up recently. They were R5 each (€0.28). The goal is to maintain a temperature of 37.5C.

The incubator consists of:

  • Arduino Nano
  • DS18B20 temperature sensor
  • 4.7K resistor
  • Solid state relay module
  • 12V power supply (for the Arduino, and relay)
  • A relatively inefficient 20W light, on a heatsink.
  • Some aluminium foil, in a mostly closed plastic container

    The eggs were placed a bit above the heatsink, in an egg carton, (hot air rises.) The idea being that we only want conductive heat, not radiative heat, since radiative heat is directional, and will heat one side of the egg more than the other side.

    The Arduino code was adapted from the example code from the OneWire library. The controller measures the temperature at the eggs, and turns on the light when the temperature is below 37.5C, and turns it off when it’s above. Using a separate temperature sensor, we confirmed that the temperature remains within a degree of the desired temperature.

There are better ways to do this, I’m sure, but this is what I came up with, on a day’s notice, with the parts available.

The gotchas encountered were, that the Chinese Arduino Nano I used required selecting to upload the sketch, for the ‘old’ bootloader, and the wiring colours of the DS18B20 were incorrectly labelled, (as it was for this forum user). 

#include <OneWire.h>

// OneWire DS18S20, DS18B20, DS1822 Temperature Example
//
// http://www.pjrc.com/teensy/td_libs_OneWire.html
//
// The DallasTemperature library can do all this work for you!
// https://github.com/milesburton/Arduino-Temperature-Control-Library



#define SENSOR 2
#define LIGHT 4

OneWire  ds(SENSOR);  // on pin 2 (a 4.7K resistor is necessary)

float desired_temp = 37.5;
float light_status = LOW;



void control(float temperature){
  if (temperature >= desired_temp)
  {
    light_status = LOW;
    Serial.println("HEATER OFF");
    
  }
  else 
  {
    light_status = HIGH;
    Serial.println("HEATER ON");
  }
  digitalWrite(LIGHT, light_status);
}

void setup(void) {
  Serial.begin(9600);
  pinMode(LIGHT, OUTPUT); 
}

void loop(void) {
  byte i;
  byte present = 0;
  byte type_s;
  byte data[12];
  byte addr[8];
  float celsius, fahrenheit;
  
  if ( !ds.search(addr)) {
    Serial.println("No more addresses.");
    Serial.println();
    ds.reset_search();
    delay(250);
    return;
  }
  
  Serial.print("ROM =");
  for( i = 0; i < 8; i++) {
    Serial.write(' ');
    Serial.print(addr[i], HEX);
  }

  if (OneWire::crc8(addr, 7) != addr[7]) {
      Serial.println("CRC is not valid!");
      return;
  }
  Serial.println();
 
  // the first ROM byte indicates which chip
  switch (addr[0]) {
    case 0x10:
      Serial.println("  Chip = DS18S20");  // or old DS1820
      type_s = 1;
      break;
    case 0x28:
      Serial.println("  Chip = DS18B20");
      type_s = 0;
      break;
    case 0x22:
      Serial.println("  Chip = DS1822");
      type_s = 0;
      break;
    default:
      Serial.println("Device is not a DS18x20 family device.");
      return;
  } 

  ds.reset();
  ds.select(addr);
  ds.write(0x44, 1);        // start conversion, with parasite power on at the end
  
  delay(1000);     // maybe 750ms is enough, maybe not
  // we might do a ds.depower() here, but the reset will take care of it.
  
  present = ds.reset();
  ds.select(addr);    
  ds.write(0xBE);         // Read Scratchpad

  Serial.print("  Data = ");
  Serial.print(present, HEX);
  Serial.print(" ");
  for ( i = 0; i < 9; i++) {           // we need 9 bytes
    data[i] = ds.read();
    Serial.print(data[i], HEX);
    Serial.print(" ");
  }
  Serial.print(" CRC=");
  Serial.print(OneWire::crc8(data, 8), HEX);
  Serial.println();

  // Convert the data to actual temperature
  // because the result is a 16 bit signed integer, it should
  // be stored to an "int16_t" type, which is always 16 bits
  // even when compiled on a 32 bit processor.
  int16_t raw = (data[1] << 8) | data[0];
  if (type_s) {
    raw = raw << 3; // 9 bit resolution default
    if (data[7] == 0x10) {
      // "count remain" gives full 12 bit resolution
      raw = (raw & 0xFFF0) + 12 - data[6];
    }
  } else {
    byte cfg = (data[4] & 0x60);
    // at lower res, the low bits are undefined, so let's zero them
    if (cfg == 0x00) raw = raw & ~7;  // 9 bit resolution, 93.75 ms
    else if (cfg == 0x20) raw = raw & ~3; // 10 bit res, 187.5 ms
    else if (cfg == 0x40) raw = raw & ~1; // 11 bit res, 375 ms
    //// default is 12 bit resolution, 750 ms conversion time
  }
  celsius = (float)raw / 16.0;
  fahrenheit = celsius * 1.8 + 32.0;
  Serial.print("  Temperature = ");
  Serial.print(celsius);
  Serial.print(" Celsius, ");
  Serial.print(fahrenheit);
  Serial.println(" Fahrenheit");

  control(celsius);
}

As the eggs reach maturity, we’ll get a ‘hatcher’ environment ready.

Categories
bio chicken research CNNs evolution sexing Vision

Speckled eggs

Finally tried shining light through an egg, and discovered calcium deposits making things opaque. Hmm.

This led to finding out about all these abnormal eggs.

Categories
AI/ML Behaviour bio chicken_research control deep dev ears evolution highly_speculative neuro UI

Hierarchical Temporal Memory

Here I’m continuing with the task of unsupervised detection of audio anomalies, hopefully for the purpose of detecting chicken stress vocalisations.

After much fussing around with the old Numenta NuPic codebase, I’m porting the older nupic.audio and nupic.critic code, over to the more recent htm.core.

These are the main parts:

  • Sparse Distributed Representation (SDR)
  • Encoders
  • Spatial Pooler (SP)
  • Temporal Memory (TM)

I’ve come across a very intricate implementation and documentation, about understanding the important parts in the HTM model, way deep, like how did I get here? I will try implement the ‘critic’ code, first. Or rather, I’ll try port it from nupic to htm. After further investigation, there’s a few options, and I’m going to try edit the hotgym example, and try shove wav files frequency band scalars through it instead of power consumption data. I’m simplifying the investigation. But I need to make some progress.

I’m using this docker to get in, mapping my code and wav file folder in:

docker run -d -p 8888:8888 --name jupyter -v /media/chrx/0FEC49A4317DA4DA/sounds/:/home/jovyan/work 3rdman/htm.core-jupyter:latest



So I've got some code working that writes to 'nupic format' (.csv) and code that reads the amplitudes from the csv file, and then runs it through htm.core. 

So it takes a while, and it's just for 1 band (of 10 bands). I see it also uses the first 1/4 of so of the time to know what it's dealing with.  Probably need to run it through twice to get predictive results in the first 1/4. 

Ok no, after a few weeks, I've come back to this point, and realise that the top graph is the important one.  Prediction is what's important.  The bottom graphs are the anomaly scores, used by the prediction.
Frequency Band 0

The idea in nupic.critic, was to threshold changes in X bands. Let’s see the other graphs…

Frequency band 0: 0-480Hz ?
Frequency band 2: 960-1440Hz ?
Frequency band 3: 1440-1920Hz ?
Frequency band 4: 1920-2400Hz ?
Frequency band 5: 2400-2880Hz ?
Frequency band 6: 2880-3360Hz ?

Ok Frequency bands 7, 8, 9 were all zero amplitude. So that’s the highest the frequencies went. Just gotta check what those frequencies are, again… 

Opening 307.wav
Sample width (bytes): 2
Frame rate (sampling frequency): 48000
Number of frames: 20771840
Signal length: 20771840
Seconds: 432
Dimensions of periodogram: 4801 x 2163

Ok with 10 buckets, 4801 would divide into 
Frequency band 0: 0-480Hz
Frequency band 1: 480-960Hz
Frequency band 2: 960-1440Hz
Frequency band 3: 1440-1920Hz
Frequency band 4: 1920-2400Hz
Frequency band 5: 2400-2880Hz
Frequency band 6: 2880-3360Hz

Ok what else. We could try segment the audio by band, so we can narrow in on the relevant frequency range, and then maybe just focus on that smaller range, again, in higher detail.

Learning features with some labeled data, is probably the correct way to do chicken stress vocalisation detections.

Unsupervised anomaly detection might be totally off, in terms of what an anomaly is. It is probably best, to zoom in on the relevant bands and to demonstrate a minimal example of what a stressed chicken sounds like, vs a chilled chicken, and compare the spectrograms to see if there’s a tell-tale visualisable feature.

A score from 1 to 5 for example, is going to be anomalous in arbitrary ways, without labelled data. Maybe the chickens are usually stressed, and the anomalies are when they are unstressed, for example.

A change in timing in music might be defined, in some way. like 4 out of 7 bands exhibiting anomalous amplitudes. But that probably won’t help for this. It’s probably just going to come down to a very narrow band of interest. Possibly pointing it out on a spectrogram that’s zoomed in on the feature, and then feeding the htm with an encoding of that narrow band of relevant data.

I’ll continue here, with some notes on filtering. After much fuss, the sox app (apt-get install sox) does it, sort of. Still working on python version.

                                                                              $ sox 307_0_50.wav filtered_50_0.wav sinc -n 32767 0-480
$ sox 307_0_50.wav filtered_50_1.wav sinc -n 32767 480-960
$ sox 307_0_50.wav filtered_50_2.wav sinc -n 32767 960-1440
$ sox 307_0_50.wav filtered_50_3.wav sinc -n 32767 1440-1920
$ sox 307_0_50.wav filtered_50_4.wav sinc -n 32767 1920-2400
$ sox 307_0_50.wav filtered_50_5.wav sinc -n 32767 2400-2880
$ sox 307_0_50.wav filtered_50_6.wav sinc -n 32767 2880-3360


So, sox does seem to be working.  The mel spectrogram is logarithmic, which is why it looks like this.

Visually, it looks like I'm interested in 2048 to 4096 Hz.  That's where I can see the chirps.

Hmm. So I think the spectrogram is confusing everything.

So where does 4800 come from? 48 kHz. 48,000 Hz (48 kHz) is the sample rate “used for DVDs“.

Ah. Right. The spectrogram values represent buckets of 5 samples each, and the full range is to 24000…? 

Sample width (bytes): 2
0.     5.    10.    15.    20.    25.    30.    35.    40.    45.    50.    55.    60.    65.    70.    75.    80.    85.    90.    95.    100.   105.   110.   115.   120.   125.   130.   135.   140.   145.
...
 23950. 23955. 23960. 23965. 23970. 23975. 23980. 23985. 23990. 23995. 24000.]

ok. So 2 x 24000. Maybe 2 channels? Anyway, full range is to 48000Hz. In that case, are the bands actually…

Frequency band 0: 0-4800Hz
Frequency band 1: 4800-9600Hz
Frequency band 2: 9600-14400Hz
Frequency band 3: 14400-19200Hz
Frequency band 4: 19200-24000Hz
Frequency band 5: 24000-28800Hz
Frequency band 6: 28800-33600Hz

Ok so no, it’s half the above because of the sample width of 2.

Frequency band 0: 0-2400Hz
Frequency band 1: 2400-4800Hz
Frequency band 2: 4800-7200Hz
Frequency band 3: 7200-9600Hz
Frequency band 4: 9600-12000Hz
Frequency band 5: 12000-14400Hz
Frequency band 6: 14400-16800Hz

So why is the spectrogram maxing at 8192Hz? Must be spectrogram sampling related.

ol_hann_win
From Berkeley document

So the original signal is 0 to 24000Hz, and the spectrogram must be 8192Hz because… the spectrogram is made some way. I’ll try get back to this when I understand it. 

sox 307_0_50.wav filtered_50_0.wav sinc -n 32767 0-2400
sox 307_0_50.wav filtered_50_1.wav sinc -n 32767 2400-4800
sox 307_0_50.wav filtered_50_2.wav sinc -n 32767 4800-7200
sox 307_0_50.wav filtered_50_3.wav sinc -n 32767 7200-9600
sox 307_0_50.wav filtered_50_4.wav sinc -n 32767 9600-12000
sox 307_0_50.wav filtered_50_5.wav sinc -n 32767 12000-14400
sox 307_0_50.wav filtered_50_6.wav sinc -n 32767 14400-16800

Ok I get it now.

Ok i don’t entirely understand the last two. But basically the mel spectrogram is logarithmic, so those high frequencies really don’t get much love on the mel spectrogram graph. Buggy maybe.

But I can estimate now the chirp frequencies…

sox 307_0_50.wav filtered_bird.wav sinc -n 32767 1800-5200

Beautiful. So, now to ‘extract the features’…

So, the nupic.critic code with 1 bucket managed to get something resembling the spectrogram. Ignore the blue.

But it looks like maybe, we can even just threshold and count peaks. That might be it. 

sox 307.wav filtered_307.wav sinc -n 32767 1800-5200
sox 3072.wav filtered_3072.wav sinc -n 32767 1800-5200
sox 237.wav filtered_237.wav sinc -n 32767 1800-5200
sox 98.wav filtered_98.wav sinc -n 32767 1800-5200

Let’s do the big files…

Ok looks good enough.

So now I’m plotting the ‘chirp density’ (basically volume).

’98.wav’
‘237.wav’
‘307.wav’
‘3072.wav’

In this scheme, we just proxy chirp volume density as a variable representing stress.  We don’t know if it is a true proxy.
As you can see, some recordings have more variation than others.  

Some heuristic could be decided upon, for rating the stress from 1 to 5.  The heuristic depends on how the program would be used.  For example, if it were streaming audio, for an alert system, it might alert upon some duration of time spent above one standard deviation from the rolling mean. I’m not sure how the program would be used though.

If the goal were to differentiate stressed and not stressed vocalisations, that would require labelled audio data.   

(Also, basically didn’t end up using HTM, lol)

Categories
Behaviour bio chicken research chicken_research deep dev ears institutes neuro The Chicken Experience

Stress Vocalisations

We’ve spoken with Dr. Maksimiljan Brus, at the University of Maribor, and he’s sent us some WAV file recordings of a large group of chickens.

There seems to be a decent amount of work done, particularly at Georgia Tech, regarding categorizing chicken sounds, to detect stress, or bronchitis, etc. They’ve also done some experiments to see how chickens react to humans and robots. (It takes them about 3 weeks to get used to either).

In researching the topic, there was a useful South African document related to smallholding size chicken businesses. It covers everything. Very good resource, actually, and puts into perspective the relative poverty in the communities where people sell chickens for a living. The profit margin per chicken in 2013 was about R12 per live chicken (less than 1 euro).

From PRODUCTION GUIDELINES
for Small-Scale Broiler Enterprises
K Ralivhesa, W van Averbeke
& FK Siebrits

So anyway, I’m having a look at the sound files, to see what data and features I can extract. There’s no labels, so there won’t be any reinforcement learning here. Anomaly detection doesn’t need labels, and can use moving window statistics, to notice when something is out of the ordinary. So that’s what I’m looking into.

I am personally interested in Numenta’s algorithms, such as HTM, which use a model of cortical columns, and sparse encodings, to predict, and detect anomalies. I looked into getting Nupic.critic working, but Nupic is so old now, written in Python 2, that it’s practically impossible to get working. There is a community fork, htm.core, updated to Python 3, but it’s missing parts of the nupic codebase that nupic.critic is relying on. I’m able to convert the sound files to the nupic format, but am stuck for now, when running the analysis.

So let’s start at a more basic level and work our way up.

I downloaded Praat, an interesting sound analysis program used for some audio research. Not sure if it’s useful here. But it’s able to show various sound features. I’ll close it again, for now.

So, first thing to do, is going to be Mel spectrograms, and possibly Mel Frequency Cepstral Coefficients (MFCCs). The Mel scale kinda allows a difference between 250Hz and 500Hz to be scaled to the same size as a difference between 13250Hz and 13500Hz. It’s log-scaled.

Mel spectrograms let you use visual tools on audio. Also, worth knowing what a feature is, in machine learning. It’s a measurable property.

Ok where to start? Maybe librosa and PyOD?

pip install librosa

Ok and this outlier detection medium writeup, PyOD, says

Neural Networks

Neural networks can also be trained to identify anomalies.

Autoencoder (and variational autoencoder) network architectures can be trained to identify anomalies without labeled instances. Autoencoders learn to compress and reconstruct the information in data. Reconstruction errors are then used as anomaly scores.

More recently, several GAN architectures have been proposed for anomaly detection (e.g. MO_GAAL).

There’s also the results of a group working on this sort of problem, here.

A relevant arxiv too:

ANOMALOUS SOUND DETECTION BASED ON
INTERPOLATION DEEP NEURAL NETWORK

And

UNSUPERVISED ANOMALOUS SOUND DETECTION VIA AUTOENCODER APPROACH

What is this DCASE?

Hmm so there is a challenge for it currently. It’s big in Japan. Here’s the winning solution:

Amazon programmers win an Amazon competition on anomaly detection.

Here was an illustrative example of an anomaly, of some machine sound.

And of course, there are more traditional? algorithms, (data-science algorithms). Here’s a medium article overview, for a submission to a heart murmur challenge. It mentions kapre, “Keras Audio Preprocessors – compute STFT, ISTFT, Melspectrogram, and others on GPU real-time.”

And I found ‘torchaudio‘,

Here’s a useful flowchart from a paper about edge sound analysis on a Teensy. Smart Audio Sensors (SASs). The code “computes the FFT and Mel coefficients of a recorded audio frame.”

Smart Audio Sensors in the Internet of Things
Edge for Anomaly Detection

I haven’t mentioned it, but of course FFT, Fast Fourier Transform, which converts audio to frequency bands, is going to be a useful tool, too. “The FFT’s importance derives from the fact that it has made working in the frequency domain equally computationally feasible as working in the temporal or spatial domain. ” – (wikipedia)

On the synthesis and possibly artistic end, there’s also MelGAN and the like.

Google’s got pipelines in kubernetes ? MLOps stuff.

Artistically speaking, it sounds like we want spectrograms. Someone implements one from scratch here, and there is a link to a good youtube video on relevant sound analysis ideas. Wide-band, vs. narrow-band, for example. Overlapping windows? They’re explaining STFT, which is used to make spectrograms.

There’s also something called Chirp Z transform.

Anyway. Good stuff. As always, I find the hardest part is finding your way back to your various dev environments. Ok I logged into the Jupyter running in the docker on the Jetson. ifconfig to get the ip, and http://192.168.101.115:8888/lab, voila).

Ok let’s see torchaudio’s colab… and pip install, ok… Here’s a summary of the colab.

Some ghostly Mel spectrogram stuff. Also, interesting ‘To recover a waveform from spectrogram, you can use GriffinLim.’

Ok let’s get our own dataset prepared. We need an anomaly detector. Let’s see…

———————— <LIBROSA INSTALLATION…> —————

Ok the librosa mel spectrogram is working, at least, so far. So these are the images for the 4 files Dr. Brus sent.

While looking for something like STFT to make a spectogram video, i came across this resource: Machine Hearing. Also this tome of ML resources.

Classification is maybe the best way to do this ML stuff. Then you can add labels to classes, and train a neural network to associate labels, and to categorise. So it would be ideal, if the data were pre-labelled, i.e. classified by chicken stress vocalisation experts. Like here is a soundset with metadata, that lets you classify sounds with labels, (with training).

So we really do need to use an anomaly detection algorithm, because I listened to the chickens for a little bit, and I’m not getting the nuances.

Here’s a relevant paper, which learns classes, for retroactive labelling. They’re recording a machine making sounds, and then humans label it. They say 1NN (k-nearest-neighbours) is hard to beat, but it’s memory intensive. “Nearest centroid (NC) combined with DBA has been shown to be competitive with kNN at a much smaller computational cost”.

Here’s pyAudioAnalysis

Perform unsupervised segmentation (e.g. speaker diarization) and extract audio thumbnails

  • Train and use audio regression models (example application: emotion recognition)
  • Apply dimensionality reduction to visualize audio data and content similarities

Here’s a cool visualiser, in tensorboard,

Ideally, I would like to use NuPIC.

https://numenta.com/resources/biological-and-machine-intelligence/

Ok, let’s hope this old link works, for a nupic docker.

sudo docker run -i -t numenta/nupic /bin/bash

Ok amazing. Ok right, trying to install matplotlib inside the docker crashes. urllib3. I’ve been here before. Right, I asked on the github issues. 14 days ago, I asked. I got htm.core working. But it doesn’t have nupic.data classes.

After bashing my head against the apparent impossibility to pip install urllib3 and matplotlib in a python 2.7 docker, I’ve decided I will have to port the older nupic.critic or nupic.audio code to htm.core.

I cleared up some harddrive space, and ran this docker:

docker run -d -p 8888:8888 --name jupyter 3rdman/htm.core-jupyter:latest

then get the token for the URL:
docker logs -f jupyter

There’s a lot to go through, and I’m a noob at HTM. So I will start a new article now, on HTM specifically, for this.

Categories
AI/ML deep dev envs meta

Codex

I applied for the OpenAI Codex beta, and tried it out. It’s very cool.

It’s like GPT trained on Github or something like that, so you can just give it a prompt, and it tries to carry on the sentence with what it expects next.

So the easy way to use it, I’ve found, is to write a comment for the code you want it to write for you. I switched the engine to ‘davinci-codex’ and cranked up the response length to 1600 or so. Sometimes it just makes up more comments, or does something silly. But now and then it hits a home run.

Here’s an example.

"""
Take an example wav file, found at file_name="/home/jovyan/work/307.wav", and plot a mel spectrogram of it. 
""" 

import librosa
import librosa.display
import matplotlib.pyplot as plt

file_name="/home/jovyan/work/307.wav"

y, sr = librosa.load(file_name)

plt.figure(figsize=(12, 4))
librosa.display.waveplot(y, sr=sr)
plt.show()

S = librosa.feature.melspectrogram(y, sr=sr, n_mels=128)
log_S = librosa.power_to_db(S, ref=np.max)
plt.figure(figsize=(12, 4))
librosa.display.specshow(log_S, sr=sr, x_axis='time', y_axis='mel')
plt.title('mel power spectrogram')
plt.colorbar(format='%+02.0f dB')
plt.tight_layout()
plt.show()

I had to install librosa, but then it worked. It did more than I asked for.

Then I fed that whole thing, with another 

"""
Plot the MFCC of file_name="/home/jovyan/work/307.wav"
"""

and it carried like an example giving champ:

"""
Take an example wav file, found at file_name="/home/jovyan/work/307.wav", and plot a mel spectrogram of it. 
""" 

import librosa
import librosa.display
import matplotlib.pyplot as plt

file_name="/home/jovyan/work/307.wav"

y, sr = librosa.load(file_name)

plt.figure(figsize=(12, 4))
librosa.display.waveplot(y, sr=sr)
plt.show()

S = librosa.feature.melspectrogram(y, sr=sr, n_mels=128)
log_S = librosa.power_to_db(S, ref=np.max)
plt.figure(figsize=(12, 4))
librosa.display.specshow(log_S, sr=sr, x_axis='time', y_axis='mel')
plt.title('mel power spectrogram')
plt.colorbar(format='%+02.0f dB')
plt.tight_layout()
plt.show()




"""
Plot the MFCC of file_name="/home/jovyan/work/307.wav"
"""

import librosa
import librosa.display
import matplotlib.pyplot as plt

file_name="/home/jovyan/work/307.wav"

y, sr = librosa.load(file_name)

plt.figure(figsize=(12, 4))
librosa.display.waveplot(y, sr=sr)
plt.show()

mfccs = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=40)
plt.figure(figsize=(12, 4))
librosa.display.specshow(mfccs, x_axis='time')
plt.colorbar()
plt.title('MFCC')
plt.tight_layout()
plt.show()


"""
Plot the chromagram of file_name="/home/jovyan/work/307.wav"
"""

import librosa
import librosa.display
import matplotlib.pyplot as plt

file_name="/home/jovyan/work/307.wav"

y, sr = librosa.load(file_name)

plt.figure(figsize=(12, 4))
librosa.display.waveplot(y, sr=sr)
plt.show()

chroma = librosa.feature.chroma_stft(y=y, sr=sr)
plt.figure(figsize=(12, 4))
librosa.display.specshow(chroma, y_axis='chroma', x_axis='time')
plt.colorbar()
plt.title('Chromagram')
plt.tight_layout()
plt.show()


"""
Plot the chromagram of file_name="/home/jovyan/work/307.wav"
"""

import librosa
import librosa.display
import matplotlib.pyplot as plt

file_name="/home/jovyan/work/307.wav"

y, sr = librosa.load(file_name)

plt.figure(figsize=(12, 4))
librosa.display.waveplot(y, sr=sr)
plt.show()

chroma = librosa.feature.chroma_stft(y=y, sr=sr)
plt.figure(figsize=(12, 4))
librosa.display.specshow(chroma, y_axis='chroma', x_axis='time')
plt.colorbar()
plt.title('Chromagram')
plt.tight_layout()
plt.show()


"""
Plot the tonnetz of file_name="/home/jovyan/work/307.wav"
"""

import librosa
import librosa.display
import matplotlib.pyplot as plt

file_name="/home/jovyan/work/307.wav"

y, sr = librosa.load(file_name)

plt.figure(figsize=(12, 4))
librosa.display.waveplot(y, sr=sr)
plt.show()

tonnetz = librosa.feature.tonnetz(y=y, sr=sr)
plt.figure(figsize=(12, 4))
librosa.display.specshow(tonnetz, y_axis='tonnetz', x_axis='time')
plt.colorbar()
plt.title('Tonal Centroids (Tonnetz)')
plt.tight_layout()
plt.show()


"""
Plot the zero-crossing rate of file_name="/home/jovyan/work/307.wav"
"""

import librosa
import librosa.display
import matplotlib.pyplot as plt

file_name="/home/jovyan/work/307.wav"

y, sr = librosa.load(file_name)

plt.figure(figsize=(12, 4))
librosa.display.waveplot(y, sr=sr)
plt.show()

zrate = librosa.feature.zero_crossing_rate(y)
plt.figure(figsize=(12, 4))
librosa.display.specshow(zrate, x_axis='time')
plt.colorbar()
plt.title('Zero Crossing Rate')
plt.tight_layout()
plt.show()


"""
Plot the spectral centroid of file_name="/home/jovyan/work/307.wav"
"""

import librosa
import librosa.display
import matplotlib.pyplot as plt

file_name="/home/jovyan/work/307.wav"

y, sr = librosa.load(file_name)

plt.figure(figsize=(12, 4))
librosa.display.waveplot(y, sr=sr)
plt.show()

cent = librosa.feature.spectral_centroid(y=y, sr=sr)
plt.figure(figsize=(12, 4))
librosa.display.specshow(cent, x_axis='time')
plt.colorbar()
plt.title('Spectral Centroids')
plt.tight_layout()
plt.show()


"""
Plot the spectral bandwidth of file_name="/home/jovyan/work/307.wav"
"""

import librosa
import librosa.display
import matplotlib.pyplot as plt

file_name="/home/jovyan/work/307.wav"

y, sr = librosa.load(file_name)

plt.figure(figsize=(12, 4))
librosa.display.waveplot(y, sr=sr)
plt.show()

spec_bw = librosa.feature.spectral_bandwidth(y=y, sr=sr)
plt.figure(figsize=(12, 4))
librosa.display.specshow(spec_bw, x_axis='time')
plt.colorbar()
plt.title('Spectral Bandwidth')
plt.tight_layout()
plt.show()


"""
Plot the spectral rolloff of file_name="/home/jovyan/work/307.wav"
"""

import librosa
import librosa.display
import matplotlib.pyplot as plt

file_name="/home/jovyan/work/307.wav"

y, sr = librosa.load(file_name)

plt.figure(figsize=(12, 4))
librosa.display.waveplot(y, sr=sr)
plt.show()

spec_rolloff = lib

After waiting a while… we get:

Ok, Chromagram twice… bold move, Codex. 
import librosa
import librosa.display
import matplotlib.pyplot as plt

file_name="/home/jovyan/work/307.wav"

y, sr = librosa.load(file_name)

plt.figure(figsize=(12, 8))
D = librosa.amplitude_to_db(np.abs(librosa.stft(y)), ref=np.max)
plt.subplot(4, 2, 1)
librosa.display.specshow(D, y_axis='linear')
plt.colorbar(format='%+2.0f dB')
plt.title('Linear-frequency power spectrogram')

plt.subplot(4, 2, 2)
librosa.display.specshow(D, y_axis='log')
plt.colorbar(format='%+2.0f dB')
plt.title('Log-frequency power spectrogram')

plt.subplot(4, 2, 3)
librosa.display.specshow(D, x_axis='time')
plt.colorbar(format='%+2.0f dB')
plt.title('Time-frequency power spectrogram')

plt.subplot(4, 2, 4)
librosa.display.specshow(D, x_axis='time', y_axis='log')
plt.colorbar(format='%+2.0f dB')
plt.title('Log-Time-frequency power spectrogram')
Categories
highly_speculative meta UI

“Mechanical Turk”ing

Audience participation could adds data points and labels, for classification training or similar. But what?

Classification needs a user interface. I saw one here:

Informatics 06 00038 g004 550
Collecting Labels for Rare Anomalies via Direct Human Feedback—An Industrial Application Study

“What type of anomaly is this?”

Informatics 06 00038 g001 550
Reporting an Anomaly

Here is Miranda demonstrating a similar skill

Categories
evolution highly speculative highly_speculative meta The Chicken Experience

Ameglian Major Cow, Chairdogs, etc.

I thought it’s probably worth noting some sci-fi ideas that I remember…

The Ameglian Major Cow (or, Dish of the Day), was the cow in Hitchhiker’s guide book 2, (Restaurant at the end of the Universe), that has been bred to want to be eaten, and when Zaphod Beeblebrox, etc., order steak, the cow goes off to shoot itself, and tells the protagonist, Arthur not to worry: “I’ll be very humane.”

The other thing was chairdogs, that show up in one of the later Dune books. The Tleilaxu are known for taking genetic engineering a bit too far, and one of their exports is a dog that’s been bred until it has become a chair, which massages you. Real ‘creature comforts’. It’s generally used for world building and character development, maybe insinuating that the characters with guilty-pleasure chairdogs, are getting soft.

Interesting, because the artificial selection or genetic engineering leading to these creations is questionable, but the final product is something that inverts or sidesteps morality.

The Ameglian Major Cow is a common thought experiment, as it postulates a question to vegetarians, whether they would eat meat, if it were intelligent, and wanted to be eaten. It’s very hypothetical though.

Chairdogs are closer: if chickens of tomorrow, or other domesticated animals, are ultimately evolved or engineered into simplified protein vats, their ‘souls’ (i.e. CNS) removed, perhaps we’re left with something less problematic, despite the apparent abominating of nature.

Side note: Alex O’Connor, Cosmic Skeptic, has a lot to say, on the philosophy of ethical veganism. Here he answers a question: “Do Animals have a “Right to Life?” – (tl;dw: no. but you should eat less meat)

Categories
AI/ML arxiv Behaviour control deep Locomotion simulation

PGA Map Elites

Though I’m generally using stable baseline algorithms for training locomotion tasks, I am sometimes drawn back to evolutionary algorithms, and especially Map Elites, which has now been upgraded to incorporate a policy gradient.

The archiving of behaviours is what attracts me to Map Elites.

PGA Map Elites based on top of QDGym, which tracks Quality Diversity, is probably worth a look.

Categories
3D AI/ML arxiv CNNs control deep envs sim2real simulation Vision

UnrealROX

A sim2real with more or less photorealism, since Unreal Engine is so pretty. arxiv: https://arxiv.org/pdf/1810.06936.pdf

And the github writeup: https://sim2realai.github.io/UnrealROX/

Beautiful. Since we’re currently just using a 256×256 view port in pybullet, this is quite a bit more advanced than required though. Learning game engines can also take a while. It took me about a month to learn Unity3d, with intermediate C# experience. Unreal Engine uses C++, so it’s a bit less accessible to beginners.