Just a quick write-up on the incubator I made, for the fertilized eggs we picked up recently. They were R5 each (€0.28). The goal is to maintain a temperature of 37.5C.
The incubator consists of:
Arduino Nano
DS18B20 temperature sensor
4.7K resistor
Solid state relay module
12V power supply (for the Arduino, and relay)
A relatively inefficient 20W light, on a heatsink.
Some aluminium foil, in a mostly closed plastic container
The eggs were placed a bit above the heatsink, in an egg carton, (hot air rises.) The idea being that we only want conductive heat, not radiative heat, since radiative heat is directional, and will heat one side of the egg more than the other side.
The Arduino code was adapted from the example code from the OneWire library. The controller measures the temperature at the eggs, and turns on the light when the temperature is below 37.5C, and turns it off when it’s above. Using a separate temperature sensor, we confirmed that the temperature remains within a degree of the desired temperature.
There are better ways to do this, I’m sure, but this is what I came up with, on a day’s notice, with the parts available.
The gotchas encountered were, that the Chinese Arduino Nano I used required selecting to upload the sketch, for the ‘old’ bootloader, and the wiring colours of the DS18B20 were incorrectly labelled, (as it was for this forum user).
#include <OneWire.h>
// OneWire DS18S20, DS18B20, DS1822 Temperature Example
//
// http://www.pjrc.com/teensy/td_libs_OneWire.html
//
// The DallasTemperature library can do all this work for you!
// https://github.com/milesburton/Arduino-Temperature-Control-Library
#define SENSOR 2
#define LIGHT 4
OneWire ds(SENSOR); // on pin 2 (a 4.7K resistor is necessary)
float desired_temp = 37.5;
float light_status = LOW;
void control(float temperature){
if (temperature >= desired_temp)
{
light_status = LOW;
Serial.println("HEATER OFF");
}
else
{
light_status = HIGH;
Serial.println("HEATER ON");
}
digitalWrite(LIGHT, light_status);
}
void setup(void) {
Serial.begin(9600);
pinMode(LIGHT, OUTPUT);
}
void loop(void) {
byte i;
byte present = 0;
byte type_s;
byte data[12];
byte addr[8];
float celsius, fahrenheit;
if ( !ds.search(addr)) {
Serial.println("No more addresses.");
Serial.println();
ds.reset_search();
delay(250);
return;
}
Serial.print("ROM =");
for( i = 0; i < 8; i++) {
Serial.write(' ');
Serial.print(addr[i], HEX);
}
if (OneWire::crc8(addr, 7) != addr[7]) {
Serial.println("CRC is not valid!");
return;
}
Serial.println();
// the first ROM byte indicates which chip
switch (addr[0]) {
case 0x10:
Serial.println(" Chip = DS18S20"); // or old DS1820
type_s = 1;
break;
case 0x28:
Serial.println(" Chip = DS18B20");
type_s = 0;
break;
case 0x22:
Serial.println(" Chip = DS1822");
type_s = 0;
break;
default:
Serial.println("Device is not a DS18x20 family device.");
return;
}
ds.reset();
ds.select(addr);
ds.write(0x44, 1); // start conversion, with parasite power on at the end
delay(1000); // maybe 750ms is enough, maybe not
// we might do a ds.depower() here, but the reset will take care of it.
present = ds.reset();
ds.select(addr);
ds.write(0xBE); // Read Scratchpad
Serial.print(" Data = ");
Serial.print(present, HEX);
Serial.print(" ");
for ( i = 0; i < 9; i++) { // we need 9 bytes
data[i] = ds.read();
Serial.print(data[i], HEX);
Serial.print(" ");
}
Serial.print(" CRC=");
Serial.print(OneWire::crc8(data, 8), HEX);
Serial.println();
// Convert the data to actual temperature
// because the result is a 16 bit signed integer, it should
// be stored to an "int16_t" type, which is always 16 bits
// even when compiled on a 32 bit processor.
int16_t raw = (data[1] << 8) | data[0];
if (type_s) {
raw = raw << 3; // 9 bit resolution default
if (data[7] == 0x10) {
// "count remain" gives full 12 bit resolution
raw = (raw & 0xFFF0) + 12 - data[6];
}
} else {
byte cfg = (data[4] & 0x60);
// at lower res, the low bits are undefined, so let's zero them
if (cfg == 0x00) raw = raw & ~7; // 9 bit resolution, 93.75 ms
else if (cfg == 0x20) raw = raw & ~3; // 10 bit res, 187.5 ms
else if (cfg == 0x40) raw = raw & ~1; // 11 bit res, 375 ms
//// default is 12 bit resolution, 750 ms conversion time
}
celsius = (float)raw / 16.0;
fahrenheit = celsius * 1.8 + 32.0;
Serial.print(" Temperature = ");
Serial.print(celsius);
Serial.print(" Celsius, ");
Serial.print(fahrenheit);
Serial.println(" Fahrenheit");
control(celsius);
}
As the eggs reach maturity, we’ll get a ‘hatcher’ environment ready.
Interesting as a guideline for comparison with international efforts, and for perspective of the sort of money in this problem. “the industry could save between $1.5 billion and $2.5 billion each year.” – News Article.
The Egg-Tech Prize Phase II criteria forms the basis for the merit-based review, outlined above.
Day and potential to utilize male eggs (up to 25 points).
Minimum: Functions on or before day 8 of incubation. Preference for solutions with reduced incubation time with pre-incubation most preferred. Protocols involving short periods of incubation during egg storage (SPIODES) will be considered preincubation and given preference. Preference will be given to technologies that enable the use of male eggs in other applications.
Accuracy (up to 20 points).
Minimum: 98 percent accuracy. Preference will be given to technologies that work with all chicken breeds/colors commonly used in commercial production.
Economic Feasibility (up to 20 points).
Score for this criterion will consider economic feasibility based on a cost-benefit analysis and business plan that should include:
Direct costs:
Capital costs incurred by technology developer, per hatchery
Capital investment for equipment/structure modification by hatchery
Predicted annual maintenance costs
Predicted annual consumables costs
Predicted personnel training and labor requirements (hours)
Indirect costs:
Expected utilities requirements of technology
Potential revenue models
Lease, subscription, sales, other.
Other revenue streams for developer
Predicted revenues gained for hatchery in diverting eggs, energy savings, labor, cost-savings from not feeding male chicks (depending on country), etc.
Throughput and physical size (up to 15 points)
Potential for sexing at least 15,000 eggs per hour (more preferred). If multiple units will be used in combination to achieve the desired throughput, only one demonstration unit will be required but all units needed to meet the desired throughput must fit into existing hatchery structures, with reasonable and appropriate modifications.
Hatchability (up to 15 points)
Minimum: Does not reduce hatching rate by more than 1.5 percent from baseline.
Speed of test results (up to 5 points)
Results returned in less than 30 min if eggs are tested during incubation (allowable time for removal, testing and return to incubator).† If eggs are tested prior to incubation, with or without SPIDES, results must be available within 48 hours of testing. Accurate tracking and identification of eggs must be demonstrated.
†Longer times until test result will require placing eggs back into the incubator, in which case they must be removed again for sorting.
We’ve got an egg in the gym environment now, so we need to collect some data for training the robot to go pick up an egg.
I’m going to have it save the rgba, depth and segmentation images to disk for Unet training. I left out the depth image for now. The pictures don’t look useful. But some papers are using the depth, so I might reconsider. Some weed bot paper uses 14-channel images with all sorts of extra domain specific data relevant to plants.
I wrote some code to take pics if the egg was in the viewport, and it took 1000 rgb and segmentation pictures or so. I need to change the colour of the egg for sure, and probably randomize all the textures a bit. But main thing is probably to make the segmentation layers with pixel colours 0,1,2, etc. so that it detects the egg and not so much the link in the foreground.
So sigmoid to softmax and so on. Switching to multi-class also begs the question whether to switch to Pytorch & COCO panoptic segmentation based training. It will have to happen eventually, as I think all of the fastest implementations are currently in Pytorch and COCO based. Keras might work fine for multiclass or multiple binary classification, but it’s sort of the beginning attempt. Something that works. More proof of concept than final implementation. But I think Keras will be good enough for these in-simulation 256×256 images.
Regarding multi-class segmentation, karolzak says “it’s just a matter of changing num_classes argument and you would need to shape your mask in a different way (layer per class??), so for multiclass segmentation you would need a mask of shape (width, height, num_classes)“
I’ll keep logging my debugging though, if you’re reading this.
So I ran segmask_linkindex.py to see what it does, and how to get more useful data. The code is not running because the segmentation image actually has an array of arrays. I presume it’s a numpy array. I think it must be the rows and columns. So anyway I added a second layer to the loop, and output the pixel values, and when I ran it in the one mode:
Ok I see. Hmm. Well the important thing is that this code is indeed for extracting the pixel information. I think it’s going to be best for the segmentation to use the simpler segmentation mask that doesn’t track the link info. Ok so I used that code from the guy’s thesis project, and that was interpolating the numbers. When I look at the unique elements of the mask without interpolation, I’ve got…
[ 0 2 255]
[ 0 2 255]
[ 0 2 255]
[ 0 2 255]
[ 0 2 255]
[ 0 1 2 255]
[ 0 1 2 255]
[ 0 2 255]
[ 0 2 255]
Ok, so I think:
255 is the sky
0 is the plane
2 is the robotable
1 is the egg
So yeah, I was just confused because the segmentation masks were all black and white. But if you look closely with a pixel picker tool, the pixel values are (0,0,0), (1,1,1), (2,2,2), (255,255,255), so I just couldn’t see it.
The interpolation kinda helps, to be honest.
As per OpenAI’s domain randomization helping with Sim2Real, we want to randomize some textures and some other things like that. I also want to throw in some random chickens. Maybe some cats and dogs. I’m afraid of transfer learning, at this stage, because a lot of it has to do with changing the structure of the final layer of the neural network, and that might be tough. Let’s just do chickens and eggs.
Both techniques increase the computational requirements: dynamics randomization slows training down by a factor of 3x, while learning from images rather than states is about 5-10x slower.
Ok that’s a bit more complex than I was thinking. I want to randomize textures and colours, first
I’ve downloaded and unzipped the ‘Describable Textures Dataset’
And ok it’s loading a random texture for the plane
and random colour for the egg and chicken
Ok, next thing is the Simulation CNN.
Interpolation doesn’t work though, for this, cause it interpolates from what’s available in the image:
[ 0 85 170 255]
[ 0 63 127 191 255]
[ 0 63 127 191 255]
I kind of need the basic UID segmentation.
[ 0 1 2 3 255]
Ok, pity about the mask colours, but anyway.
Let’s train the UNet on the new dataset.
We’ll need to make karolzak’s changes.
I’ve saved 2000+ rgb.jpg and seg.png files and we’ve got [0,1,2,3,255] [plane, egg, robot, chicken, sky]
So num_classes=5
And
“for multiclass segmentation you would need a mask of shape (width, height, num_classes) “
What is y.shape?
(2001, 256, 256, 1)
which is 2001 files, of 256 x 256 pixels, and one class. So if I change that to 5…? ValueError: cannot reshape array of size 131137536 into shape (2001,256,256,5)
Um… Ok I need to do more research. Brb.
So the keras_unet library is set up to input binary masks per class, and output binary masks per class.
I coded it up using the library author’s suggested method, as he pointed out that the gains of the integer encoding method are minimal. I’ll check it out another time. I think it might still make sense for certain cases.
Ok that’s pretty awesome. We have 4 masks. Human, chicken, egg, robot. I left out plane and sky for now. That was just 2000 images of training, and I have 20000. I trained on another 2000 images, and it’s down to 0.008 validation loss, which is good enough!
So now I want to load the CNN model in the locomotion code, and feed it the images from the camera, and then have a reward function related to maximizing the egg pixels.
I also need to look at the pybullet-planning project and see what it consists of, as I imagine they’ve made some progress on the next steps. “built-in implementations of standard motion planners, including PRM, RRT, biRRT, A* etc.” – I haven’t even come across these acronyms yet! Ok, they are motion planning. Solvers of some sort. Hmm.
The attempted training of the U-Net on the Jetson NX has been a bit slow, making odd progress over 2 nights, and I’m not sure if it’s working. I’ve had to reduce batch size to 1, and the filter size, which has reduced the number of parameters by about a factor of 10, and still, loading the NN into memory sometimes dies on a concatenation call. The number of images per batch can also crash it, so perhaps some memory can be saved with a better image loading process.
Anyway, projects under an official NVIDIA repo are suggesting that we should be able to train smaller networks like resnet18, with 11 million parameters, on the Jetson. So maybe we can still avoid the cloud.
But judging by the NVIDIA TLT info, any training of resnet50s or 100s are going to need serious GPUs and memory and space for training.
After looking at Google, Amazon and Microsoft offerings, the AWS g4dn.xlarge instance looks like it might be the best option, at $0.526/hr, or Google’s got a T4 based compute engine for only $0.35/hr. These are good options, if 16GB of video ram will be enough. It should be, because we’re working with like 5GB on the Jetson.
Microsoft has the NC6 option, which looks good for a much more beefy GPU and memory, at $0.90/hr.
We’re just looking at Pay-as-you-go prices, as the 1-year and 3-year commitments will end up being expensive.
I’m still keen to try train on the Jetson, but the cloud is becoming more and more probable. In Sweden, visiting Miranda, we’re unable to order a Jetson AGX Xavier, the 32GB version. Arrow won’t ship here without a VAT number, and SiliconHighway is out of stock.
So, attempting Cloud GPUs. If you want to cut to the chase, read this one backwards. So many problems. In the end, it turned out setting it up yourself is practically impossible, but there is an ‘AI Platform’ section that works.
Amazon AWS. Tried to log in to AWS. “Authentication failed because your account has been suspended.” Tells me to create a new account. But then brings me back to the same failure screen. Ok, sending email to their accounts department. Next.
Google Cloud. I tried to create a VM and add a T4 GPU, but none of the regions have them. So I need to download the Gcloud SDK and CLI tool first, to run a command to describe the regions, according to the ‘Before you begin‘ instructions..
Ok, GPUs will only run on N1 and A2 VMs. The A2 VMs are only for A100s, so I need an N1 VM in one of these regions, and we add a T4 GPU.
There’s an option to load a specific docker, and unfortunately they don’t seem to have one with both Pytorch and TF2. Let’s start with TF2 gcr.io/deeplearning-platform-release/tf2-gpu.2-4
So this looks like a good enough VM. 30GB RAM, 8 cpus. For europe-west3, the cost is about 50 cents / hr for the VM and 41 cents / hr for the GPU.
n1-standard-8
8
30GB
$0.4896
$0.09840
1 GPU
16 GB GDDR6
$0.41 per GPU
So let’s round up to about $1/hour. I ended up picking the n1-standard-4 (4 cpus, 15 gb ram).
At these prices I’ll want to get things up and running asap. So I am going to prep a bit, before I click the Create VM button.
I had to try a few things to find a cloud instance with a gpu, because the official list didn’t really work. I eventually got one with a T4 GPU from europe-west4-c.
It seems like Google Drive isn’t really part of the google cloud platform ecosystem, so I started a storage bucket with 50GB of space, and am uploading the chicken images to it.
The instance doesn’t have pip or jupyter installed. So let’s do that…
ok so when I sudo’ed, I got this error
Jul 20 14:45:01 chicken-vm konlet-startup[1665]: {"errorDetail":{"message":"write /var/lib/docker/tmp/GetImageBlob362062711: no space left on device"},"error":"write /var/lib/docker/tmp/GetImageBl
Jul 20 14:45:01 chicken-vm konlet-startup[1665]: ).
Jul 20 14:45:01 chicken-vm konlet-startup[1665]: 2021/07/20 14:43:04 No containers created by previous runs of Konlet found.
Jul 20 14:45:01 chicken-vm konlet-startup[1665]: 2021/07/20 14:43:04 Found 0 volume mounts in container chicken-vm declaration.
Jul 20 14:45:01 chicken-vm konlet-startup[1665]: 2021/07/20 14:43:04 Error: Failed to start container: Error: No such image: gcr.io/deeplearning-platform-release/tf2-gpu.2-4
Jul 20 14:45:01 chicken-vm konlet-startup[1665]: 2021/07/20 14:43:04 Saving welcome script to profile.d
So 10GB wasn’t enough to load gcr.io/deeplearning-platform-release/tf2-gpu.2-4 , I guess.
Ok deleting the VM. Next time, bigger hard drive. I’m now adding a cloud storage bucket and uploading the chicken images, so I can copy them to the VM’s drive later. It’s taking forever. Wow. Ok.
Now I am trying to spin up a VM again, and it’s practically impossible. I’ve tried every region and zone possible. Ok europe-west1-c. Finally. I also upped my ‘quota’ of gpus, under IAM->Quotas, in case that is a reason I couldn’t find a GPU VM. They reviewed and approved it in about 15 minutes.
+------------------+--------+-----------------+ | Name | Region | Requested Limit | +------------------+--------+-----------------+ | GPUS_ALL_REGIONS | GLOBAL | 1 | +------------------+--------+-----------------+
So after like 10 minutes of nothing, I see the docker container started up.
68ee22bf268f gcr.io/deeplearning-platform-release/tf2-gpu.2-4 "/entrypoint.sh /run…" 5 minutes ago Up 4 minutes klt-chicken-vm-template-1-ursn
I’ve enabled tcp:8080 port in the firewall settings, but the external ip and new port don’t seem to connect. https://35.195.66.139:8080/ Ah ha. http. We’re in!
Jupyter Lab starting up.
So I tried to download the gcloud tools to get gsutil to access my storage bucket, but was getting ‘Permission denied’, even as root. I chown’ed it to my user, but still no.
I had to go out, so I stopped the VM. Seems you can’t suspend a VM with a GPU. I also saw when I typed ‘sudo -i’ to switch user to root, it said to ‘docker attach’ to my container. But the container is just like a tty printing out logs, so you can get stuck in the docker, and need to ssh in again.
I think the issue was just that I need to be inside the docker to do things. The VM you log into is just a minimal container running environment. So I think that was my issue. Next time I install gsutil, I’ll run ‘docker exec -it 68ee22bf268f bash’ to get into the docker first.
Ok fired up the VM again. This time I exec’ed into the docker, and gsutil was already installed. gsutil cp -r gs://chicken-drive . is copying the files now. It’s slow, and it says to try with -m, for parallel copying, but I’m just going to let it carry on for now. It’s slow, but I can do some other stuff for now. So far our gcloud bill is $1.80.
Ok, /opt/jupyter/chicken-drive has my data now. But according to /opt/jupyter/.jupyter/jupyter_notebook_config.py, I need to move it under /home/jupyter.
Hmm. No space left on drive. What? 26GB all full. But it wasn’t full a second ago. How can moving files cause this? I guess the mv operation must copy and then delete. Ok, so deleting the new one. Let’s try again, one folder at a time. Oh boy. This is something a bit off about the google process. I didn’t start my container, and if I did, I’d probably map a volume. But the host is sort of read only. Anyway. We’re in. I can see the files in Jupyter Lab.
So now we’re training U-Net binary classification using keras-unet, by karolzak, based on the kz-isbi-chanllenge.ipynb notebook.
But now I’m getting this error when it’s clearly there…
FileNotFoundError: [Errno 2] No such file or directory: '/OID/v6/images/Chicken/train/'
Ok well I can’t work it out but changing it to a path relative to the notebook worked. base_dir = “../../../”
Ok first test round of training, binary classification: chicken, not-chicken. Just 173 image/mask pairs, 10 epochs of 40 steps.
Now let’s try with the training set. 1989 chickens this time. 50/50 split. 30 epochs of 50 steps. Ok second round… hmm, not so good. Pretty much all black.
Ok I’m changing the parameters of the network, fixing some code, and starting again.
I see that the pngs were loading float values, whereas in the example, they were loading ints. I fixed it by adding a m = m.convert(‘L’) to the mask (png) loading code. I think previously, it was training with the float values from 0 to 1, divided by 255, whereas the original example had int values from 0 to 255, divided by 255.
So I’m also resetting the parameters, to make this a larger network, since we’re training in the cloud. 512×512 instead of 256×256. Batch size of 3. Horizontal flip augmentation. 64 filters. 10 epochs of 100 steps. Go go go. Ok, out of memory. Batch size of 1. Still out of memory. Back to test set of 173 chickens. Ok it’s only maxing at 40% RAM now. I’ll let it run.
Ok, honestly I don’t know anymore. What is it even doing? Looks like it’s inversing black and white. That’s not very useful.
Ok before giving up, I’m going to make some changes.
The next day, I’m starting up the VM. Total cost so far, $8.84. The files are all missing, so I’m recopying, though using the gsutil -m cp -R gs://chicken-drive . option, and yes it is a lot faster. Though it slows down.
I think the current setup is maybe failing because we’re using 173 images with one kind of augmentation. Instead of 10 epochs of 100 steps of the same shit, let’s rather swap out the training images.
First problem is that Keras is basically broken, in this regard. I’ve immediately discovered that saving and loading a checkpoint does not save and load the metrics, and so it keeps evaluating against a loss of infinity, instead of what your saved model achieved. Very annoying.
Now, after stopping and restarting the VM, and enabling all cloud APIs, I’m having a new problem. gsutil no longer works. After 4% copied, network throughput drops to 0.0B/s. I tried reconnecting and now get:
Connection via Cloud Identity-Aware Proxy Failed
Code: 4003
Reason: failed to connect to backend
You may be able to connect without using the Cloud Identity-Aware Proxy.
I’ve switched back to ‘Allow default access’. Still getting 4003.
Ok, I’ve deleted the instance. Trying again. Started it up. It’s not installing the docker I asked for, after 22 minutes. Something is wrong. Let’s try again. Stopping VM. I’m ticking the ‘Run as priviliged’ box this time.
Ok now it’s working again. It even started up with the docker ready. I’m trying with the multiprocess copying again, and it slowed down at 55%, but is still going. Phew. Ok.
I changed to using the TF2 SavedModel format. Still restarts the ‘best’ metric. What a piece of shit. I can’t actually believe it. Ok I wrote my own code for finding the best, by saving all weights with the val_loss in the filename, and then loading the best weights for the next epoch. It’s still not perfect, but it’s better than Keras overwriting the best weights every time.
Interestingly, it seems like maybe my training on the Jetson was actually working, because the same weird little vignette-ing is occurring.
Ok we’re up to $20 billing, on gcloud. It’s adding up, but not too badly yet. Nothing seems to be beating a round of training from like 4 hours ago, so to keep things more exploratory, I added a 50/50 chance to pick from the saved weights at random, rather than loading the winner every time.
Something seems to be happening. The vignette is shrinking, but some chicken border action, maybe.
I left it running overnight, and this morning, we’re up to $33 spent, and today, we can’t log into the VM again. Pretty annoying. Of the 3 reasons for ‘Permission denied’, only one makes sense, Your key expired and Compute Engine deleted your ~/.ssh/authorized_keys file.
Same story if I run the gcloud commands: gcloud beta compute ssh –zone “europe-west4-c” “chicken-vm-template-1” –project “gpu-ggr”
So I apparently need to add a new public key to the Metadata section. I just know something is going to go wrong. Yeah, so I did everything I know I’m supposed to do, and it didn’t work. I generated an OpenSSH private/public key pair in PuttyGen, I changed the permissions on the private key so that only I have access, I updated the SSH Keys in the VM instance metadata, and the metadata for good measure. And ssh -i opensshprivate daniel_brownell@34.91.21.245 -v just ends up with Permission denied (publickey).
Ok and then print the public key, and copy paste it to the VM Instance ‘Edit…’ / SSH Keys… and connect with PuTTY with the private key and… nope. Permission denied (publickey).. Ok I need to go through these answers and find one that works. Same error with windows cmd line ssh, except also complains that the openssh key is an invalid format. Try again later.
Fuck you gcloud. Ok I’m stopping and deleting the VM. $43 used so far.
Also, the training through the night didn’t improve on the val_loss score. Something’s fucked.
Ok I’ve started it up again a few days later. I was wondering about the warnings at the beginning of my training that carious CUDA things were not installed. So apparently I need:
So I increased the boot disk to 35GB and called ‘ cos-extensions install gpu’ again, after cd’ing into /mnt/stateful-partition and it worked a bit better. Still has ‘ERROR: Unable to load the kernel module 'nvidia.ko'.‘ in the logs though. But install logs at ./mnt/stateful_partition/var/lib/nvidia/nvidia-installer.log say its ok…
So the error now is ‘Could not load dynamic library ‘libcuda.so.1′; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64’
And so we need to modify the docker container run command, something like the example in the instructions.
Ok so our container is… gcr.io/deeplearning-platform-release/tf2-gpu.2-4
According to this stackoverflow answer, this already has everything installed. Ok but the host needs the drivers installed.
tf.config.list_physical_devices('GPU')
[]
So yeah, i think i need to install the cos crap, and restart the container with those volume and device bits.
docker stop klt-chicken-vm-template-1-ursn
docker run \
--volume /var/lib/nvidia/lib64:/usr/local/nvidia/lib64 \
--volume /var/lib/nvidia/bin:/usr/local/nvidia/bin \
--device /dev/nvidia0:/dev/nvidia0 \
--device /dev/nvidia-uvm:/dev/nvidia-uvm \
--device /dev/nvidiactl:/dev/nvidiactl \
gcr.io/deeplearning-platform-release/tf2-gpu.2-4
...
[I 14:54:49.167 LabApp] Jupyter Notebook 6.3.0 is running at:
[I 14:54:49.168 LabApp] http://46fce08b5770:8080/
[I 14:54:49.168 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
^C^C^C^C^C^C^C^C^C^C
Not so good. Ok can’t access it either. -p 8080:8080 fixes that. It didn’t like --gpus all.
“Unable to determine GPU information”. Container optimised shit.
Ok I’m going to delete the VM again. Going to check out these nvidia cloud containers. There’s 21.07-tf2-py3 and NGC stuff.
So I can’t pull the dockers cause there’s no space, and even after attaching a persistent disk, not, because things are stored on the boot disk. Ok but I can tell docker to store stuff on a persistent disk.
/etc/docker/daemon.json:
{
"data-root": "/mnt/x/y/docker_data"
}
root@nvidia-ngc-tensorflow-test-b-1-vm:/mnt/disks/disk# docker run --gpus all --rm -it -p 8080:8080 -p 6006:6006 nvcr.io/nvidia/tensorflow:21.07-tf2-py3
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.
wget https://developer.download.nvidia.com/compute/cuda/11.1.0/local_installers/cuda_11.1.0_455.23.05_linux.run
chmod +x cuda_11.1.0_455.23.05_linux.run
sudo ./cuda_11.1.0_455.23.05_linux.run
or some newer version:
wget https://developer.download.nvidia.com/compute/cuda/11.4.1/local_installers/cuda_11.4.1_470.57.02_linux.run
sudo sh cuda_11.4.1_470.57.02_linux.run
‘boost::filesystem::filesystem_error’
Ok using all the space again. 32GB. Not enough. Fuck this. I’m deleting the VM again. 64GB. SSD persistent disk. Ok installed driver. Running docker…
And…
FFS. Something is compromised. In the time it took to install CUDA and run docker on an Ubuntu VM, an army of Indian hackers managed to delete my root user.
Ok. Maybe it’s time to consider AWS again for GPUs. I think I can officially count GCP GPU as unusable. Learned a few useful things, but overall, yeesh.
I think maybe I’ll just run the training on a cheap non-GPU VM on GCP for now, so that I’m not paying for a GPU that I’m not using.
docker run -d -p 8080:8080 -v /home/daniel_brownell:/home/jupyter gcr.io/deeplearning-platform-release/tf2-cpu.2-4
Ok wow so now with the cpu version, the loss is improving like crazy. It went from 0.28 to 0.24 in 10 epochs (10 minutes or so). That sort of improvement was not happening after like 10 hours on the ‘gpu’.
So yeah, amazing. The code now does a sort of population based training, by picking a random previous set of weights instead of the best weights, half of the time. Overall it slows things down, but should result in a bit more variation in the end.
What finally worked
Ok there’s also an ‘AI platform – notebook’ option. I might try that too.
Ok the instance started up. But it failed to start 4 cron services: nscd, unscd, crond, sshd. CPU use goes to zero. Nothing. Ok so I need to ssh tunnel apparently.
Successfully opened dynamic library libcudart.so.11.0
‘ModelCheckpoint’ object has no attribute ‘_implements_train_batch_hooks’
Ok, needed to change all keras.* etc. to tensorflow.keras.*
Ok fuck me that’s a lot faster than CPU.
Permission denied: ‘weights-0.2439.hdf5’
Ok, let’s sudo it.
Ok there she goes. It’s like 20 times faster maybe. Strangely isn’t doing much better than the CPU though. But I’ll let it run for a bit. It’s only been a minute. I think maybe the CPU doing well was just good luck. Perhaps we trained them too well on the original set of like 173 images, and it was getting good results on those original images.
Ok now it’s been an hour or so, and it’s not beating the CPU. I’ve changed the train / validation set to 50/50 now, and the learning rate is randomly chosen between 0.001 and 0.0003. And I’m upping the epochs to 30. And the filters to 64. batch_size=4, use_batch_norm=True.
We’re down to 23.3 after an hour and a half. 21 now… 3 hours maybe now
Ok 5 hours, lets check:
Holy shit it’s working. That’s great. I’ll leave it running overnight. The overnight results didn’t improve much for some reason.
(TODO: learn about focal loss / dice loss / jaccard distance as possible change to loss function.? less necessary now.)
So it’s cool but it’s 364MB. We need it 1/4 size to run it on the Jetson NX I think.
So, retraining, with filters=32. We’re already down to 0.24 after an hour. Ok I stopped at 0.2104 after a few hours.
So yeah. Good enough for now.
There’s some other things to train, too.
The eggs in simulation: generate views, save images to disk. save segmentation images to disk.
Train the walking again with the gripper.
Eggs in the real world. Use augmentation to place real egg pics in scenes. Possibly use Mask-RCNN/YOLACT code with COCO, instead of continuing in Keras.
The now-working U-net binary chicken segmentation is in Keras, so there will be some tricks required, to run a multi-class segmentation detector, or multiple binary classifiers. Advice for multi-class segmentation is here and the multiple binary classifier advice is here.
When we finally try running it all on a Jetson, we will maybe need to shrink the neural network further. But that can be done last minute. It looks like we can save the h5fs file to TF2’s SavedModel format with model.save(model_fname) and convert to frozen graph, to import into TensorRT, the NVIDIA format. Similar to this. TensorRT shrinks neurons to single bytes, I believe.
Turns out Dan’s grandfather, the chicken sexer, had to determine the sex of day old chicks by feeling something in their cloaca. He had to grow his fingernail in some special way to do so. OG robotic prosthetics.
” For color sexing, we make use of the principle of sex-link crosses. Sex-link crosses (hybrid chicken breeds) involve the crossing of two different breeds of chickens to get offspring that can be sexed at hatch by differences in down color. For example, mating Rhode Island Red males with Rhode Island White females. Their offspring will consist of yellow/white colored day-old male chicks, and brown colored day-old female chicks (as always, exceptions are there). Based on the color of the down, the day-old chicks can easily be sorted by sex.
The most common sex link crosses are the red sex link. Close to half of the global chicken population consists of red sex link chickens. Our common brown breeds (ISA Brown, Shaver Brown, Bovans Brown, Babcock Brown, Hisex Brown, Dekalb Brown and Warren) are all red sex link crossbreds.
Please note: this trait does not breed true and breeding two sex linked commercial chickens (hybrids) will not produce offspring that can be sexed at hatch.
Color sexing female day-old chicks
Female day-old chicks have uniformly brown colored down, or brown with a light stripe in the middle of the back.
The following common exceptions can be found: female day-old chicks that have a broad yellow stripe with brown edging on a somewhat lighter background, or female day-old chicks that have a brown colored head with a lighter body.
Color sexing male day-old chicks
Male day-old chicks are yellow/white or show slight brown stripes.
Exceptions can be found: day-old male chicks that have distinct yellow/light stripes with brown borders, or day-old male chicks that have a dark brown stripe in the middle of their back.
Feather-sexing can only be carried out in chicks resulting from the mating of fast normal-feathering males with delayed-feathering females, while vent sexing can be applied to any chicks. This means that feather-sexing cannot be applied to chicks of the same breed, unless geneticists preserve that breed segregating for both types of feathering and birds are strictly controlled for feathering type.
The preparation of new breeds to produce feather sexable crosses implies considerable previous genetic work which is completely beyond the means of most poultry farmers.”
Figure 1. Feathering types Diagram of the wing of a one-day-old chick in dorsal view: A fast normal-feathering with the primary remiges (1a) longer than the coverlets (2a). B sex-linked delayed feathering with the primary remiges (1a) of the same length than the coverlets (2a). C modified delayed feathering with the primary remiges. (1a) shorter than the coverlets (2a).
In-ovo sexing of 14-day-old chicken embryos by pattern analysis in hyperspectral images (VIS/NIR spectra): A non-destructive method for layer lines with gender-specific down feather color
Up to now there is no economically maintainable modality for chicken sexing in early embryonic stages (first 3 d) that is suitable for large-scale application in the commercial hatcheries. Hence, the culling of male day-old chicks of layer lines is still the normal procedure.In this paper we present a non-destructive optical technique for gender determination in layer lines with gender-specific down feather color. This particular chicken strain presents a sexual dimorphism in feather color, where the female day-old chicks have brown down feathers and the males have yellow down feathers.The eggs are candled with halogen lamps and a hyperspectral camera collects the transmitted light within the spectral range from 400 nm to 1,000 nm. For data analysis and classification, common methods like principal component analysis and linear discriminant analysis are used. The accuracy of gender determination was determined for 11- to 14-day-old embryos. At 14 d of incubation (7 d before hatch) the sex can be determined with an overall accuracy of approximately 97%.
Keywords: hyperspectral imaging; in ovo sexing; sexing of chicken embryo.
Carrefour’s cage-free progress // Ending the culling of male chicks
Published 10/02/2020
For the first time in France, an in ovo sexing technique is being trialled by French retailer Carrefour, in partnership with its supplier Les Fermiers de Loué and the AAT group, a global specialist in hatching.
The fast and non-invasive technology for sexing the egg by spectrophotometry (i.e. colour analysis) makes it possible to identify the sex of the birds before they hatch, thus avoiding the need to cull male chicks after their birth in the egg production cycle.
Sustainable use of the male embryos as high-quality protein source: Schaffelaarbos partners with Agri Advanced Technologies GMBH (AAT)
The German company AAT offers a non-invasive in ovo sexing method with respectful treatment of sorted out male embryos. Schaffelaarbos will further process these male embryos into Dried Egg Proteins for the Animal Feed Market.
Sex determination in the egg is ready for high-volume application
“CHEGGY” is the name of the machine that marks a breakthrough in hyperspectral measurement technology. Developed by the German manufacturer AAT, this innovative technology is now ready for high-volume, practical application for brown-laying hens in France, Germany, Belgium, Spain and other European countries.
Hyperspectral measurement technology for sex determination in the egg is ready for practical application
Visbek, 3 August 2020. AAT (part of the EW GROUP), the specialist for the development of special application technologies in animal breeding, announces that its solution for automatic sex determination in the egg is ready for use.
Carrefour – the first retailer to introduce a method for avoiding male chicks to be killed
Carrefour press release: As part of its food transition plan, Carrefour and Loué have entered into a partnership to produce and sell eggs laid by chickens produced using a spectrophotometric in-ovo sexing technique. Consumers will be able to purchase the products from 1 May.
Published 10/02/2020