This https://github.com/facebookresearch/meshrcnn is maybe getting closer to holy grail in my mind. I like the idea of bridging the gap between simulation and reality in the other direction too. By converting the world into object meshes. Real2Sim.
The OpenAI Rubik’s cube hand policy transfer was done with camera in simulation and camera in real world. This could allow a sort of dreaming, i.e., running simulations on new 3d obj data.)
It could acquire data that it could mull over, when chickens are asleep.
PyTorch3d: https://arxiv.org/pdf/2007.08501.pdf
Pixel2Mesh: Generating 3D Mesh Models
from Single RGB Images https://arxiv.org/pdf/1804.01654.pdf
Remember Hinton’s dark knowledge. The trick is having a few models distill into one.
In trying to get Mesh R-CNN working, I had to add DEVICE=CPU to the config.
python3 demo/demo.py --config-file configs/pix3d/meshrcnn_R50_FPN.yaml --input /home/chrx/Downloads/chickenegg.jpg --output output_demo --onlyhighest MODEL.WEIGHTS meshrcnn://meshrcnn_R50.pth
Success! It’s a chair.
There’s no chicken category in Pix3d. But getting closer. Just need a chicken and egg dataset.
Downloading blender again, to check out the obj file that was generated. Ok Blender doesn’t want to show it, but here’s a handy site https://3dviewer.net/ to view OBJ files. The issue in blender required selecting the obj, then View > Frame Selected to make it zoom in. Switching to orthographic from perspective view also helps.
Chair is a pretty adaptable class.
