AI Camera and AI Accelerator Hailo trial on Industrial Raspberry Pi

The official Raspberry Pi AI camera module can be connected in the same way as previous camera modules. The CSI camera port is provided on the main unit and the basics are the same.
The AI camera module differs from the previous ones in that it has a small AI accelerator (RP2040) on the camera side.
The RP2040 is the same MCU as the Raspberry Pi Pico.

The RP2040 enables the camera to perform some of the processing required for AI, so the processing power of the Raspberry Pi itself can be used to achieve this.

The industrial Raspberry Pi was tested with Compute Module 5 (CM5 ). The conventional PL-R4 is equipped with CM4, but it can be replaced with CM5 of the same form factor. In this case, we used the PL-R5-M model with CM5. In addition, we connected Hailo, one of the AI accelerators.
The basic operation was experienced with reference to the official documentation.

AI camera modules and Hailo are commercially available and anyone can try them out.

Differences between conventional camera modules and AI camera modules

There is a clear diagram in the official documentation. The left side represents the previous camera module. The optional AI accelerator can be input/output.
On the right is the AI camera module, showing the AI accelerator inside the IMX500.
The chip (MCU) in the AI camera module is the RP2040, familiar from the Raspberry Pi Pico.
The chip is used as a supplemental chip, so it can work in an environment without the optional AI accelerator.

Environment of this time

This time, not only the AI camera module, but also Hailo (AI accelerator) is already connected to the M.2 slot of the main unit as an option.
The unit is set up with PL-R5-M with ComputeModule 5.

• PL-R5-M USB IP20 with Compute Module 5 (Industrial Raspi)
• Raspberry Pi OS bookworm
• AI Camera Module (IMX500)
• Hailo (AI Accelerator)

Hailo setup is omitted here. Please refer to the official documentation.

The AI camera module has a dedicated package; I installed it with apt after updating the OS.

sudo apt update && sudo apt full-upgrade
sudo apt install imx500-all

As previously reported, the parameters in config.txt must be changed for the camera module to work. (Reference: Easy to connect cameras even on an industrial Raspberry Pi. )

For Pi 5 (bookworm), config.txt is under firmware.

sudo nano /boot/firmware/config.txt

The changes in config.txt explicitly turn off automatic camera recognition and specify the AI camera module, IMX500.

camera_auto_detect=0

[all]

dtoverlay=imx500,cam0

*The PL-R5-M used in the trial is a proprietary device with Compute Module 5, so there are other parts to be added or changed.

After rebooting, use the command to see if the AI camera was automatically recognized.
imx500 was found. You are now ready to go.

libcamera-hello --list-cameras

Available cameras
-----------------
0 : imx500 [4056x3040 10-bit RGGB] (/base/axi/pcie@120000/rp1/i2c@88000/imx500@1a)
    Modes: 'SRGGB10_CSI2P' : 2028x1520 [30.02 fps - (0, 0)/4056x3040 crop]
                             4056x3040 [10.00 fps - (0, 0)/4056x3040 crop]

Try object detection

Now that the AI camera has been recognized, we ran the sample.
/usr/share/rpi-camera-assetsThe files that can be used with rpicam-app are installed at . They are in json format.
Also, the AI Model is installed at /usr/share/imx500-models.
I see that this can be specified by Python command. The extension is rpk.

As documented, I was able to draw a box-like border on the object by running imx500_mobilenet_ssd.jsonwith the rpicam-hello command.

rpicam-hello -t 0s --post-process-file /usr/share/rpi-camera-assets/imx500_mobilenet_ssd.json

Since we are setting up Hailo this time, we also ran the json file labeled Hailo.
The result will show the same border.

rpicam-hello -t 0s --post-process-file /usr/share/rpi-camera-assets/Hailo_yolov6_inference.json

Comparing the two, the percentages of detection differed.
Previously it was 62%, but in Hailo_yolov6_inference.json it is 91%.

After execution, a progress bar (Network Firmware Update) appears on the terminal screen, which was fast (a few seconds) with Hailo installed.

By the way, the command option --viewfinder-width 1920 --viewfinder-height 1080 --framerate 30allows you to specify image aspect, frame rate, etc.
The “YOLO” in Hailo_yolov6~ is the algorithm for object detection. You have version 6, version 8, etc.

If you look inside the imx500_mobilenet_ssd.jsonfile I mentioned earlier, you will see that I am using the AI model found at /usr/share/imx500-models. (rpk file)

{
    "imx500_object_detection":
    {
        "max_detections" : 5,
        "threshold" : 0.6,
        "network_file": "/usr/share/imx500-models/imx500_network_ssd_mobilenetv2_fpnlite_320x320_pp.rpk",
...（以下、略）

Other asset files

I have tried other json files installed.
You can specify each parameter in the json file.

Here are some results of running it as is.

./usr/share/rpi-camera-assets
├── annotate_cv.json
├── face_detect_cv.json
├── Hailo_classifier.json
├── Hailo_pose_inf_fl.json
├── Hailo_scrfd.json
├── Hailo_yolov5_personface.json
├── Hailo_yolov5_segmentation.json
├── Hailo_yolov6_inference.json
├── Hailo_yolov8_inference.json
├── Hailo_yolov8_pose.json
├── Hailo_yolox_inference.json
├── hdr.json
├── imx500_mobilenet_ssd.json
├── imx500_posenet.json
├── motion_detect.json
├── negate.json
└── sobel_cv.json

annotate_cv.json
The number of frames and other processing details (annotations) were displayed.

Hailo_classifier.json
You can see the words change in the upper left corner as you change the camera orientation.

Hailo_scrfd.json
Another window was launched and I was able to zoom in, copy to the clipboard, etc.

negate.json
Negative display.

sobel_cv.json
Sobel filter to enhance edges.

Up to this point, the rpicam command has been used.
Next, we will use Picamera2 to run a demo of the Python file.

Installation of Picamera2

To control the camera module in the Python code, we need to import Picamera2 in the code.

Picamera2 comes pre-installed with the current OS; it must be installed separately with an OS image file prior to Raspberry Pi OS Bullseye or with an OS lite image.
(Reference: https://github.com/raspberrypi/picamera2 )

Most examples use OpenCV. Install the dependencies.

sudo apt install python3-opencv python3-munkres

I git cloned Picamera2 and found a directory for the AI camera module (IMX500) in the examples with the demo.

git clone https://github.com/raspberrypi/picamera2.git
cd picamera2/examples/imx500

Five demo files written in Python.

./picamera2/examples/imx500
├── imx500_classification_demo.py
├── imx500_object_detection_demo_mp.py
├── imx500_object_detection_demo.py
├── imx500_pose_estimation_higherhrnet_demo.py
└── imx500_segmentation_demo.py

There were also three in the Hailo directory.

./picamera2/examples/Hailo
├── coco.txt
├── detect.py
├── pose.py
└── pose_utils.py

Try out the Python demo program

As per the official documentation, run imx500_object_detection_demo.pywith the AI model imx500_network_ssd_mobilenetv2_fpnlite_320x320_pp.rpk.

python imx500_object_detection_demo.py --model /usr/share/imx500-models/imx500_network_ssd_mobilenetv2_fpnlite_320x320_pp.rpk

You got it right with the teddy bear.

This plush toy is the official Raspberry Pi mascot. It is called Babbage Bear but it was not identified with that name without learning it. Teddy bear is pre-dataset as a stuffed animal, and since it is actually a teddy bear, there is no problem.

ssd_mobilenetv2″ is a model to detect objects in environments where resources are scarce, such as mobile devices, as the word “mobile” indicates.
It has been released up to MobileNET v3.

Next, we also tried a pose (skeletal) detection demo.
This one only needed to run the demo as is.

python imx500_pose_estimation_higherhrnet_demo.py

Sorry I have a problem with the distance to the camera module and just the arm.
Without doing anything, the three points? I did not have to do anything, but 3 points were recognized and connected as a skeleton. I was told that the number of skeletons can be adjusted by specifying the maximum number of skeletons and so on.
I could not get it to work if the distance was too close, but if the distance was 60 cm, the skeletal lines would automatically appear.

There were a total of 24 files for rpk, the AI Model.
If you know about deep learning, AI, and the Python language, why not write a Python program specifying each AI Model?

./usr/share/imx500-models
├── imx500_network_deeplabv3plus.rpk
├── imx500_network_efficientdet_lite0_pp.rpk
├── imx500_network_efficientnet_bo.rpk
├── imx500_network_efficientnet_lite0.rpk
├── imx500_network_efficientnetv2_b0.rpk
├── imx500_network_efficientnetv2_b1.rpk
├── imx500_network_efficientnetv2_b2.rpk
├── imx500_network_higherhrnet_coco.rpk
├── imx500_network_inputtensoronly.rpk
├── imx500_network_levit_128s.rpk
├── imx500_network_mnasnet1.0.rpk
├── imx500_network_mobilenet_v2.rpk
├── imx500_network_mobilevit_xs.rpk
├── imx500_network_mobilevit_xxs.rpk
├── imx500_network_nanodet_plus_416x416_pp.rpk
├── imx500_network_nanodet_plus_416x416.rpk
├── imx500_network_posenet.rpk
├── imx500_network_regnetx_002.rpk
├── imx500_network_regnety_002.rpk
├── imx500_network_regnety_004.rpk
├── imx500_network_resnet18.rpk
├── imx500_network_shufflenet_v2_x1_5.rpk
├── imx500_network_squeezenet1.0.rpk
└── imx500_network_ssd_mobilenetv2_fpnlite_320x320_pp.rpk

Try the AI camera.

Even though it is a sample operation, if you execute the command and hold something in front of the camera, it appears to identify it without any stress.
The process was very smooth because we used both the AI camera module and the AI accelerator.
With only the AI camera module, after executing the command Network Firmware Update takes a bit of time after the command is executed. Once it works, the rest is the same.

There are many words and mechanisms used in AI, such as YOLO, TensorFlow, and Hailo, that seem complicated without previous knowledge. Since this is a camera, it is also a good idea to have knowledge of images and video.
On the other hand, if you are familiar with them, you may be surprised at what you can achieve with a Raspberry Pi.

Although the published documentation is quite detailed, the sample examples will give you a better understanding of what can be done with the AI camera module.

Reference: AI Camera Module Official Document