Image file compression with PiShrink
How to backup using RPI BOOT and dd command

Introduction.

Industrial Raspberry Pi with Raspberry Pi Compute Module (hereafter referred to as CM) can boot from SSD drive or microSD card as we have reported so far. The basic idea is eMMC booting.
Unlike the Raspberry Pi 4 (“Pi 4”) and 5, the CM4 cannot write the boot loader on the eMMC as it is either. We previously reported that the PL-R4 can be replaced by setting the DIP switches to write-enabled and then transferring it from another device via USB.
The OS (system area) in the eMMC works the same way. The image file is written via USB cable. It may seem somewhat troublesome, but it is a system that cannot be easily replaced.
For industrial raspi, the procedure is a bit more complicated, but it is convenient to backup with the dd command and compress with PiShrink to shorten the writing time.
For general Pi 4 and Pi 5, when backing up the whole system as an image file, use the dd command in the same way as this time. Please refer to the following.

Environment of this time

Since the destination for transferring image files is CM4, we used Pi 4 as the working device. The working device can also be a Windows machine.

• Working device: Raspberry Pi 4 (bookworm) 64bit (Pi 4)
• Device to be backed up is PL-R4 (CM4)
• Use RPI BOOT (rpiboot_setup.exe for Windows machines)
• Use USB-A to C cable
• dd command (or bmap method with wic extension is also supported)
• Compression with Pi Shrink

The Raspberry Pi OS has been updated and upgraded. (Latest as of the writing of this article)

It is common to use the Win32 Disk Manager for Windows or the dd command for Linux environments to back up image files.

• Win32 Disk Manager (Windows)
• dd command (Linux)

Furthermore, since this is a backup of CM4, RPI BOOT needs to be executed before executing the dd command.

Installing RPI BOOT

Install RPI BOOT on a working Pi4. Created in the home directory.
(If you are on a Windows machine, install with rpiboot_setup.exe: https://github.com/raspberrypi/usbboot/tree/master/win32 )

sudo apt install git libusb-1.0-0-dev pkg-config build-essential
git clone --recurse-submodules --shallow-submodules --depth=1
https://github.com/raspberrypi/usbboot
cd usbboot
make
cd recovary

After running RPI BOOT on the Pi 4 (or Windows machine), connect the working device and the PL-R4 (CM4) with a USB cable and use the dd command to make a backup.

Execution of RPI BOOT

After executing RPI BOOT, switch the PL-R4 DIP switch to the write-enabled side and connect it to the working Pi4 with a USB cable.
The same procedure is used for RPI BOOT for Windows.

RPI BOOT is executed as follows

cd ~/usbboot/recovary
sudo ../rpiboot -d .

In this case, the image files are backed up in the recovary directory.
If there are no errors after connecting the USB cable, a dialog box recognizing it as storage will appear on the desktop of the Pi 4 that has been made the working device.
The dd command can now be executed.

The PL-R4 and the working device you want to back up should be connected via USB cable after running RPI BOOT. one note:
CM4’s RPI BOOT works with USB 2.0.

Therefore, connect it to the USB 2.0 port on the Pi 4 side of the working device.
Only CM4 was connected to, and USB 3.0 ports were stuck with I/O errors.

The official documentation also states.

• The CM5 EEPROM supports MMC, USB-MSD, USB 2.0 (CM4 only), Network and NVMe boot by default. Try booting to Linux from an alternate boot mode (e.g. network) to verify the nRPIBOOT GPIO can be pulled low and that the USB 2.0 interface is working.
  https://github.com/raspberrypi/usbboot?tab =readme-ov-file#troubleshooting

Running the dd command

It is necessary to specify the area that is the backup source to execute the dd command.
The fdisk command can be used to check.sudo fdisk -l

This time, you can see that the CM4 connected via USB cable is /dev/sdb.

sudo fdisk -l /dev/sdb
Disk /dev/sdb: 29.12 GiB, 31268536320 bytes, 61071360 sectors
Disk model: 0001
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x4ef66861
Device Boot Start End Sectors Size Id Type
/dev/sdb1 8192 1056767 1048576 512M c W95 FAT32 (LBA)
/dev/sdb2 1056768 61071359 60014592 28.6G 83 Linux

Now that you know that it is /dev/sdb, you can back it up using the dd command.

Block size specification

The dd command specifies the file and the block size (bs).
There are many options. You can omit count if you specify bs and conv=sync at least.
The number of bytes specified in bs is the block size per read/write. 512 bytes is the default value.
It should be a multiple of the physical block size of the media being used. (512 is 1024 (1M)…4096 (4M), etc.)
Although a larger block size is simply faster, it is safer to use a smaller value because the entire size will be destroyed if there is an error. Even if the block size is increased, completion may be slower as a result.
Add status=progressas an option to track progress. Without it, it is impossible to know when it will be finished.

Example format of the dd command (image filename)

sudo dd if=/dev/sdb of=backup-bookworm-4m.img bs=4M status=progress

The if stands for InFile and the of stands for OutFile. So if = read destination, of is write destination.
In the official Raspberry Pi documentation, bs=4M when writing, so I set the backup to 4M as well.
Official documentation: https://www.raspberrypi.com/documentation/computers/compute-module.html#flash-the-emmc

The result was 31 GB with a transfer rate of 9.9 MB/s, which took approximately 52 minutes to complete.

31268536320 bytes (31 GB, 29 GiB) copied, 3159.45 s, 9.9 MB/s

I think bs=4M is fine this time, as it is USB 2.0.
Personally, on my Raspberry Pi, I used bs=1M a lot in the past because it was USB 2.0 with 16GB capacity.
Restoring a backed up image file is still written with the dd command.
Since the file size is large as it is and restoration takes time, the resulting .img file is compressed.

Compression with PiShrink

The procedure for writing the resulting image file also takes nearly an hour for a 29 GB file.
In fact, the actual space used by each partition (bootfs, rootfs) is not that much.
In this example, the actual usage of sdb1 and sdb2 is 75MB/512MB and 4.7GB/29G.

Filesystem   Size   Used Avail Use%        Mounted on
/dev/sdb1    510M   75M  436M  15%    /media/raspida/bootfs
/dev/sdb2    29G    4.7G 23G   18%    /media/raspida/rootfs

When restoring from a backup, in order to write as quickly as possible, the resulting image file should be compressed before writing.
If you compress only the area you are using, the file size will be smaller, and you will not be overwhelmed with space to store the image file. The total time to write is also reduced, killing two birds with one stone.

The application that compresses image files is PiShrink.
PiShrink: https://github.com/Drewsif/PiShrink

After writing with a compressed image file, the partition information shows more free space.

In the previous Raspberry Pi OS, this free space was manually expanded from raspi-config.
However, the current version automatically expands the space (Expanded) at boot time.
PiShrink shrinks the partition from the backed up image file to the used size. This means that the file size is smaller because it does not include free space.

Installing and running PiShrink

First, install the necessary packages.
At the time of writing, only pigz was not installed in the latest version of bookworm.

sudo apt update && sudo apt install -y wget parted gzip pigz xz-utils udev e2fsprogs

Download the script file with wget and copy it to /usr/local/binwith execute rights.

wget https://raw.githubusercontent.com/Drewsif/PiShrink/master/pishrink.sh
chmod +x pishrink.sh
sudo mv pishrink.sh /usr/local/bin

Now you are ready to go.
PiShrink is used in the following format.
This time, the backed up image file was created at ~/usbboot/recovary, so move it and use the following command.

cd ~/usbboot/recovary
sudo pishrink.sh backup-image-name.img

The original .img file is replaced with the compressed size.
As a result, the file size has been reduced from 30 GB to 7.1 GB.

pishrink.sh: Gathering data
Creating new /etc/rc.local
pishrink.sh: Checking filesystem
rootfs: 144755/1846656 files (0.1% non-contiguous), 1345322/7501824 blocks
resize2fs 1.47.0 (5-Feb-2023)
pishrink.sh: Shrinking filesystem
resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/loop0 to 1720340 (4k) blocks.
Begin pass 3 (max = 229)
Scanning inode table XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
The filesystem on /dev/loop0 is now 1720340 (4k) blocks long.
pishrink.sh: Zeroing any free space left
pishrink.sh: Zeroed 1.8G
pishrink.sh: Shrinking partition
pishrink.sh: Truncating image
pishrink.sh: Shrunk backup-image-name.img from 30G to 7.1G

It will resize sufficiently without options.

Write the backed up image file

As a test, I specified the restore destination as a USB flash drive and wrote to it.
Since it is from the same Pi4, Raspberry Pi Imager could be used, but since I went through the trouble, I tried writing to it with the dd command.
The location of the USB memory can be found by looking up sudo fdisk -lin the same way. In this case, it was /dev/sdc.
The block size (bs) is 4M, and the format is the reverse of the if= and of= paths from the backup.

sudo dd bs=4M if=backup-image-name.img of=/dev/sdc conv=fsync status=progress

This is an image file reduced to 7.1 GB by PiShrink.
Although it depends on the area used, this time the write time was only about 10 minutes.
The USB flash drive used has a capacity of 64 GB.

Write by bmap method is also supported

In some image files, you will see .wic instead of the .img extension used to be. This is the bmap method.
bmap image files also support writing to CM4 using this RPI BOOT.
The bmap method is said to be faster than the dd command.
This is because it is optimized for block mapping and is faster than the method of copying the entire image because less data is copied.
In addition, wic can also generate the image file in a wks file after the image is generated (bitbake).
This is most commonly used to create your own images in a Yocto meta-raspberrypi environment.
It can also be generated from an existing image file with the bmaptool create command.
In this case, there is not much advantage as write speed. It is a bit faster, though.
The bmap method is an efficient way to write in a way that is familiar to those working with embedded Linux.
To try it out, I installed bmap-tools and created a bmap file from a backed up image file.

sudo apt install bmap-tools

Output to bmap format from an image file (RAW disk image) before compression by PiShrink or other means.

bmaptool create -o bmap-file-image.bmap backup-image-name.img

You can specify the .bmap file generated here and write it with the bamptool command as well.
For more information about bmap, please refer to Yocto Project and bmap-tools.

• https://ja.wikipedia.org/wiki/Yocto_Project
• https://github.com/intel/bmap-tools
• https://manpages.ubuntu.com/manpages/focal/man1/bmaptool.1.html

Image File Backup

With Pi 4 and Pi 5, the dd command goes straight through without any action.
For CM4 including PL-R4 like this, you can handle dd commands in the same way by changing the settings to a write-enabled state in advance and executing RPI BOOT.
Image files can be compressed with PiShrink to reduce writing time.
The smaller file size would make it more convenient to store image files as backups.
It was a way to backup and restore with the dd command even on an industrial raspi.