hook: introduce post-linuxkit initramfs compressor/optimizer/reporter (#299)

#### hook: introduce post-linuxkit initramfs
compressor/optimizer/reporter

- lk-containers: implement wrappers around linuxkit/* images
  - for each linuxkit/<xyz> image used:
    - auto-generate Dockerfile under images/hook-linuxkit-xyz
    - build it as regular lk_containers for the arch
- replace usage in hook.template.yaml with
HOOK_CONTAINER_LINUXKIT_XYZ_IMAGE
  - this way we capitalize on the caching mechanisms already in place
  Signed-off-by: Ricardo Pardini <[email protected]>
- kernel: armbian: do not remove modules from kernel
  - to make the (rather large) Armbian kernel's modules fit in 2gb RAM
    machines, we used to (quite bluntly) remove modules that weren't
    strictly needed for Hook:
    - drivers/net/wireless
    - kernel/sound
    - drivers/media
    - drivers/infiniband
  - with the initramfs optimization work, we can stop removing those
    - this fixes certain video/DRM problems
    - opens up the wireless path for the future
  Signed-off-by: Ricardo Pardini <[email protected]>
- bash: bump shellcheck 0.10.0 -> 0.11.0; shellfmt 3.10.0 -> 3.12.0
  - squash out some instances of
     `SC2268 (style): Avoid x-prefix in comparisons`
    - I'm too old and it shows, I wrote _all_ of them
  - squash one unused var and wrong comment ref
  Signed-off-by: Ricardo Pardini <[email protected]>
- bash: common: use `cat` if `bat` is missing in `log_file_bat()`
  - otherwise contents wouldn't be shown at all
- maybe it's time to add bat to downloaded dependencies (like linuxkit)?
  Signed-off-by: Ricardo Pardini <[email protected]>
- kernel: config: update hook-{default|latest-lts}-defconfigs, no
changes
  - this is done via
    - `./build.sh kernel-config hook-latest-lts-amd64`
    - `./build.sh kernel-config hook-default-amd64`
    - `./build.sh kernel-config hook-default-arm64`
    - `./build.sh kernel-config hook-latest-lts-arm64`
  - and then making no changes and exiting
  - here, `ARM_SDE_INTERFACE=y` is being removed as it is implied
    by having `ACPI_APEI_GHES=y`, `ACPI=y`, `ACPI_APEI=y` and `ARM64=y`
  Signed-off-by: Ricardo Pardini <[email protected]>
- hook: introduce post-linuxkit initramfs compressor/optimizer/reporter
  - turns out LinuxKit ends up producing initramfs's that are very large
    - multiple copies of exact same files in different fs paths
    - gzip compression
- multiple cpio layers causing multiple instances of same filepath
stored
- add a Docker-based postprocess step that does multiple tricks to fix
it
    - extract/repack cpio to flatten it
    - rdfind to replace duplicates with hardlinks
    - zstd (-9, multithread) compression
    - reports usage and large duplicate files in different paths
      - so we can optimize our lk containers for better dedupe
  - gains are at
    - 25% for zstd compression (meson64)
    - 10Mb for rdfind, without any lk optimization yet
  Signed-off-by: Ricardo Pardini <[email protected]>
- initramfs: unpack and repack cpio's like the kernel does
  - unpack and repack cpio's like the kernel does; order vs timestamps
    matter, since the kernel overwrites without mtime comparision
  Signed-off-by: Ricardo Pardini <[email protected]>
- bash: inventory: drop old linuxkit version from 'peg'
  - 'peg' has served its purpose, should we remove it?
  Signed-off-by: Ricardo Pardini <[email protected]>
- hook: lk-containers: bump linuxkit images to latest matching versions
  - this way we drastically reduce the number of different binaries
    found in the final initramfs, allowing the initramfs compressor
    to deduplicate them into hardlinks
    - libssl / libcrypto
    - busybox
    - musl
  Signed-off-by: Ricardo Pardini <[email protected]>
- linuxkit: bump LinuxKit 1.6.0 -> 1.8.2
  Signed-off-by: Ricardo Pardini <[email protected]>
- hook-lk-containers.sh: fix for TARGETARCH
  Signed-off-by: Ricardo Pardini <[email protected]>
- initramfs: compressor: report gains for both cpio and compressed
initramfs
  Signed-off-by: Ricardo Pardini <[email protected]>

Author: jacobweinstock

bash/bootable/armbian-u-boot.sh

@@ -128,6 +128,7 @@ function build_bootable_armbian_uboot() {
 
 	declare -i initramfs_size_bytes
 	initramfs_size_bytes=$(stat --format="%s" "${fat32_root_dir}/initramfs")
+	log debug "Initramfs size (bytes): ${initramfs_size_bytes}"
 
 	# DTBs go into a dtb subdirectory
 	mkdir -p "${fat32_root_dir}/dtb"

bash/bootable/fat32-image.sh

@@ -13,7 +13,7 @@ function create_image_fat32_root_from_dir() {
 	# Create a Dockerfile; install parted and mtools
 	mkdir -p "bootable"
 	declare dockerfile_helper_filename="undefined.sh"
-	produce_dockerfile_helper_apt_oras "bootable/" # will create the helper script in bootable/ directory; sets helper_name
+	produce_dockerfile_helper_apt_oras "bootable/" # will create the helper script in bootable/ directory; sets dockerfile_helper_filename
 
 	# Lets create a Dockerfile that will be used to create the FAT32 image
 	cat <<- MKFAT32_SCRIPT > "bootable/Dockerfile.autogen.helper.mkfat32.sh"

bash/cli.sh

@@ -20,7 +20,7 @@ function parse_command_line_arguments() {
 				log error "Invalid command line parameter '${param_name}=${param_value_desc}'"
 				exit 8
 			fi
-		elif [[ "x${arg}x" != "xx" ]]; then # not a param, not empty, store it in the non-param array for later usage
+		elif [[ "${arg}" != "" ]]; then # not a param, not empty, store it in the non-param array for later usage
 			local non_param_value="${arg}"
 			local non_param_value_desc="${non_param_value:-(empty)}"
 			log debug "Command line: non-param argument" "'${non_param_value_desc}'"

bash/common.sh

@@ -51,7 +51,8 @@ function log_file_bat() {
 	elif command -v batcat > /dev/null; then
 		batcat --color=always --paging=never "${extra_bat_args[@]}" "${file}"
 	else
-		log "${level}" "'bat' utility not installed; install it to see file contents in logs."
+		log "${level}" "'bat' utility not installed; install it to see file contents colorized in logs."
+		cat "${file}"
 	fi
 }
 

bash/hook-lk-containers.sh

@@ -14,6 +14,43 @@ function build_all_hook_linuxkit_containers() {
 	build_hook_linuxkit_container hook-containerd "HOOK_CONTAINER_CONTAINERD_IMAGE" "${EXPORT_LK_CONTAINERS}" "${EXPORT_LK_CONTAINERS_DIR}"
 	build_hook_linuxkit_container hook-runc "HOOK_CONTAINER_RUNC_IMAGE" "${EXPORT_LK_CONTAINERS}" "${EXPORT_LK_CONTAINERS_DIR}"
 	build_hook_linuxkit_container hook-embedded "HOOK_CONTAINER_EMBEDDED_IMAGE" "${EXPORT_LK_CONTAINERS}" "${EXPORT_LK_CONTAINERS_DIR}"
+
+	# We also use a bunch of linuxkit/xxx:v1.0.0 images; those would be pulled from Docker Hub (and thus subject to rate limits) for each Hook build.
+	# Instead, we'll wrap them into a Dockerfile with just a FROM line, and build/push them ourselves.
+	# Those versions are obtained from the references in https://github.com/linuxkit/linuxkit/tree/master/examples
+	declare -A linuxkit_proxy_images=()
+	linuxkit_proxy_images+=(["init"]="linuxkit/init:b5506cc74a6812dc40982cacfd2f4328f8a4b12a")
+	linuxkit_proxy_images+=(["ca_certificates"]="linuxkit/ca-certificates:256f1950df59f2f209e9f0b81374177409eb11de")
+	linuxkit_proxy_images+=(["firmware"]="linuxkit/firmware:68c2b29f28f2639020b9f8d55254d333498a30aa")
+	linuxkit_proxy_images+=(["rngd"]="linuxkit/rngd:984eb580ecb63986f07f626b61692a97aacd7198")
+	linuxkit_proxy_images+=(["sysctl"]="linuxkit/sysctl:97e8bb067cd9cef1514531bb692f27263ac6d626")
+	linuxkit_proxy_images+=(["sysfs"]="linuxkit/sysfs:6d5bd933762f6b216744c711c6e876756cee9600")
+	linuxkit_proxy_images+=(["modprobe"]="linuxkit/modprobe:4248cdc3494779010e7e7488fc17b6fd45b73aeb")
+	linuxkit_proxy_images+=(["dhcpcd"]="linuxkit/dhcpcd:b87e9ececac55a65eaa592f4dd8b4e0c3009afdb")
+	linuxkit_proxy_images+=(["openntpd"]="linuxkit/openntpd:2508f1d040441457a0b3e75744878afdf61bc473")
+	linuxkit_proxy_images+=(["getty"]="linuxkit/getty:a86d74c8f89be8956330c3b115b0b1f2e09ef6e0")
+	linuxkit_proxy_images+=(["sshd"]="linuxkit/sshd:08e5d4a46603eff485d5d1b14001cc932a530858")
+
+	# each of those will handled the following way:
+	# - create+clean a directory under images; eg for key "init" create images/hook-linuxkit-init
+	#  (all of images/hook-linuxkit-* are .gitignored)
+	# - create a Dockerfile with "FROM --platform=xxx linuxkit/init:v1.1.0" in that directory
+	# - determine HOOK_CONTAINER_LINUXKIT_<key:toUpper>_IMAGE variable name
+	# - call build_hook_linuxkit_container with that directory and variable name
+	# that way, everything else works exactly as with the other images, and there's  now a DockerHub-free way of getting those images
+	# it works because build_hook_linuxkit_container does content-based hashing; so tags should be stable for the same version
+	# that potentializes the use of caching with docker save/load or other local caching mechanisms.
+	declare lk_proxy_image_key="undetermined" lk_proxy_image_ref="undetermined" lk_proxy_image_dir="undetermined" lk_proxy_image_var="undetermined"
+	for lk_proxy_image_key in "${!linuxkit_proxy_images[@]}"; do
+		lk_proxy_image_ref="${linuxkit_proxy_images[${lk_proxy_image_key}]}"
+		lk_proxy_image_dir="hook-linuxkit-${lk_proxy_image_key}"
+		lk_proxy_image_var="HOOK_CONTAINER_LINUXKIT_$(echo "${lk_proxy_image_key}" | tr '[:lower:]' '[:upper:]')_IMAGE"
+		log info "Preparing LinuxKit proxy image ${lk_proxy_image_ref} in ${lk_proxy_image_dir}, variable name ${lk_proxy_image_var}"
+		rm -rf "images/${lk_proxy_image_dir}"
+		mkdir -p "images/${lk_proxy_image_dir}"
+		echo "FROM --platform=\${TARGETARCH} ${lk_proxy_image_ref}" > "images/${lk_proxy_image_dir}/Dockerfile"
+		build_hook_linuxkit_container "${lk_proxy_image_dir}" "${lk_proxy_image_var}" "${EXPORT_LK_CONTAINERS}" "${EXPORT_LK_CONTAINERS_DIR}"
+	done
 }
 
 function build_hook_linuxkit_container() {

bash/inventory.sh

@@ -35,7 +35,7 @@ function produce_default_kernel_inventory() {
 
 	## A 'peg' is not really a 'hook': for development purposes; testing new LK version and simpler LK configurations, using the default kernel
 	define_id "peg-default-amd64" METHOD='default' ARCH='x86_64' TAG='dev' \
-		USE_KERNEL_ID='hook-default-amd64' TEMPLATE='peg' LINUXKIT_VERSION='1.2.0' \
+		USE_KERNEL_ID='hook-default-amd64' TEMPLATE='peg' \
 		KERNEL_MAJOR='5' KERNEL_MINOR='10' KCONFIG='generic'
 
 	## development purposes: trying out kernel 6.6.y

bash/json-matrix.sh

@@ -198,7 +198,7 @@ function json_matrix_find_runner() {
 	declare -a vars_to_try=("CI_RUNNER_${matrix_type^^}_${docker_arch^^}" "CI_RUNNER_${matrix_type^^}" "CI_RUNNER_${docker_arch^^}" "CI_RUNNER")
 	for var in "${vars_to_try[@]}"; do
 		log debug "Checking var '${var}'"
-		if [[ -n "${!var}" && "x${!var}x" != "xx" ]]; then # if var is set, and not empty...
+		if [[ -n "${!var}" && "${!var}" != "" ]]; then # if var is set, and not empty...
 			log debug "Found runner '${!var}' for matrix type '${matrix_type}' and docker arch '${docker_arch}' via var '${var}'"
 			runner="${!var}"
 			break

bash/kernel/kernel_armbian.sh

@@ -72,8 +72,8 @@ function calculate_kernel_version_armbian() {
 		RUN mv /armbian/image/lib /armbian/modules_only/
 		RUN echo "Before cleaning: " && du -h -d 10 -x lib/modules | sort -h | tail -n 20
 		# Trim the kernel modules to save space; hopefully your required hardware is not included here
-		RUN rm -rf ./lib/modules/*/kernel/drivers/net/wireless ./lib/modules/*/kernel/sound ./lib/modules/*/kernel/drivers/media
-		RUN rm -rf ./lib/modules/*/kernel/drivers/infiniband
+		# DISABLED # RUN rm -rf ./lib/modules/*/kernel/drivers/net/wireless ./lib/modules/*/kernel/sound ./lib/modules/*/kernel/drivers/media
+		# DISABLED # RUN rm -rf ./lib/modules/*/kernel/drivers/infiniband
 		RUN echo "After cleaning: " &&  du -h -d 10 -x lib/modules | sort -h | tail -n 20
 		RUN tar -cf /armbian/output/kernel.tar .
 

bash/kernel/kernel_default.sh

@@ -9,7 +9,7 @@ function obtain_kernel_output_id_default() {
 	# If that is not set, and KCONFIG != generic, an output will be generated with KCONFIG, MAJOR, MINOR, ARCH.
 	# Lastly if using USE_KERNEL_ID, that will be used instead of the default inventory_id.
 	declare -g OUTPUT_ID="${ARCH}"
-	if [[ "x${FORCE_OUTPUT_ID}x" != "xx" ]]; then
+	if [[ "${FORCE_OUTPUT_ID}" != "" ]]; then
 		declare -g OUTPUT_ID="${FORCE_OUTPUT_ID}-${ARCH}"
 	elif [[ "${KCONFIG}" != "generic" ]]; then
 		OUTPUT_ID="${KCONFIG}-${KERNEL_MAJOR}.${KERNEL_MINOR}.y-${ARCH}"

bash/linuxkit.sh

@@ -136,17 +136,143 @@ function linuxkit_build() {
 	"${linuxkit_bin}" build "--format" "kernel+initrd" "${lk_debug_args[@]}" "${lk_args[@]}"
 
 	declare initramfs_path="${lk_output_dir}/hook-initrd.img"
+
 	# initramfs_path is a gzipped file. obtain the uncompressed byte size, without decompressing it
-	declare -i initramfs_size_bytes=0
-	initramfs_size_bytes=$(gzip -l "${initramfs_path}" | tail -n 1 | awk '{print $2}')
-	log info "Uncompressed initramfs size in bytes: ${initramfs_size_bytes}"
-	# If the size is larger than 900mb, it is unlikely to boot on a 2gb RAM machine. Warn.
-	if [[ "${initramfs_size_bytes}" -gt 943718400 ]]; then
-		log warn "${inventory_id}: Uncompressed initramfs size (${initramfs_size_bytes} bytes) is larger than 900mb; it may not boot on a 2gb RAM machine."
-	else
-		log notice "${inventory_id}: Uncompressed initramfs size (${initramfs_size_bytes} bytes) is smaller than 900mb."
+	declare -i initramfs_size_bytes_initial=0 initramfs_size_bytes_gzip=0 initramfs_size_bytes_zstd=0
+	initramfs_size_bytes_gzip=$(stat -c%s "${initramfs_path}")
+	initramfs_size_bytes_initial=$(gzip -l "${initramfs_path}" | tail -n 1 | awk '{print $2}')
+	log info "Compressed-gzip (initial) initramfs size in bytes: ${initramfs_size_bytes_gzip}"
+	log info "Uncompressed initial initramfs size in bytes: ${initramfs_size_bytes_initial}"
+
+	# Brief detour to:
+	# 1) Decompress the initramfs (`gunzip`) and extract it to a directory (`cpio`)
+	#    This de-duplicates some cpio-duplicates leftover by linuxkit (some kb's)
+	# 2) Produce a reports on the initramfs contents:
+	#    - disk usage (by size) of the initramfs contents (du -h -d 10 -x | sort -h | tail -n 20)
+	#    - aggregated basename-identical files in the initramfs, with their size and hash
+	#      This will help us find things to optimize in the lkcontainers:
+	#      - use same base image for all cotntainers (deduplicate musl + others)
+	#      - avoid different versions of stuff (containerd in hook-containerd but also in hook-docker)
+	#      - avoid large files that are not needed in the initramfs (docs)
+	# 3) Use `rdfind` to replace exact duplicates with hardlinks (many mb's!)
+	# 4) Repack the initramfs into `cpio` and compress it with `zstd` level 9 (about 30% better, many mb's!)
+	#    All the Hook kernels already support zstd initramfs decompression, so this is safe to do. Performance might be better too.
+	#
+	# Since we need tools and do-it-as-root for this, its best done using a Docker container
+	declare -a compressor_deps=("bash" "gawk" "cpio" "zstd" "rdfind" "gzip" "pigz" "coreutils" "findutils" "file" "du-dust")
+	declare initramfs_compressor_dockerfile="${lk_output_dir}/Dockerfile.initramfs_compressor"
+	declare -r output_compressed_initramfs_name="initramfs-compressed.img" output_report_name="report.md"
+
+	declare find_same_name_files_command cpio_extract_like_the_kernel_does_command cpio_repack_for_kernel_command
+	# I *really* don't want to escape this; bear with me
+	find_same_name_files_command="$(
+		cat <<- 'FIND_SAME_NAME_FILES_COMMAND'
+			find . -type f -size +512k -printf "%f %p\n" | sort | awk '{files[$1]=files[$1] ? files[$1] "\n"$2 : $2; count[$1]++} END {for (f in count) if (count[f]>1) print f "\n" files[f]}' | while read -r line; do if [[ -f "$line" ]]; then stat --printf="%s bytes " "$line"; md5sum "$line"; else echo "### duplicate: '$line'"; fi; done
+		FIND_SAME_NAME_FILES_COMMAND
+	)"
+
+	# cpio command that mimics what the kernel does when extracting initramfs
+	#  -i: extract from archive (copy-in)
+	#  -d: create directories as needed
+	#  -m: preserve modification times (so files get the archive mtime, like the kernel does)
+	#  -u: unconditionally replace existing files (this is the key to "kernel-like" behavior)
+	#  --no-absolute-filenames: avoid writing absolute paths (shouldn't be any anyway, but don't trust - not a kernel concern)
+	cpio_extract_like_the_kernel_does_command="cpio -idmu --no-absolute-filenames"
+
+	# cpio repack, newc is the format the kernel expects
+	cpio_repack_for_kernel_command="cpio -o -H newc"
+
+	log info "Creating Dockerfile '${initramfs_compressor_dockerfile}'... "
+	cat <<- INITRAMFS_COMPRESSOR_DOCKERFILE > "${initramfs_compressor_dockerfile}"
+		FROM debian:stable AS builder
+		RUN mkdir -p /output
+		ENV DEBIAN_FRONTEND=noninteractive
+		RUN apt-get -qq -o "Dpkg::Use-Pty=0" update || apt-get -o "Dpkg::Use-Pty=0" update
+		RUN apt-get -qq install -o "Dpkg::Use-Pty=0" -q -y ${compressor_deps[*]} || apt-get install -o "Dpkg::Use-Pty=0" -q -y ${compressor_deps[*]}
+		SHELL ["/bin/bash", "-c"]
+
+		ADD hook-initrd.img /input/initramfs.img
+		WORKDIR /work/dir
+		RUN echo "# Tinkerbell Hook LinuxKit initramfs compressor report" > /output/${output_report_name}
+		RUN { echo -n "## input magic: " && file /input/initramfs.img; }>> /output/${output_report_name}
+
+		RUN pigz -d -c /input/initramfs.img > /input/initramfs_decompress.cpio
+		#RUN zcat /input/initramfs.img > /input/initramfs_decompress.cpio
+
+		RUN { echo -n "## ungzipped input magic: " && file /input/initramfs_decompress.cpio; }>> /output/${output_report_name}
+
+		RUN cat /input/initramfs_decompress.cpio | ${cpio_extract_like_the_kernel_does_command}
+
+		# Reporting on original...
+		RUN { echo "## original: dust report: " && dust -x --no-colors --no-percent-bars ; }>> /output/${output_report_name}
+		RUN { echo "## original: top-40 dirs usage 5-deep (du): " && du -h -d 5 -x . | sort -h | tail -40 ; }>> /output/${output_report_name}
+		RUN { echo "## original: same-name files, larger than 512kb: " && $find_same_name_files_command ; }>> /output/${output_report_name}
+		RUN { echo -n "## original: hardlinked files: " && find . -type f -links +1 | wc -l ; }>> /output/${output_report_name}
+
+		# -> Deduplicate exact files into hardlinks with rdfind
+		RUN { echo "## rdfind run: " && rdfind -makehardlinks true -deleteduplicates true -makeresultsfile false . ; }>> /output/${output_report_name}
+
+		# Reporting after deduplication
+		RUN { echo "## deduped: dust report: " && dust -x --no-colors --no-percent-bars ; }>> /output/${output_report_name}
+		RUN { echo -n "## deduped: hardlinked files: " && find . -type f -links +1 | wc -l ; }>> /output/${output_report_name}
+
+		RUN find . | ${cpio_repack_for_kernel_command} > /output/repacked.cpio
+		RUN { echo -n "## output, pre compression magic: " && file /output/repacked.cpio; }>> /output/${output_report_name}
+
+		RUN zstdmt -9 -o /output/${output_compressed_initramfs_name} /output/repacked.cpio
+		RUN { echo -n "## output magic: " && file /output/${output_compressed_initramfs_name}; }>> /output/${output_report_name}
+
+		# Report on the reductions done.
+		# First, the original cpio vs the repacked cpio; this is twice in memory and uncompressed so most relevant
+		RUN { echo "## size reduction: original cpio vs repacked cpio: " && ls -lh /input/initramfs_decompress.cpio /output/repacked.cpio ; }>> /output/${output_report_name}
+		# Then, the original gzipped initramfs vs the final zstd initramfs; this is only once in memory but affects download/TFTP time
+		RUN { echo "## size reduction: original gzipped initramfs vs final zstd initramfs: " && ls -lh /input/initramfs.img /output/${output_compressed_initramfs_name} ; }>> /output/${output_report_name}
+		FROM scratch
+		COPY --from=builder /output/* /
+	INITRAMFS_COMPRESSOR_DOCKERFILE
+
+	declare docker_compressor_output_dir="${lk_output_dir}/initramfs_compressor_output"
+	mkdir -p "${docker_compressor_output_dir}"
+
+	# Now, build the Dockerfile and output the fat32 image directly
+	log info "Building Dockerfile for initramfs compressor and outputting directly to '${docker_compressor_output_dir}'..."
+	declare -a compressor_docker_buildx_args=(
+		--output "type=local,dest=${docker_compressor_output_dir}" # output directly to a local dir, not an image
+		"--progress=${DOCKER_BUILDX_PROGRESS_TYPE}"                # show progress
+		-f "${initramfs_compressor_dockerfile}"                    # Dockerfile path
+		"${lk_output_dir}")                                        # build context, for easy access to the input initramfs file
+
+	log_file_bat "${initramfs_compressor_dockerfile}" "debug" "Dockerfile for initramfs compressor"
+
+	docker buildx build "${compressor_docker_buildx_args[@]}"
+
+	# If output not in place, something went wrong
+	if [[ ! -f "${docker_compressor_output_dir}/${output_compressed_initramfs_name}" ]]; then
+		log error "Failed to produce compressed initramfs at expected location '${docker_compressor_output_dir}/${output_compressed_initramfs_name}'"
+		exit 8
 	fi
 
+	# If report not in place, something went wrong
+	if [[ ! -f "${docker_compressor_output_dir}/${output_report_name}" ]]; then
+		log error "Failed to produce compressed initramfs at expected location '${docker_compressor_output_dir}/${output_report_name}'"
+		exit 9
+	fi
+
+	# Output the report (use DEBUG=yes to see it)
+	log_file_bat "${docker_compressor_output_dir}/${output_report_name}" "info" "Compression report for initramfs ${inventory_id}:"
+
+	# Move the outputted compressed initramfs into the original location
+	mv "${debug_dash_v[@]}" "${docker_compressor_output_dir}/${output_compressed_initramfs_name}" "${initramfs_path}"
+
+	# Clean up the temporary Dockerfile and output dir - not if debugging
+	if [[ "${DEBUG}" != "yes" ]]; then
+		rm -rf "${initramfs_compressor_dockerfile}" "${docker_compressor_output_dir}"
+	fi
+
+	# Calculate the final initramfs zstd-compressed size, then brag about zstd's prowess
+	initramfs_size_bytes_zstd=$(stat -c%s "${initramfs_path}")
+	log notice "${inventory_id}: Final zstd+deduped initramfs size (${initramfs_size_bytes_zstd} bytes) vs initial gzip-compressed size (${initramfs_size_bytes_gzip} bytes): size reduced by $((100 - (initramfs_size_bytes_zstd * 100 / initramfs_size_bytes_gzip)))%"
+
 	if [[ "${LK_RUN}" == "qemu" ]]; then
 		linuxkit_run_qemu
 		return 0