hook: introduce post-linuxkit initramfs compressor/optimizer/reporter

[rpardini]

hook: introduce post-linuxkit initramfs compressor/optimizer/reporter

• lk-containers: implement wrappers around linuxkit/* images
  • for each linuxkit/ image used:
    • auto-generate Dockerfile under images/hook-linuxkit-xyz
    • build it as regular lk_containers for the arch
    • replace usage in hook.template.yaml with HOOK_CONTAINER_LINUXKIT_XYZ_IMAGE
  • this way we capitalize on the caching mechanisms already in place
    Signed-off-by: Ricardo Pardini [email protected]
• kernel: armbian: do not remove modules from kernel
  • to make the (rather large) Armbian kernel's modules fit in 2gb RAM
    machines, we used to (quite bluntly) remove modules that weren't
    strictly needed for Hook:
    • drivers/net/wireless
    • kernel/sound
    • drivers/media
    • drivers/infiniband
  • with the initramfs optimization work, we can stop removing those
    • this fixes certain video/DRM problems
    • opens up the wireless path for the future
      Signed-off-by: Ricardo Pardini [email protected]
• bash: bump shellcheck 0.10.0 -> 0.11.0; shellfmt 3.10.0 -> 3.12.0
  • squash out some instances of
    SC2268 (style): Avoid x-prefix in comparisons
    • I'm too old and it shows, I wrote all of them
  • squash one unused var and wrong comment ref
    Signed-off-by: Ricardo Pardini [email protected]
• bash: common: use cat if bat is missing in log_file_bat()
  • otherwise contents wouldn't be shown at all
  • maybe it's time to add bat to downloaded dependencies (like linuxkit)?
    Signed-off-by: Ricardo Pardini [email protected]
• kernel: config: update hook-{default|latest-lts}-defconfigs, no changes
  • this is done via
    • ./build.sh kernel-config hook-latest-lts-amd64
    • ./build.sh kernel-config hook-default-amd64
    • ./build.sh kernel-config hook-default-arm64
    • ./build.sh kernel-config hook-latest-lts-arm64
  • and then making no changes and exiting
  • here, ARM_SDE_INTERFACE=y is being removed as it is implied
    by having ACPI_APEI_GHES=y, ACPI=y, ACPI_APEI=y and ARM64=y
    Signed-off-by: Ricardo Pardini [email protected]
• hook: introduce post-linuxkit initramfs compressor/optimizer/reporter
  • turns out LinuxKit ends up producing initramfs's that are very large
    • multiple copies of exact same files in different fs paths
    • gzip compression
    • multiple cpio layers causing multiple instances of same filepath stored
  • add a Docker-based postprocess step that does multiple tricks to fix it
    • extract/repack cpio to flatten it
    • rdfind to replace duplicates with hardlinks
    • zstd (-9, multithread) compression
    • reports usage and large duplicate files in different paths
      • so we can optimize our lk containers for better dedupe
  • gains are at
    • 25% for zstd compression (meson64)
    • 10Mb for rdfind, without any lk optimization yet
      Signed-off-by: Ricardo Pardini [email protected]
• initramfs: unpack and repack cpio's like the kernel does
  • unpack and repack cpio's like the kernel does; order vs timestamps
    matter, since the kernel overwrites without mtime comparision
    Signed-off-by: Ricardo Pardini [email protected]
• bash: inventory: drop old linuxkit version from 'peg'
  • 'peg' has served its purpose, should we remove it?
    Signed-off-by: Ricardo Pardini [email protected]
• hook: lk-containers: bump linuxkit images to latest matching versions
  • this way we drastically reduce the number of different binaries
    found in the final initramfs, allowing the initramfs compressor
    to deduplicate them into hardlinks
    • libssl / libcrypto
    • busybox
    • musl
      Signed-off-by: Ricardo Pardini [email protected]
• linuxkit: bump LinuxKit 1.6.0 -> 1.8.2
  Signed-off-by: Ricardo Pardini [email protected]
• hook-lk-containers.sh: fix for TARGETARCH
  Signed-off-by: Ricardo Pardini [email protected]
• initramfs: compressor: report gains for both cpio and compressed initramfs
  Signed-off-by: Ricardo Pardini [email protected]

[rpardini]

Done a week of tests, this seems to be working fine. Will force-push with Sign-off's and undraft.

[jacobweinstock]

Thanks, @rpardini !

[mergify]

Merge Queue Status

🚫 The pull request has left the queue (rule: default) at a57e278

This pull request spent 4 seconds in the queue, with no time running CI.

Reason

The pull request can't be updated

Mergify needs the author permission to update the base branch of the pull request.
@rpardini needs to authorize modification on its head branch.

Hint

You should update or rebase your pull request manually. If you do, this pull request will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

[jacobweinstock]

@Mergifyio requeue

[mergify]

requeue

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

[mergify]

Merge Queue Status

🚫 The pull request has left the queue (rule: default) at a57e278

This pull request spent 3 seconds in the queue, with no time running CI.

Reason

The pull request can't be updated

Mergify needs the author permission to update the base branch of the pull request.
@rpardini needs to authorize modification on its head branch.

Hint

You should update or rebase your pull request manually. If you do, this pull request will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.