image build failure - dpkg: error processing package grub-efi-amd64-signed

[Nobuto Murata]

The symptom is pretty close to the previous bug:
https://bugs.launchpad.net/snap-octavia-diskimage-retrofit/+bug/1995130

With yoga deployment, the Octavia retrofitting process started failing this week. I'm not sure if it's the root cause, but I see one update was released for jammy on 2023-09-29 for grub-efi-amd64-signed.
https://launchpad.net/ubuntu/+source/grub2-signed

[/var/log/juju/unit-octavia-diskimage-retrofit-0.log]

[ 63.1] Running: pre-install.d/42-ubuntu-ppa
[ 63.2] Running: pre-install.d/99-apt-get-update
[ 65.2] Running: pre-install.d/99-package-uninstalls
[ 65.3] Installing: initramfs-tools
[ 66.0] Running: install.d/00-baseline-environment
[ 66.8] Running: install.d/00-up-to-date
virt-dib: error: libguestfs error: sh: Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages will be upgraded:
  grub-efi-amd64-bin grub-efi-amd64-signed
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2980 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64
grub-efi-amd64-signed amd64 1.187.4~22.04.1+2.06-2ubuntu14.2 [1389 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64
grub-efi-amd64-bin amd64 2.06-2ubuntu14.2 [1591 kB]
Fetched 2980 kB in 2s (1573 kB/s)
(Reading database ...

...

(Reading database
... 64309 files and directories currently installed.)
Preparing to unpack
.../grub-efi-amd64-signed_1.187.4~22.04.1+2.06-2ubuntu14.2_amd64.deb ...
Unpacking grub-efi-amd64-signed (1.187.4~22.04.1+2.06-2ubuntu14.2) over
(1.187.3~22.04.1+2.06-2ubuntu14.1) ...
Preparing to unpack .../grub-efi-amd64-bin_2.06-2ubuntu14.2_amd64.deb ...
Unpacking grub-efi-amd64-bin (2.06-2ubuntu14.2) over (2.06-2ubuntu14.1)
...
Setting up grub-efi-amd64-bin (2.06-2ubuntu14.2) ...
Setting up grub-efi-amd64-signed (1.187.4~22.04.1+2.06-2ubuntu14.2) ...
Trying to migrate /boot/efi into esp config
dpkg: error processing package grub-efi-amd64-signed (--configure):
 installed grub-efi-amd64-signed package post-installation script
subprocess returned error exit status 1
Errors were encountered while processing:
 grub-efi-amd64-signed
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

If reporting bugs, run virt-dib with debugging enabled and include the
complete output:

  virt-dib -v -x [...]
"
2023-10-02 04:05:06 ERROR unit.octavia-diskimage-retrofit/0.juju-log server.go:316 action "retrofit-image" failed: "Command '['octavia-diskimage-retrofit', '/var/snap/octavia-diskimage-retrofit/common/tmp/tmpb0m5_m00', '/var/snap/octavia-diskimage-retrofit/common/tmp/tmpzmylc49x']' returned non-zero exit status 1." "Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-octavia-diskimage-retrofit-0/charm/actions/retrofit-image", line 66, in main
    action(args)
  File "/var/lib/juju/agents/unit-octavia-diskimage-retrofit-0/charm/actions/retrofit-image", line 44, in retrofit_image
    instance.retrofit(
  File "/var/lib/juju/agents/unit-octavia-diskimage-retrofit-0/charm/lib/charm/openstack/octavia_diskimage_retrofit.py", line 274, in retrofit
    output = subprocess.check_output(
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['octavia-diskimage-retrofit', '/var/snap/octavia-diskimage-retrofit/common/tmp/tmpb0m5_m00', '/var/snap/octavia-diskimage-retrofit/common/tmp/tmpzmylc49x']' returned non-zero exit status 1.

[Nobuto Murata (nobuto)]

Attaching ~field-critical as a workaround is not known yet.

[Nobuto Murata (nobuto)]

with debug on and the following steps:

# cd /var/snap/octavia-diskimage-retrofit/common

# wget https://cloud-images.ubuntu.com/releases/jammy/release-20230927/ubuntu-22.04-server-cloudimg-amd64.img

# octavia-diskimage-retrofit -d \
    ubuntu-22.04-server-cloudimg-amd64.img \
    output.img |& tee octavia-diskimage-retrofit_debug.log

[Nobuto Murata (nobuto)]

It was a simple thing. The fix from the previous bug is not included in the stable channel nor any snap channel.

[added]
https://bugs.launchpad.net/snap-octavia-diskimage-retrofit/+bug/1995130
https://github.com/openstack-charmers/octavia-diskimage-retrofit/pull/32/files

[removed]
https://github.com/openstack-charmers/octavia-diskimage-retrofit/commit/6d294e6325078e2175f4950d7ef83814192d2f7c

So a workaround is to use the daily stream of cloud images. But I will keep the field-critical as it's not considered as a safe workaround.

channels:
  latest/stable: –
  latest/candidate: –
  latest/beta: 0.9.12 2021-07-16 (236) 185MB devmode
  latest/edge: 1.0.0+git7.gf0dac14 2023-03-19 (352) 279MB classic
  1.0/stable: 1.0.1 2023-01-19 (344) 277MB classic
  1.0/candidate: ↑
  1.0/beta: ↑
  1.0/edge: 1.0.1+git2.g35bba4a 2023-05-27 (356) 279MB classic
installed: 1.0.1 (344) 277MB classic

$ diff -u /snap/octavia-diskimage-retrofit/{344,356}/usr/local/lib/elements/bug1895835/pre-install.d/42-drop-grub-efi-pkgs
-> no diff

# cat /snap/octavia-diskimage-retrofit/356/usr/local/lib/elements/bug1895835/pre-install.d/42-drop-grub-efi-pkgs
====
#!/bin/sh

set -e

if [ "$DIB_RELEASE"x != jammyx ]; then
    # Remove EFI packages to work around LP: #1895835
    echo "Removing GRUB EFI packages to work around LP: #1895835..."
    DEBIAN_FRONTEND=noninteractive apt-get remove \
        --allow-remove-essential \
        --ignore-missing \
        --purge \
        -y \
        grub-efi-amd64-bin grub-efi-amd64-signed
fi
====

-> no logic for jammy.

[James Page (james-page)]

revision 352 in latest/edge should have this fix in it.

We need to sort out pushing that to a stable branch and backporting to the 1.0 track

[James Page (james-page)]

Tested successfully with 1.0/edge - promoted to 1.0/candidate in preparation for release to stable.

[James Page (james-page)]

https://paste.ubuntu.com/p/44FfW3sZNy/

[Nobuto Murata (nobuto)]

I've tested the revision 362, and it works for me.

Also, I've confirmed that the generated image works for amphora instances to act as a load balancer for Kubernetes control plane.

octavia-diskimage-retrofit 1.0.1+git4.gb7bdb03 362 1.0/candidate canonical** classic

$ juju run-action --wait octavia-diskimage-retrofit/leader retrofit-image
unit-octavia-diskimage-retrofit-0:
  UnitId: octavia-diskimage-retrofit/0
  id: "52"
  results: {}
  status: completed
  timing:
    completed: 2023-10-04 02:13:43 +0000 UTC
    enqueued: 2023-10-04 02:06:58 +0000 UTC
    started: 2023-10-04 02:07:02 +0000 UTC

$ openstack image show amphora-haproxy-x86_64-ubuntu-22.04-20230927 --format yaml
checksum: 9489e807188adab30605134911dae6f2
container_format: bare
created_at: '2023-10-04T02:13:27Z'
disk_format: qcow2
file: /v2/images/2ba11f7c-a1db-47d4-84a5-c8d2102bf8dc/file
id: 2ba11f7c-a1db-47d4-84a5-c8d2102bf8dc
min_disk: 0
min_ram: 0
name: amphora-haproxy-x86_64-ubuntu-22.04-20230927
owner: ec98e711291f42048d8161a6b50f9491
properties:
  direct_url: rbd://5bf8c2a8-60cc-11ee-b944-7fe79167e4b1/glance/2ba11f7c-a1db-47d4-84a5-c8d2102bf8dc/snap
  locations:
  - metadata:
      store: ceph
    url: rbd://5bf8c2a8-60cc-11ee-b944-7fe79167e4b1/glance/2ba11f7c-a1db-47d4-84a5-c8d2102bf8dc/snap
  os_hash_algo: sha512
  os_hash_value: a49e8c5e1d06dfe7820811f3c8b4df14c89c5821b31e4b9da1ef1e630e3cdf2ad40b49bf81c54ed966d7b3cd69da678cedda585b630fb203cb435d28bf06f563
  os_hidden: false
  source_product_name: com.ubuntu.cloud:server:22.04:amd64
  source_version_name: '20230927'
  stores: ceph
protected: false
schema: /v2/schemas/image
size: 920911872
status: active
tags:
- octavia-amphora
- octavia-diskimage-retrofit
updated_at: '2023-10-04T02:13:42Z'
virtual_size: 6656360448
visibility: shared

$ openstack loadbalancer list --format yaml
- id: a4724fe2-e6fa-47d9-8b4d-75dee5548046
  name: openstack-integrator-d4793dc51966-kubernetes-control-plane
  operating_status: ONLINE
  project_id: fa1080b882c44deeb58303548ccb56ce
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 10.5.5.14

[James Page (james-page)]

Testing looks good - just doing a quick run through CI to confirm:

https://review.opendev.org/c/openstack/charm-octavia-diskimage-retrofit/+/897326

[James Page (james-page)]

Pushed to 1.0/stable channel - marking Fix Released.