supermin/liguestfs fails to configure network

[Ioanna Alifieraki]

[Impact]
libguestfs cannot configure network on Bionic onward.

This bug is a combination of libguestfs/supermin package and
/etc/dhcp/dhclient-enter-hooks.d/resolved script from systemd,
present on Bionic onward.
When supermin creates the appliance does chroot and executes its init script.
If networking is enabled init will call dhclient sript to configure the network.

On Bionic onward the make_resolv_conf function of dhclient_script is overwritten
in /etc/dhcp/dhclient-eneter-hooks.d/resolved which before exiting restarts
the systemd.resolved service.
However, this happening in chroot environment fails with
"System has not been booted with systemd as init system (PID 1). Can't operate."
and network is left unconfigured.

[Test Case]

$ sudo guestfish -a xenial-server-cloudimg-amd64-disk1.img --network -v << EOF
run
mount /dev/sda1 /
command 'apt update'
EOF

libguestfs: launch: program=guestfish
libguestfs: launch: version=1.36.13
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
...
supermin: deleting initramfs files
supermin: chroot
Starting /init script ...
...
+ dhclient --version
+ dhclient eth0
System has not been booted with systemd as init system (PID 1). Can't operate.
...
commandrvf: apt update

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-backports/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/xenial-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
...

[Regression Potential]

Minimal. The fix removes the /etc/dhcp/dhclient-eneter-hooks.d/resolved hook installed by systemd.
systemd is not used inside the appliance so it should not cause any regression.

https://bugs.launchpad.net/ubuntu/cosmic/+source/libguestfs/+bug/1824236/comments/18
https://bugs.launchpad.net/ubuntu/cosmic/+source/libguestfs/+bug/1824236/comments/19

[Other]

Affects B,C,D,E.

Upstream fix : https://github.com/libguestfs/libguestfs/commit/2bb6be333e6347d3f18856627d8ad8e50b8e5427

Debian bug : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=929807

-----

Workaround

1) Assume that libguestfs is installed, if not :
$ sudo apt-get install libguestfs-tools

2) Move the base.tar.gz to a temp dir, extract and remove tarball
$ sudo mv /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/base.tar.gz ~/tempdir/
$ cd ~/tempdir
$ sudo tar -xzvf base.tar.gz
$ sudo rm base.tar.gz

3) Remove the etc/dhcp/dhclient-enter-hooks.d/resolved file
$ sudo rm etc/dhcp/dhclient-enter-hooks.d/resolved

4) Create tarball again
$ sudo tar -czvf base.tar.gz etc

5) Move it back to installation dir
$ sudo mv base.tar.gz /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/

6) Clean cache
$ sudo rm -rf /var/tmp/.guestfs*

[Richard Jones (rjones-redhat)]

I posted a potential fix here:
https://www.redhat.com/archives/libguestfs/2019-April/msg00129.html

[Ioanna Alifieraki (joalif)]

Debdiff for Bionic.

[Ioanna Alifieraki (joalif)]

Debdiff for Cosmic.

[Ioanna Alifieraki (joalif)]

Debdiff for Disco.

[Ioanna Alifieraki (joalif)]

Debdiff for Disco.

[Ioanna Alifieraki (joalif)]

Debdiff for Eoan.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1824236_bionic.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Eric Desrochers (slashd)]

[sts-sponsor]

I have some concern about the patch being more or less suggested by upstream, but never merged upstream. I see they previously merge a workaround for Ubuntu, can you discuss with them and see if that can be merged again upstream ? I'm trying to avoid to have to carry a patch in Ubuntu for libguestfs forever because upstream didn't merge it.

As I read the code so far, the workaround does (and only needed in ubuntu FWIU):
    touch /etc/fstab
    rm -f /etc/dhcp/dhclient-enter-hooks.d/resolved

but the workaround is applied to all, regardless of the Linux distribution.

If we PR something upstream, would it be possible to suggest a separate if statement which will detect if it's ubuntu and/or debian

What would you think of something like this (or something along those line, so that the workaround is only applied in a ubuntu context) it would look like cleaner IMHO:
-----
++ # Workarounds for ubuntu/debian
++ ubuntu = <SOMETHING>
++ if test "$guestfs_network" = 1 and ubuntu = <SOMETHING>; then
++ touch /etc/fstab
++ rm -f /etc/dhcp/dhclient-enter-hooks.d/resolved
fi

+ if test "$guestfs_network" = 1; then
+ iface=$(ls -I all -I default -I lo /proc/sys/net/ipv4/conf)
+- touch /etc/fstab # Workaround for Ubuntu.
-- # Two workarounds for Ubuntu:
-- touch /etc/fstab
-- rm -f /etc/dhcp/dhclient-enter-hooks.d/resolved
+ if dhclient --version >/dev/null 2>&1; then
+ dhclient $iface
+ else
-----

I don't feel comfortable to sponsor this in Eoan, before further discussion with upstream to merged the change.

[Ioanna Alifieraki (joalif)]

Patch merged upstream :
https://github.com/libguestfs/libguestfs/commit/2bb6be333e6347d3f18856627d8ad8e50b8e5427

Preparing new debdiffs.

[Ioanna Alifieraki (joalif)]

New debdiff for E.

[Ioanna Alifieraki (joalif)]

New debdiff for D.

[Ioanna Alifieraki (joalif)]

New debdiff for C.

[Ioanna Alifieraki (joalif)]

New debdiff for B.

[Eric Desrochers (slashd)]

Ok I'll review the debdiff for 'Eoan' tomorrow.

Thanks Ionna.

[Eric Desrochers (slashd)]

Thanks Ioanna for providing a patch and work with upstream maintainer.

I have some concern that I want to address before talking about sponsorship.
I haven't tested libguestfs, so my observation are only base on the patch you have provided upstream, and my concern inside Ubuntu.

I would have prefer to see upstream adopting an approach based on detecting if the host run 'Ubuntu' then apply the workaround, otherwise don't apply it. Instead of deploying the workaround everywhere regardless of the Linux distribution. IMHO, it would have look more cleaner and less risky since the ubuntu workaround involve a remove file operation. I would have prefer to isolate the 'rm' operation at the maximum and limit it only when really needed.

Talking about this 'rm' operation[0], it deletes a 'dpkg-managed' file part of systemd package[1]. It's a pretty critical package here, so you would understand I'm taking this seriously. I'm afraid this could impact user who benefit of this libguestfs workaround at next systemd package upgrade if the file is absent for instance, and possibly break the next pkg upgrade of systemd

Could you please verify that aspect ?

I'm also concern about what could be other potential impact in the absence of this file.
Ideally, I would definitely prefer not allowing libguestfs to do such operation.

Can you think of another way to fix this ?

Feel free to share your thoughts and correct me If I miss something I should know about this change, but so far this is some of my concern looking again looking the patch.

[0] - rm -f /etc/dhcp/dhclient-enter-hooks.d/resolved
[1] - apt-file search /etc/dhcp/dhclient-enter-hooks.d/resolved
systemd: /etc/dhcp/dhclient-enter-hooks.d/resolved

Here's some changes to apply to the current debdiff
- The 'Eoan' version need to be changed from "1.40.2-2ubuntu1.1" to "1.40.2-2ubuntu2". Let's use "1.40.2-2ubuntu1.1" for Disco instead.
- Your quilt patch need to be rename from "appliance-Remove-etc-dhcp-dhclient-enter-hooks.d-res.patch" "0016-appliance-Remove-etc-dhcp-dhclient-enter-hooks.d-res.patch" to stay consistent with the already existing patch numeric ordering.

Regards,
Eric

[Eric Desrochers (slashd)]

to continue my comment #16 ....

Looking further and if I understand this correctly, this piece of code seems to only operate inside supermin appliances which doesn't use systemd inside it ?

Is there any other circumstances where this piece of code can be run outside a non-systemd appliance ?

[Richard Jones (rjones-redhat)]

It doesn't delete the system file, just a copy in a private tree. This
code can never run outside the appliance.

[Ioanna Alifieraki (joalif)]

Thanks Eric for reviewing the debdiffs.

To answer your question I need to take a step back and explain how the supermin appliance
is created and used, at least to the extend I've understood.
Supermin appliances are tiny appliances similar to a virtual machines.
Libguestfs tools depend on supermin to access and modify disk images.
For example, in the test case of the bug description guestfish lunches a supermin
appliance to mount the xenial disk image and operate on it.

The patch concerns the appliance and not the host system; it removes the resolved
hook from the appliance tree.
When libguestfs package is built some packages are copied from the host (along with
their dependencies) to be included in the supermin appliance.
Although systemd is not used inside the appliance it is listed into these packages
and exists in the appliance.
This is how we end up with the resolved hook in Ubuntu.
Given that systemd is not used by the appliance removing the hook should not be
an issue.
AFAIK this hook exists only in Ubuntu, so on any other distro this patch will
try to remove a non-existing file from the appliance, it will fail silently and
execution will continue without affecting the functionality of the appliance.

I tried to do what you suggest, first checking whether we run on Ubuntu and then
remove the hook.
However, this wouldn't work because the files which tell us the distro such as
/etc/lsb-release is not present in the appliance.
The appliance copies in some files from the build host and some from the host at
runtime, however it does not copy in all of them.
It is similar to the case in LP #1650740 where the /etc/fstab file was missing.

[Ioanna Alifieraki (joalif)]

Concerning the changes to debdiff :

>- Your quilt patch need to be rename from "appliance-Remove-etc-dhcp-
>dhclient-enter-hooks.d-res.patch" "0016-appliance-Remove-etc-dhcp-
>dhclient-enter-hooks.d-res.patch" to stay consistent with the already
>existing patch numeric ordering.

Sure, you're right.

>- The 'Eoan' version need to be changed from "1.40.2-2ubuntu1.1" to
>"1.40.2-2ubuntu2". Let's use "1.40.2-2ubuntu1.1" for Disco instead.

I still struggle with package versioning so I may be wrong but currently
the Disco version is 1.40.2-1ubuntu1 .
Why should we change it to 1.40.2-2ubuntu1.1 and not 1.40.2-1ubuntu1.1?

[Eric Desrochers (slashd)]

Thanks to both of you Ioanna and Richard for your quick reply.

It addresses my concern, you would understand why I ask for clarification, this patch at first glance look unusual.

Ioanna, you are right, the version are so alike that I missed the version difference at first.

 libguestfs | 1:1.40.2-1ubuntu1 | disco/universe | source
 libguestfs | 1:1.40.2-2ubuntu1 | eoan/universe | source

I see the difference now, thanks for bringing this up.
Please don't produce a new debdiff. I'll do the necessary change manually during the sponsorship:

- Versioning:
Eoan: 1.40.2-2ubuntu2
Disco: 1.40.2-1ubuntu1.1

and

- Quilt patch renaming byt adding the numeric order.

Regards,
Eric

[Eric Desrochers (slashd)]

Sponsoring for 'E'.

Considering libguestfs maintainer and Ioanna answers to my concern that this code can never run outside the appliance and that can't create any harm inside the appliance as well.

Also trusting that Ioanna will soon file a bug and submit the patch to Debian as well.

-Eric

[Ioanna Alifieraki (joalif)]

Patch submitted to Debian :
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=929807

[Launchpad Janitor (janitor)]

This bug was fixed in the package libguestfs - 1:1.40.2-2ubuntu2

---------------
libguestfs (1:1.40.2-2ubuntu2) eoan; urgency=medium

  * d/p/0016-appliance-remove-etc-dhcp-dhclient-enter-hooks.d-res.patch:
    - Remove /etc/dhcp/dhclient-enter-hooks.d/resolved (LP: #1824236)

 -- Ioanna Alifieraki <email address hidden> Thu, 30 May 2019 12:30:29 +0100

[Brian Murray (brian-murray)]

Hello Ioanna, or anyone else affected,

Accepted libguestfs into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libguestfs/1:1.40.2-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Ioanna, or anyone else affected,

Accepted libguestfs into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libguestfs/1:1.38.4-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Ioanna, or anyone else affected,

Accepted libguestfs into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libguestfs/1:1.36.13-1ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ioanna Alifieraki (joalif)]

Verification on Disco.

Following the test case in the description :

$ sudo guestfish -a xenial-server-cloudimg-amd64-disk1.img --network -v << EOF
> run
> mount /dev/sda1 /
> command 'apt update'
> EOF
libguestfs: launch: program=guestfish
libguestfs: launch: version=1.40.2
libguestfs: launch: backend registered: unix
...
Hit:1 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Hit:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:5 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [667 kB]
Get:6 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [269 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [438 kB]
Get:8 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [178 kB]
Get:9 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [966 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [384 kB]
Get:11 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [750 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [312 kB]
Fetched 4182 kB in 11s (354 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
56 packages can be upgraded. Run 'apt list --upgradable' to see them.
...

Now 'apt update' works.

$ dpkg -l | grep libguestfs
ii libguestfs-perl 1:1.40.2-1ubuntu1.1 amd64 guest disk image management system - Perl bindings
ii libguestfs-tools 1:1.40.2-1ubuntu1.1 amd64 guest disk image management system - tools
ii libguestfs0:amd64 1:1.40.2-1ubuntu1.1 amd64 guest disk image management system - shared library

[Ioanna Alifieraki (joalif)]

Verification on Cosmic.

Following the test case in the description :

$ sudo guestfish -a xenial-server-cloudimg-amd64-disk1.img --network -v << EOF
run
mount /dev/sda1 /
command 'apt update'
EOF
libguestfs: launch: program=guestfish
libguestfs: launch: version=1.38.4
libguestfs: launch: backend registered: unix
...
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:5 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [667 kB]
Get:6 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [269 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [438 kB]
Get:8 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [178 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [5600 B]
Get:10 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7532 kB]
Get:11 http://security.ubuntu.com/ubuntu xenial-security/multiverse Translation-en [2676 B]
Get:12 http://archive.ubuntu.com/ubuntu xenial/universe Translation-en [4354 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [144 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial/multiverse Translation-en [106 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [966 kB]
Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [384 kB]
Get:17 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [7616 B]
Get:18 http://archive.ubuntu.com/ubuntu xenial-updates/restricted Translation-en [2272 B]
Get:19 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [750 kB]
Get:20 http://archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [312 kB]
Get:21 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [16.7 kB]
Get:22 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse Translation-en [8440 B]
Get:23 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [7280 B]
Get:24 http://archive.ubuntu.com/ubuntu xenial-backports/main Translation-en [4456 B]
Get:25 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [7804 B]
Get:26 http://archive.ubuntu.com/ubuntu xenial-backports/universe Translation-en [4184 B]
Fetched 16.5 MB in 28s (573 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
109 packages can be upgraded. Run 'apt list --upgradable' to see them.
...

Now 'apt update' works.

$ dpkg -l | grep libguestfs
ii libguestfs-hfsplus:amd64 1:1.38.4-1ubuntu2.1 amd64 guest disk image management system - HFS+ support
ii libguestfs-perl 1:1.38.4-1ubuntu2.1 amd64 guest disk image management system - Perl bindings
ii libguestfs-reiserfs:amd64 1:1.38.4-1ubuntu2.1 amd64 guest disk image ...

[Ioanna Alifieraki (joalif)]

Verification on Bionic.

Following the test case in the description :

$ sudo guestfish -a xenial-server-cloudimg-amd64-disk1.img --network -v << EOF
run
mount /dev/sda1 /
command 'apt update'
EOF
libguestfs: launch: program=guestfish
libguestfs: launch: version=1.36.13
libguestfs: launch: backend registered: unix
...
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:5 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [667 kB]
Get:6 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [269 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [438 kB]
Get:8 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [178 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [5600 B]
Get:10 http://security.ubuntu.com/ubuntu xenial-security/multiverse Translation-en [2676 B]
Get:11 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7532 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial/universe Translation-en [4354 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [144 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial/multiverse Translation-en [106 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [966 kB]
Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [384 kB]
Get:17 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [7616 B]
Get:18 http://archive.ubuntu.com/ubuntu xenial-updates/restricted Translation-en [2272 B]
Get:19 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [750 kB]
Get:20 http://archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [312 kB]
Get:21 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [16.7 kB]
Get:22 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse Translation-en [8440 B]
Get:23 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [7280 B]
Get:24 http://archive.ubuntu.com/ubuntu xenial-backports/main Translation-en [4456 B]
Get:25 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [7804 B]
Get:26 http://archive.ubuntu.com/ubuntu xenial-backports/universe Translation-en [4184 B]
Fetched 16.5 MB in 32s (504 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
109 packages can be upgraded. Run 'apt list --upgradable' to see them.
...

Now 'apt update' works.

$ dpkg -l | grep libguestfs
ii libguestfs-hfsplus:amd64 1:1.36.13-1ubuntu3.3 amd64 guest disk image management system - HFS+ support
ii libguestfs-perl 1:1.36.13-1ubuntu3.3 amd64 guest disk image management system - Perl bindings
ii libguestfs-reiserfs:amd64 1:1.36.13-1ubuntu3.3 amd64 guest disk image management system - ReiserFS support
i...

[Launchpad Janitor (janitor)]

This bug was fixed in the package libguestfs - 1:1.40.2-1ubuntu1.1

---------------
libguestfs (1:1.40.2-1ubuntu1.1) disco; urgency=medium

  * d/p/appliance-Remove-etc-dhcp-dhclient-enter-hooks.d-res.patch:
    - remove /etc/dhcp/dhclient-enter-hooks.d/resolved (LP: #1824236)

 -- Ioanna Alifieraki <email address hidden> Thu, 30 May 2019 11:53:30 +0000

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for libguestfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libguestfs - 1:1.38.4-1ubuntu2.1

---------------
libguestfs (1:1.38.4-1ubuntu2.1) cosmic; urgency=medium

  * d/p/appliance-Remove-etc-dhcp-dhclient-enter-hooks.d-res.patch:
    - remove /etc/dhcp/dhclient-enter-hooks.d/resolved (LP: #1824236)

 -- Ioanna Alifieraki <email address hidden> Thu, 30 May 2019 12:03:15 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package libguestfs - 1:1.36.13-1ubuntu3.3

---------------
libguestfs (1:1.36.13-1ubuntu3.3) bionic; urgency=medium

  * d/p/appliance-Remove-etc-dhcp-dhclient-enter-hooks.d-res.patch:
    - remove /etc/dhcp/dhclient-enter-hooks.d/resolved (LP: #1824236)

 -- Ioanna Alifieraki <email address hidden> Thu, 30 May 2019 13:07:37 +0000