Apt lock race on Oracle

Bug #1944611 reported by James Falcon
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
High
James Falcon

Bug Description

When trying to do any kind of apt operation via cloud-init on Oracle, the operation often fails because snap seeding is holding the apt lock:

cloud-init.log:
2021-06-15 21:32:31,809 - util.py[WARNING]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed
2021-06-15 21:32:31,809 - util.py[DEBUG]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 885, in _run_modules
    freq=freq)
  File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 186, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py", line 161, in handle
    install_drivers(cfg['drivers'], cloud.distro.install_packages)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py", line 115, in install_drivers
    pkg_install_func(['ubuntu-drivers-common'])
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 111, in install_packages
    self.update_package_sources()
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 206, in update_package_sources
    ["update"], freq=PER_INSTANCE)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 186, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 202, in package_command
    args=(cmd,), kwargs={'env': e, 'capture': False})
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2348, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 295, in subp
    cmd=args)
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update']

cloud-init-output.log:
Reading package lists...
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/
Cloud-init v. 21.1-19-gbad84ad4-0ubuntu1~18.04.2 running 'modules:final' at Tue, 15 Jun 2021 21:32:29 +0000. Up 36.68 seconds.
2021-06-15 21:32:31,809 - util.py[WARNING]: Running module ubuntu-drivers (<module 'cloudinit.config.cc_ubuntu_drivers' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ubuntu_drivers.py'>) failed

apt/history.log:
Start-Date: 2021-06-15 21:32:39
Commandline: apt install -o Dpkg::Options::=--force-confold -y dpkg-sig
Requested-By: snap_daemon (584788)
Install: libconfig-file-perl:amd64 (1.50-3, automatic), dpkg-sig:amd64 (0.13.1+nmu4)
End-Date: 2021-06-15 21:32:40

I've also seen another snap entry in a separate log:
Start-Date: 2021-09-21 21:35:53
Commandline: apt --allow-downgrades install -o Dpkg::Options::=--force-confold -y /var/lib/oracle-cloud-agent/plugins/unifiedmonitoring/temp-unified-monitoring.deb
Requested-By: snap_daemon (584788)
Install: unified-monitoring-agent:amd64 (2.5.8-0)
End-Date: 2021-09-21 21:36:12

This is most easily observed when launching an instance with the following cloud-config:
#cloud-config
drivers:
  nvidia:
    license-accepted: true

James Falcon (falcojr)
Changed in cloud-init:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Chad Smith (chad.smith) wrote :

upstream PR in progress on this bug https://github.com/canonical/cloud-init/pull/1034

Changed in cloud-init:
status: Triaged → In Progress
assignee: nobody → James Falcon (falcojr)
Revision history for this message
Chad Smith (chad.smith) wrote :
Changed in cloud-init:
status: In Progress → Fix Committed
Revision history for this message
Fabio Augusto Miranda Martins (fabio.martins) wrote :

Hi Chad / James,

Given that the commit has been merged upstream, do we have an update on SRU'ing it into Ubuntu cloud-init packages?

Regards,
Fabio Martins

Revision history for this message
James Falcon (falcojr) wrote :

Hey Fabio, based on https://discourse.ubuntu.com/t/cloud-init-2022-release-schedule/25413 , the next upstream release is scheduled for Feb 15th which will include this change. We then plan to SRU the upstream release immediately afterwards.

Revision history for this message
Fabio Augusto Miranda Martins (fabio.martins) wrote :

Thank you very much for the update.

Revision history for this message
Brett Holman (holmanb) wrote : Fixed in cloud-init version 22.1.

This bug is believed to be fixed in cloud-init in version 22.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
Revision history for this message
James Falcon (falcojr) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.