While upgrading cinder with openstack-origin=cloud:bionic-ussuri, after upgrading from bionic:distro, bionic-rocky, bionic-stein, bionic-train, I've run into an issue with the upgrade/install of the package python3-rtslib-fb. The configure script of the package tries to start the service rtslib-fb-targetctl, which has a "Requires=sys-kernel-config.mount" in it's service definition.
This mount is not present in the cinder container.
Looking at some related bug: https://bugs.launchpad.net/ubuntu/+source/python-rtslib-fb/+bug/1916715
It seems that the module configfs is needed for this mountpoint to exist.
My container's kernel is 4.15.0-76-generic.
cinder-1: 18:20:59 INFO unit.cinder/1.juju-log [cinder] Performing OpenStack upgrade to ussuri.
unit-cinder-1: 18:20:59 INFO unit.cinder/1.juju-log DEPRECATION WARNING: Function configure_installation_source is being removed on/around 2017-07 : use charmhelpers.fetch.add_source() instead.
unit-cinder-1: 18:21:00 INFO unit.cinder/1.juju-log Installing [] with options: ['--option=Dpkg::Options::=--force-confold']
unit-cinder-1: 18:21:00 DEBUG unit.cinder/1.config-changed Reading package lists...
unit-cinder-1: 18:21:00 DEBUG unit.cinder/1.config-changed Building dependency tree...
unit-cinder-1: 18:21:00 DEBUG unit.cinder/1.config-changed Reading state information...
unit-cinder-1: 18:21:00 DEBUG unit.cinder/1.config-changed The following packages were automatically installed and are no longer required:
unit-cinder-1: 18:21:00 DEBUG unit.cinder/1.config-changed python-pyudev python-rtslib-fb python3-ceilometerclient python3-contextlib2
unit-cinder-1: 18:21:00 DEBUG unit.cinder/1.config-changed Use 'apt autoremove' to remove them.
unit-cinder-1: 18:21:00 DEBUG unit.cinder/1.config-changed 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
unit-cinder-1: 18:21:00 DEBUG unit.cinder/1.config-changed 1 not fully installed or removed.
unit-cinder-1: 18:21:00 DEBUG unit.cinder/1.config-changed After this operation, 0 B of additional disk space will be used.
unit-cinder-1: 18:21:00 DEBUG unit.cinder/1.config-changed Setting up python3-rtslib-fb (2.1.66-3.2~cloud0) ...
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed A dependency job for rtslib-fb-targetctl.service failed. See 'journalctl -xe' for details.
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed invoke-rc.d: initscript rtslib-fb-targetctl, action "start" failed.
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed ● rtslib-fb-targetctl.service - Restore LIO kernel target configuration
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed Loaded: loaded (/lib/systemd/system/rtslib-fb-targetctl.service; disabled; vendor preset: enabled)
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed Active: inactive (dead)
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed Jun 01 18:15:17 juju-bf8473-19-lxd-2 systemd[1]: Dependency failed for Restore LIO kernel target configuration.
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed Jun 01 18:15:17 juju-bf8473-19-lxd-2 systemd[1]: rtslib-fb-targetctl.service: Job rtslib-fb-targetctl.service/start failed with result 'dependency'.
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed Jun 01 18:15:30 juju-bf8473-19-lxd-2 systemd[1]: Dependency failed for Restore LIO kernel target configuration.
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed Jun 01 18:15:30 juju-bf8473-19-lxd-2 systemd[1]: rtslib-fb-targetctl.service: Job rtslib-fb-targetctl.service/start failed with result 'dependency'.
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed Jun 01 18:15:43 juju-bf8473-19-lxd-2 systemd[1]: Dependency failed for Restore LIO kernel target configuration.
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed Jun 01 18:15:43 juju-bf8473-19-lxd-2 systemd[1]: rtslib-fb-targetctl.service: Job rtslib-fb-targetctl.service/start failed with result 'dependency'.
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed Jun 01 18:15:55 juju-bf8473-19-lxd-2 systemd[1]: Dependency failed for Restore LIO kernel target configuration.
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed Jun 01 18:15:55 juju-bf8473-19-lxd-2 systemd[1]: rtslib-fb-targetctl.service: Job rtslib-fb-targetctl.service/start failed with result 'dependency'.
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed Jun 01 18:21:01 juju-bf8473-19-lxd-2 systemd[1]: Dependency failed for Restore LIO kernel target configuration.
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed Jun 01 18:21:01 juju-bf8473-19-lxd-2 systemd[1]: rtslib-fb-targetctl.service: Job rtslib-fb-targetctl.service/start failed with result 'dependency'.
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed dpkg: error processing package python3-rtslib-fb (--configure):
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed installed python3-rtslib-fb package post-installation script subprocess returned error exit status 1
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed Errors were encountered while processing:
unit-cinder-1: 18:21:01 DEBUG unit.cinder/1.config-changed python3-rtslib-fb
unit-cinder-1: 18:21:02 WARNING unit.cinder/1.config-changed E: Sub-process /usr/bin/dpkg returned an error code (1)
I tried modprobing configfs on the metal, then restarting the container, but I am still seeing failure to mount, permission denied.
ubuntu@juju-bf8473-18-lxd-2:/sys/kernel/config$ sudo systemctl start sys-kernel-config.mount
Job for sys-kernel-config.mount failed.
See "systemctl status sys-kernel-config.mount" and "journalctl -xe" for details.
ubuntu@juju-bf8473-18-lxd-2:/sys/kernel/config$ journalctl -xe
-- The result is RESULT.
Jun 01 18:31:45 juju-bf8473-18-lxd-2 systemd[1]: Dependency failed for Restore LIO kernel target configuration.
-- Subject: Unit rtslib-fb-targetctl.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit rtslib-fb-targetctl.service has failed.
--
-- The result is RESULT.
Jun 01 18:31:45 juju-bf8473-18-lxd-2 systemd[1]: rtslib-fb-targetctl.service: Job rtslib-fb-targetctl.service/start failed with result 'dependency'.
Jun 01 18:31:49 juju-bf8473-18-lxd-2 sudo[4421]: ubuntu : TTY=pts/0 ; PWD=/sys/kernel/config ; USER=root ; COMMAND=/bin/systemctl start sys-kernel-config.mount
Jun 01 18:31:49 juju-bf8473-18-lxd-2 sudo[4421]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
Jun 01 18:31:49 juju-bf8473-18-lxd-2 systemd[1]: sys-kernel-config.mount: Failed to reset devices.list: Operation not permitted
Jun 01 18:31:49 juju-bf8473-18-lxd-2 systemd[1]: Mounting Kernel Configuration File System...
-- Subject: Unit sys-kernel-config.mount has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit sys-kernel-config.mount has begun starting up.
Jun 01 18:31:49 juju-bf8473-18-lxd-2 mount[4426]: mount: /sys/kernel/config: permission denied.
Jun 01 18:31:49 juju-bf8473-18-lxd-2 systemd[1]: sys-kernel-config.mount: Mount process exited, code=exited status=32
Jun 01 18:31:49 juju-bf8473-18-lxd-2 systemd[1]: sys-kernel-config.mount: Failed with result 'exit-code'.
Jun 01 18:31:49 juju-bf8473-18-lxd-2 systemd[1]: Failed to mount Kernel Configuration File System.
-- Subject: Unit sys-kernel-config.mount has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit sys-kernel-config.mount has failed.
--
-- The result is RESULT.
Jun 01 18:31:49 juju-bf8473-18-lxd-2 sudo[4421]: pam_unix(sudo:session): session closed for user root
As my installation is not using iSCSI, I've removed the package on the cinder units and re-triggered the config-changed hook with juju resolved.
juju run -a cinder 'apt-get remove -y python3-rtslib-fb'
juju resolved cinder/0
juju resolved cinder/1
juju resolved cinder/2