make the service fail gracefully if unable to load modules

Bug #1865037 reported by Christian Ehrhardt 
26
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Triaged
High
Unassigned
Ussuri
Fix Committed
Undecided
Unassigned
rtslib-fb
Unknown
Unknown
python-rtslib-fb (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
Undecided
Unassigned
Groovy
Fix Released
Undecided
Unassigned

Bug Description

[Impact]
========

The LIO interfaces is inherently tied to the kernel.
That makes the service fail on e.g. a container install:

See "systemctl status rtslib-fb-targetctl.service" and "journalctl -xe" for details.
root@f:~# systemctl status rtslib-fb-targetctl.service
● rtslib-fb-targetctl.service - Restore LIO kernel target configuration
     Loaded: loaded (/lib/systemd/system/rtslib-fb-targetctl.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2020-02-27 14:10:35 UTC; 3s ago
    Process: 23831 ExecStart=/usr/bin/mkdir -p /etc/rtslib-fb-target (code=exited, status=0/SUCCESS)
    Process: 23832 ExecStart=/usr/bin/targetctl restore (code=exited, status=1/FAILURE)
   Main PID: 23832 (code=exited, status=1/FAILURE)

Feb 27 14:10:35 f target[23832]: File "/usr/bin/targetctl", line 47, in restore
Feb 27 14:10:35 f target[23832]: errors = RTSRoot().restore_from_file(restore_file=from_file)
Feb 27 14:10:35 f target[23832]: File "/usr/lib/python3/dist-packages/rtslib_fb/root.py", line 85, in __init__
Feb 27 14:10:35 f target[23832]: modprobe('target_core_mod')
Feb 27 14:10:35 f target[23832]: File "/usr/lib/python3/dist-packages/rtslib_fb/utils.py", line 428, in modprobe
Feb 27 14:10:35 f target[23832]: raise RTSLibError(stderrdata)
Feb 27 14:10:35 f target[23832]: rtslib_fb.utils.RTSLibError: b"modprobe: ERROR: ../libkmod/libkmod.c:611 kmod_search_moddep() could not open moddep file '/lib/modules/5.3.0-40-generic/modu>
Feb 27 14:10:35 f systemd[1]: rtslib-fb-targetctl.service: Main process exited, code=exited, status=1/FAILURE
Feb 27 14:10:35 f systemd[1]: rtslib-fb-targetctl.service: Failed with result 'exit-code'.
Feb 27 14:10:35 f systemd[1]: Failed to start Restore LIO kernel target configuration.

It is ok that this doesn't work in a container, but it also breaks the package installation status which should be avoided.

[Test Plan]
===========

To reproduce this bug, simply do the following:

$ lxc launch ubuntu-daily:groovy python-rtslib-fb-lp1865037-groovy

$ lxc shell python-rtslib-fb-lp1865037-groovy

# apt update && apt upgrade

# reboot

# apt install python3-rtslib-fb

...and this should fail to install.

To make sure that this bug is indeed fixed, install the patched version of this package and that should install fine.

[Where problems could occur]
============================

This is a workaround since there doesn't seem to be a in-container use case. But in case there is one, then that'd fail to work, so we might better find a better solution for this in the long term.

The disccusion has been initiated upstream (cf: https://github.com/open-iscsi/rtslib-fb/issues/157) and probably we'll wait for upstream to state some in-container use case(s).

Related branches

summary: - make the init script fail gracefully if unable to load modules
+ make the service fail gracefully if unable to load modules
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Filed upstream as I'd want their input on how to best solve this.
=> https://github.com/open-iscsi/rtslib-fb/issues/157

FYI: the debian init script is a copy of the one from upstream (which is always dangerous as they can get out of sync, I much prefer copying them in d/rules to stay with upstream - we might include that when suggesting to Debian)

Subscribing and assigning Rafael who is driving this for HA.

Changed in python-rtslib-fb (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → Rafael David Tinoco (rafaeldtinoco)
Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

I have suggest medium sized changes to the package with its latest upstream version update:

    Suggesting:
    https://salsa.debian.org/openstack-team/python/python-rtslib-fb/-/merge_requests/2
    https://salsa.debian.org/openstack-team/python/python-rtslib-fb/-/merge_requests/3

And service is disabled by default... together with the suggestion for targetcli:

    Suggesting:
    - https://salsa.debian.org/linux-blocks-team/targetcli-fb/-/merge_requests/8

Let's see what upstream (Debian) says about those)

Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

CURRENT STATUS

Targets:

- I: ceph-iscsi

- II: targetcli-fb

Summary for the current state:

I: ceph-iscsi
    [-] MIR ack (if everything else is)

I/II: python-configshell-fb
    [.] MIR ack
    [.] Security ack - needs DEP8 inclusion

    Suggesting:
    - https://salsa.debian.org/linux-blocks-team/python-configshell-fb/-/merge_requests/3/

II: targetcli-fb
    [.] MIR ack - needs DEP8 and services/sockets fix
    [.] Security nack->ack - 2 upstream fixes done

    Suggesting:
    - https://salsa.debian.org/linux-blocks-team/targetcli-fb/-/merge_requests/8

I/II: python-rtslib-fb
    [.] MIR ack - needs packaging/lintian fixes
    [.] Security ack - upstream fix done

    - Upstream version v2.1.73 incudes the security fix.

    Suggesting:
    https://salsa.debian.org/openstack-team/python/python-rtslib-fb/-/merge_requests/2
    https://salsa.debian.org/openstack-team/python/python-rtslib-fb/-/merge_requests/3

    - Might not need fix for LP: #1865037 (put a comment, waiting Debian)
    - Highlighted @jamespage in both MR in salsa (hopefully he can help)

I: tcmu
    [.] MIR ack
    [+] Security ack - but needs dbus fix

I: urwid
    [.] MIR ack
    [.] Security: ack

Only thing missing: tcmu fix and wait changes to be accepted so we can put all those packages on sync (and then do the MIR if everybody is on the same page).

Changed in python-rtslib-fb (Ubuntu):
assignee: Rafael David Tinoco (rafaeldtinoco) → nobody
Revision history for this message
James Page (james-page) wrote :

Bumping to high as the switch in cinder to use this iSCSI stack is now failing autopkgtests - can we get this into the hirsute package please?

Changed in python-rtslib-fb (Ubuntu):
importance: Medium → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-rtslib-fb - 2.1.74-0ubuntu3

---------------
python-rtslib-fb (2.1.74-0ubuntu3) hirsute; urgency=medium

  * d/python3-rtslib-fb.rtslib-fb-targetctl.service: Add check to skip
     starting this service when running in a container (LP: #1865037).

 -- Chris MacNaughton <email address hidden> Fri, 05 Mar 2021 16:26:14 +0000

Changed in python-rtslib-fb (Ubuntu):
status: Triaged → Fix Released
no longer affects: python-rtslib-fb (Ubuntu Groovy)
Utkarsh Gupta (utkarsh)
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Christian, or anyone else affected,

Accepted python-rtslib-fb into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-rtslib-fb/2.1.73-1ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-rtslib-fb (Ubuntu Groovy):
status: New → Fix Committed
tags: added: verification-needed verification-needed-groovy
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Before - Tried to start and fail in a container

root@python-rtslib-fb-lp1865037-groovy:~# systemctl status target.service
● target.service - Restore LIO kernel target configuration
     Loaded: loaded (/lib/systemd/system/target.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
root@python-rtslib-fb-lp1865037-groovy:~# systemctl status rtslib-fb-targetctl
● rtslib-fb-targetctl.service - LSB: Start LIO targets
     Loaded: loaded (/etc/init.d/rtslib-fb-targetctl; generated)
     Active: active (exited) since Wed 2021-04-28 05:28:43 UTC; 45s ago
       Docs: man:systemd-sysv-generator(8)
      Tasks: 0 (limit: 38268)
     Memory: 0B
     CGroup: /system.slice/rtslib-fb-targetctl.service

Apr 28 05:28:43 python-rtslib-fb-lp1865037-groovy systemd[1]: Starting LSB: Start LIO targets...
Apr 28 05:28:43 python-rtslib-fb-lp1865037-groovy rtslib-fb-targetctl[710]: modprobe: FATAL: Module configfs not found in directory /lib/modules/5.4.0-72-generic
Apr 28 05:28:43 python-rtslib-fb-lp1865037-groovy rtslib-fb-targetctl[704]: Could not load configfs module: exiting!
Apr 28 05:28:43 python-rtslib-fb-lp1865037-groovy systemd[1]: Started LSB: Start LIO targets.

After:
- proper service file (in case one wants to modify)
- properly dtecting
root@python-rtslib-fb-lp1865037-groovy:~# systemctl status target.service
● target.service - Restore LIO kernel target configuration
     Loaded: loaded (/lib/systemd/system/target.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
root@python-rtslib-fb-lp1865037-groovy:~# systemctl status rtslib-fb-targetctl
● rtslib-fb-targetctl.service - Restore LIO kernel target configuration
     Loaded: loaded (/lib/systemd/system/rtslib-fb-targetctl.service; enabled; vendor preset: enabled)
     Active: inactive (dead)

Apr 28 05:33:29 python-rtslib-fb-lp1865037-groovy systemd[1]: Condition check resulted in Restore LIO kernel target configuration being skipped.

=> verified

tags: added: verification-done verification-done-groovy
removed: verification-needed verification-needed-groovy
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

...also always remember to note the version number used in testing!

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for python-rtslib-fb has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-rtslib-fb - 2.1.73-1ubuntu4.1

---------------
python-rtslib-fb (2.1.73-1ubuntu4.1) groovy; urgency=medium

  * Fix service fail on install due to name mismatch and lost mkdir
    (LP: #1916715)
    - d/python3-rtslib-fb.rtslib-fb-targetctl.service: add back debianized
      systemd service file.
    - d/python3-rtslib-fb.rtslib-fb-targetctl.service: Add check to skip
      starting this service when running in a container (LP: #1865037).
    - d/rules: install rtslib-fb-targetctl.service as it was in Focal.

 -- Utkarsh Gupta <email address hidden> Fri, 12 Mar 2021 15:52:34 +0530

Changed in python-rtslib-fb (Ubuntu Groovy):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in python-rtslib-fb (Ubuntu Focal):
status: New → Confirmed
Revision history for this message
Billy Olsen (billy-olsen) wrote :

This bug is also a problem on focal. When trying to install in a container, the service fails to start with:

invoke-rc.d: initscript rtslib-fb-targetctl, action "start" failed.
● rtslib-fb-targetctl.service - Restore LIO kernel target configuration
     Loaded: loaded (/lib/systemd/system/rtslib-fb-targetctl.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2021-06-24 18:08:27 UTC; 13ms ago
    Process: 1232 ExecStart=/usr/bin/mkdir -p /etc/rtslib-fb-target (code=exited, status=0/SUCCESS)
    Process: 1233 ExecStart=/usr/bin/targetctl restore (code=exited, status=1/FAILURE)
   Main PID: 1233 (code=exited, status=1/FAILURE)

Jun 24 18:08:27 closing-anemone target[1233]: File "/usr/bin/targetctl", line 47, in restore
Jun 24 18:08:27 closing-anemone target[1233]: errors = RTSRoot().restore_from_file(restore_file=from_file)
Jun 24 18:08:27 closing-anemone target[1233]: File "/usr/lib/python3/dist-packages/rtslib_fb/root.py", line 85, in __init__
Jun 24 18:08:27 closing-anemone target[1233]: modprobe('target_core_mod')
Jun 24 18:08:27 closing-anemone target[1233]: File "/usr/lib/python3/dist-packages/rtslib_fb/utils.py", line 428, in modprobe
Jun 24 18:08:27 closing-anemone target[1233]: raise RTSLibError(stderrdata)
Jun 24 18:08:27 closing-anemone target[1233]: rtslib_fb.utils.RTSLibError: b'modprobe: FATAL: Module target_core_mod not found in directory /lib/modules/5.8.0-55-generic\n'
Jun 24 18:08:27 closing-anemone systemd[1]: rtslib-fb-targetctl.service: Main process exited, code=exited, status=1/FAILURE
Jun 24 18:08:27 closing-anemone systemd[1]: rtslib-fb-targetctl.service: Failed with result 'exit-code'.
Jun 24 18:08:27 closing-anemone systemd[1]: Failed to start Restore LIO kernel target configuration.
dpkg: error processing package python3-rtslib-fb (--configure):
 installed python3-rtslib-fb package post-installation script subprocess returned error exit status 1
Processing triggers for systemd (2dmesg: read kernel buffer failed: Operation not permitted

I can confirm that the ConditionVirtualization=!container directive added to the service file addresses the issue.

Changed in cloud-archive:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Utkarsh Gupta (utkarsh) wrote :

Hi Billy,

Thanks! I'll prep the update in the coming days.

Changed in python-rtslib-fb (Ubuntu Focal):
assignee: nobody → Utkarsh Gupta (utkarsh)
Revision history for this message
Billy Olsen (billy-olsen) wrote :

Thanks Utkarsh!

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Just a warning, the debdiff is wrong 2.1.71-0ubuntu2 needs to be 2.1.71-0ubuntu1.1

Revision history for this message
Billy Olsen (billy-olsen) wrote :

ah thanks for the tip there Christian (that the version number is wrong). Is this because of where the change lies or ? Wanting to make sure I understand for future patches.

Revision history for this message
Utkarsh Gupta (utkarsh) wrote :

Hi Billy,

From point 4.1 of https://wiki.ubuntu.com/StableReleaseUpdates#Procedure:
> The version number does not conflict with any later and future
> version in other Ubuntu releases (the security policy document
> has a well-working scheme which can be used for SRUs.)

And the security policy document reference leads here:
https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging

Let me know if the versioning is still not clear.

Revision history for this message
Billy Olsen (billy-olsen) wrote :

Hi Utkarsh - it is indeed more clear. Thanks for the links. I'm deleting the previous patch and providing a new one with updated versioning.

Revision history for this message
Billy Olsen (billy-olsen) wrote :
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Christian, or anyone else affected,

Accepted python-rtslib-fb into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-rtslib-fb/2.1.71-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-rtslib-fb (Ubuntu Focal):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-focal
removed: verification-done
Revision history for this message
Billy Olsen (billy-olsen) wrote :

Verified this package for focal.

First, create focal container
Second, apt-get install python3-rtslib-fb

Fails with:
dpkg: error processing package python3-rtslib-fb (--configure):
 installed python3-rtslib-fb package post-installation script subprocess returned error exit status 1
dmesg: read kernel buffer failed: Operation not permitted
                                                         Errors were encountered while processing:
 python3-rtslib-fb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Next, install proposed pocket, apt-get update.
sudo apt-get install python3-rtslib-fb completes successfully:
...
Unpacking python3-rtslib-fb (2.1.71-0ubuntu1.1) ...
Setting up python3-rtslib-fb (2.1.71-0ubuntu1.1) ...
Processing triggers for systemd (245.4-4ubuntu3.6) ...

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Changed in python-rtslib-fb (Ubuntu Focal):
assignee: Utkarsh Gupta (utkarsh) → nobody
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Christian, or anyone else affected,

Accepted python-rtslib-fb into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-ussuri-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-rtslib-fb - 2.1.71-0ubuntu1.1

---------------
python-rtslib-fb (2.1.71-0ubuntu1.1) focal; urgency=medium

  * d/python3-rtslib-fb.rtslib-fb-targetctl.service: Add check
    to skip starting this service when running in a container.
    (LP: #1865037)

 -- Utkarsh Gupta <email address hidden> Fri, 25 Jun 2021 18:31:16 +0530

Changed in python-rtslib-fb (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.