rbd-target-api crashes with python TypeError

Bug #1883112 reported by Julian Jacobsen
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
ceph-iscsi (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
High
Unassigned
Impish
Fix Released
High
Unassigned

Bug Description

[Impact]

 * rbd-target-api service fails to start if there is a blocklist
   entry for the unit making the service unavailable.

 * When the rbd-target-api service starts it checks if any of the
   ip addresses on the machine it is running on are listed as
   blocked. If there are entries it tries to remove them. In the
   process of removing the entries the code attempts to test whether
   a string is in the result of a subprocess.check_output call. This
   would have worked in python2 but with python3 a byte like object
   is returned and check now throws a TypeError. This fix, taken from
   upstream, changes the code to remove the `in` check and replace it
   with a try/except

[Test Plan]

 If an existing ceph-iscsi deployment is available then skip to
 step 3.

 1) Deploy the bundle below (tested with OpenStack providor).

series: focal
applications:
  ceph-iscsi:
    charm: cs:ceph-iscsi
    num_units: 2
  ceph-osd:
    charm: ch:ceph-osd
    num_units: 3
    storage:
      osd-devices: 'cinder,10G'
    options:
      osd-devices: '/dev/test-non-existent'
    channel: latest/edge
  ceph-mon:
    charm: ch:ceph-mon
    num_units: 3
    options:
      monitor-count: '3'
    channel: latest/edge
relations:
  - - 'ceph-mon:client'
    - 'ceph-iscsi:ceph-client'
  - - 'ceph-osd:mon'
    - 'ceph-mon:osd'

 2) Connect to ceph-iscsi unit:

juju ssh -m zaza-a1d88053ab85 ceph-iscsi/0

 3) Stop rbd-target-api via systemd to make test case clearer:

sudo systemctl stop rbd-target-api

 4) Add 2 blocklist entries for this unit (due to another issue the ordering of the output from `osd blacklist ls` matters which can lead to the reproduction of this bug being intermittent. To avoid this add two entries which ensures there is always an entry for this node in the list of blocklist entries to be removed).

sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/1
sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/2
sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist ls
  listed 2 entries
  172.20.0.135:0/2 2022-02-23T11:14:54.850352+0000
  172.20.0.135:0/1 2022-02-23T11:14:52.502592+0000

 5) Attempt to start service:

sudo /usr/bin/python3 /usr/bin/rbd-target-api
Traceback (most recent call last):
  File "/usr/bin/rbd-target-api", line 2952, in <module>
    main()
  File "/usr/bin/rbd-target-api", line 2862, in main
    osd_state_ok = ceph_gw.osd_blacklist_cleanup()
  File "/usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py", line 111, in osd_blacklist_cleanup
    rm_ok = self.ceph_rm_blacklist(blacklist_entry.split(' ')[0])
  File "/usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py", line 46, in ceph_rm_blacklist
    if ("un-blacklisting" in result) or ("isn't blacklisted" in result):
TypeError: a bytes-like object is required, not 'str'

[Where problems could occur]

 * Problems could occur with the service starting as this blocklist check is done at startup.

 * Blocklist entries could fail to be removed.

Old bug description:

$ lsb_release -rd
Description: Ubuntu 20.04 LTS
Release: 20.04

$ dpkg -S /usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py
ceph-iscsi: /usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py

$ apt-cache policy ceph-iscsi
ceph-iscsi:
  Installed: 3.4-0ubuntu2
  Candidate: 3.4-0ubuntu2
  Version table:
 *** 3.4-0ubuntu2 500
        500 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 Packages
        500 http://de.archive.ubuntu.com/ubuntu focal/universe i386 Packages
        100 /var/lib/dpkg/status

On second startup after a reboot, rbd-target-api crashes with a TypeError:

Traceback (most recent call last):
  File "/usr/bin/rbd-target-api", line 2952, in <module>
    main()
  File "/usr/bin/rbd-target-api", line 2862, in main
    osd_state_ok = ceph_gw.osd_blacklist_cleanup()
  File "/usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py", line 110, in osd_blacklist_cleanup
    rm_ok = self.ceph_rm_blacklist(blacklist_entry.split(' ')[0])
  File "/usr/lib/python3/dist-packages/ceph_iscsi_config/gateway.py", line 46, in ceph_rm_blacklist
    if ("un-blacklisting" in result) or ("isn't blacklisted" in result):
TypeError: a bytes-like object is required, not 'str'

Upstream has a fix: https://github.com/ceph/ceph-iscsi/commit/e48dcb0d3099b27595b9f4433da8493f5edb9206#diff-f1381af4114a1e777ef5e8b7b7452a01

summary: - rbd-target-api crashes with python TypError
+ rbd-target-api crashes with python TypeError
tags: added: focal
Liam Young (gnuoy)
Changed in ceph-iscsi (Ubuntu):
status: New → Confirmed
Revision history for this message
Liam Young (gnuoy) wrote :
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "deb.diff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Liam Young (gnuoy)
description: updated
James Page (james-page)
Changed in ceph-iscsi (Ubuntu):
status: Confirmed → Fix Released
Changed in ceph-iscsi (Ubuntu Focal):
status: New → Triaged
Changed in ceph-iscsi (Ubuntu Impish):
status: New → Triaged
Changed in ceph-iscsi (Ubuntu Focal):
importance: Undecided → High
Changed in ceph-iscsi (Ubuntu Impish):
importance: Undecided → High
Revision history for this message
James Page (james-page) wrote (last edit ):

Updates for impish and focal uploaded to the UNAPPROVED queues for SRU team review.

Thanks Liam and Julian!

Revision history for this message
Robie Basak (racb) wrote :

The obvious minimal fix would be something like:

- if ("un-blacklisting" in result) or ("isn't blacklisted" in result):
+ if (b"un-blacklisting" in result) or (b"isn't blacklisted" in result):

...since in Ubuntu Focal and Impish we know that we're running on Python 3, so 2+3 compatibility isn't required.

Cherry-picking the upstream change is additionally making new assumptions about the behaviour of ceph's CLI I think - that it returns the correct exit status. Have you checked to ensure that the behaviour of the version of ceph in Focal and Impish is as expected in both the success and failure cases? If so then since presumably this code never worked on Focal and Impish before, then the cherry-pick is fine. But if you want to use the upstream (non-minimal) fix that makes this additional assumption then think it needs to be confirmed before accepting.

For reference, SRU policy says the following. In this case I think either way is fine providing that assumptions are checked.

> In line with this, the requirements for stable updates are not necessarily the same as those in the development release. When preparing future releases, one of our goals is to construct the most elegant and maintainable system possible, and this often involves fundamental improvements to the system's architecture, rearranging packages to avoid bundled copies of other software so that we only have to maintain it in one place, and so on. However, once we have completed a release, the priority is normally to minimise risk caused by changes not explicitly required to fix qualifying bugs, and this tends to be well-correlated with minimising the size of those changes. As such, the same bug may need to be fixed in different ways in stable and development releases.

Revision history for this message
Brian Murray (brian-murray) wrote :

I'm setting this to Incomplete due to the questions Robie has about the cli interface in Impish and Focal.

Changed in ceph-iscsi (Ubuntu Impish):
status: Triaged → Incomplete
Revision history for this message
Liam Young (gnuoy) wrote :

Thank you for the update Robie. I proposed the deb diff based on the fix that had landed upstream because I (wrongly) thought that was what the SRU policy required. I think it makes more sense to go for the minimal fix you suggest.

Revision history for this message
Liam Young (gnuoy) wrote :
Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello Julian, or anyone else affected,

Accepted ceph-iscsi into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph-iscsi/3.4-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ceph-iscsi (Ubuntu Impish):
status: Incomplete → Fix Committed
tags: added: verification-needed verification-needed-impish
Changed in ceph-iscsi (Ubuntu Focal):
status: Triaged → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Robie Basak (racb) wrote :

Hello Julian, or anyone else affected,

Accepted ceph-iscsi into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph-iscsi/3.4-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Liam Young (gnuoy) wrote :

Tested successfully on focal with 3.4-0ubuntu2.1

Tested with ceph-iscsi charms functional tests which were previously failing.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal

$ apt-cache policy ceph-iscsi
ceph-iscsi:
  Installed: 3.4-0ubuntu2.1
  Candidate: 3.4-0ubuntu2.1
  Version table:
 *** 3.4-0ubuntu2.1 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     3.4-0ubuntu2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/universe amd64 Packages

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Liam Young (gnuoy) wrote :
Revision history for this message
Brian Murray (brian-murray) wrote :

What is the plan for getting this fixed in Impish then? While its unlikely people will upgrade from Focal to Impish at this point in time will they encounter a regression because this is unfixed there?

Changed in ceph-iscsi (Ubuntu Focal):
status: Fix Committed → Incomplete
Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :
Download full text (3.2 KiB)

I've performed verification on impish with the reproducer steps and confirmed that the proposed fix resolves this issue.

ubuntu@juju-2e9148-rbd-mirror-0:~$ apt-cache policy ceph-iscsi
ceph-iscsi:
  Installed: 3.4-1ubuntu1.1
  Candidate: 3.4-1ubuntu1.1
  Version table:
 *** 3.4-1ubuntu1.1 500
        500 http://archive.ubuntu.com/ubuntu impish-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     3.4-1ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu impish/universe amd64 Packages
ubuntu@juju-2e9148-rbd-mirror-0:~$ lsb_release -a
No LSB modules are available.

ubuntu@juju-2e9148-rbd-mirror-0:~$ sudo systemctl stop rbd-target-api
ubuntu@juju-2e9148-rbd-mirror-0:~$ sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/1
sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/2
sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist ls
blocklisting 10.5.1.220:0/1 until 2022-06-02T17:28:46.213863+0000 (3600 sec)
blocklisting 10.5.1.220:0/2 until 2022-06-02T17:28:47.402322+0000 (3600 sec)
10.5.1.220:0/2 2022-06-02T17:28:47.402322+0000
10.5.1.220:0/1 2022-06-02T17:28:46.213863+0000
listed 2 entries
ubuntu@juju-2e9148-rbd-mirror-0:~$ sudo /usr/bin/python3 /usr/bin/rbd-target-api
ubuntu@juju-2e9148-rbd-mirror-0:~$ sudo systemctl start rbd-target-api
ubuntu@juju-2e9148-rbd-mirror-0:~$ sudo systemctl status rbd-target-api
● rbd-target-api.service - Ceph iscsi target configuration API
     Loaded: loaded (/lib/systemd/system/rbd-target-api.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-06-02 16:29:20 UTC; 7s ago
   Main PID: 8385 (rbd-target-api)
      Tasks: 26 (limit: 2338)
     Memory: 39.4M
        CPU: 1.096s
     CGroup: /system.slice/rbd-target-api.service
             └─8385 /usr/bin/python3 /usr/bin/rbd-target-api

Jun 02 16:29:21 juju-2e9148-rbd-mirror-0 rbd-target-api[8385]: Processing osd blacklist entries for this node
Jun 02 16:29:21 juju-2e9148-rbd-mirror-0 rbd-target-api[8385]: Checking for config object changes every 1s
Jun 02 16:29:22 juju-2e9148-rbd-mirror-0 rbd-target-api[8385]: Reading the configuration object to update local LIO configuration
Jun 02 16:29:22 juju-2e9148-rbd-mirror-0 rbd-target-api[8385]: Configuration does not have an entry for this host(juju-2e9148-rbd-mirror-0.project.serverstack) - nothing to define to LIO
Jun 02 16:29:22 juju-2e9148-rbd-mirror-0 rbd-target-api[8385]: * Serving Flask app "rbd-target-api" (lazy loading)
Jun 02 16:29:22 juju-2e9148-rbd-mirror-0 rbd-target-api[8385]: * Environment: production
Jun 02 16:29:22 juju-2e9148-rbd-mirror-0 rbd-target-api[8385]: WARNING: This is a development server. Do not use it in a production deployment.
Jun 02 16:29:22 juju-2e9148-rbd-mirror-0 rbd-target-api[8385]: Use a production WSGI server instead.
Jun 02 16:29:22 juju-2e9148-rbd-mirror-0 rbd-target-api[8385]: * Debug mode: off
Jun 02 16:29:22 juju-2e9148-rbd-mirror-0 rbd-target-api[8385]: * Running on http://[::]:5000/ (Press CTRL+C to quit)
Distribu...

Read more...

tags: added: verification-done verification-done-impish
removed: verification-needed verification-needed-impish
Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

The model used to verify the above looks like:

Model Controller Cloud/Region Version SLA Timestamp
rbd-mirror icey-serverstack serverstack/serverstack 2.9.29 unsupported 16:34:08Z

App Version Status Scale Charm Channel Rev Exposed Message
ceph-iscsi active 2 ceph-iscsi 1 no Unit is ready
ceph-mon 16.2.7 active 3 ceph-mon candidate 71 no Unit is ready and clustered
ceph-osd 16.2.7 active 3 ceph-osd candidate 511 no Unit is ready (1 OSD)

Unit Workload Agent Machine Public address Ports Message
ceph-iscsi/0 active idle 0 10.5.1.220 Unit is ready
ceph-iscsi/1* active idle 1 10.5.1.10 Unit is ready
ceph-mon/0 active idle 2 10.5.1.205 Unit is ready and clustered
ceph-mon/1 active idle 3 10.5.0.166 Unit is ready and clustered
ceph-mon/2* active idle 4 10.5.1.50 Unit is ready and clustered
ceph-osd/0* active idle 5 10.5.1.149 Unit is ready (1 OSD)
ceph-osd/1 active idle 6 10.5.1.210 Unit is ready (1 OSD)
ceph-osd/2 active idle 7 10.5.2.178 Unit is ready (1 OSD)

Machine State DNS Inst id Series AZ Message
0 started 10.5.1.220 a48a6a91-8d18-41af-86fb-0fb12c75cc40 impish nova ACTIVE
1 started 10.5.1.10 1e786e43-119c-42eb-b95d-bdc5aa37d8e7 impish nova ACTIVE
2 started 10.5.1.205 c60362a7-fe77-4267-87e7-02bfcdf68ace impish nova ACTIVE
3 started 10.5.0.166 bad1f0ac-896e-45d1-bc53-67caf48fecf7 impish nova ACTIVE
4 started 10.5.1.50 cd432ac9-3442-497f-83e0-5816d6c4be07 impish nova ACTIVE
5 started 10.5.1.149 220e20cc-11dd-4152-adb2-e2a7cf8b5caf impish nova ACTIVE
6 started 10.5.1.210 05921fb6-1e97-40ca-9b89-a03efd5b8af3 impish nova ACTIVE
7 started 10.5.2.178 1d968bfc-97bf-491d-9778-76cc2fe2e1cc impish nova ACTIVE

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

I didn't run into the same issues with verifying as in comment #11 because my test deployment doesn't have TLS configured, and by having "api_secure = False" in the /etc/ceph/iscsi-gateway.cfg file, the problematic werkzeug codepath is bypassed, which allows this verification to be performed.

Revision history for this message
Liam Young (gnuoy) wrote :

+1 makes sense. Thanks for doing this validation @chris.macnaughton

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph-iscsi - 3.4-1ubuntu1.1

---------------
ceph-iscsi (3.4-1ubuntu1.1) impish; urgency=medium

  * d/p/ceph-rm-blacklist-decode.patch: Fix issue with checking for
    string in a byte type object under Python 3 (LP: #1883112).

 -- Liam Young <email address hidden> Tue, 15 Mar 2022 10:36:54 +0000

Changed in ceph-iscsi (Ubuntu Impish):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for ceph-iscsi has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
James Page (james-page) wrote :

This bug got into a slightly odd status for the focal target - AFAICT from reading the history the version in focal-proposed has been validated in #10 and the impish update has now been released.

Please could the focal update be released to -updates as well.

Changed in ceph-iscsi (Ubuntu Focal):
status: Incomplete → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph-iscsi - 3.4-0ubuntu2.1

---------------
ceph-iscsi (3.4-0ubuntu2.1) focal; urgency=medium

  * d/p/ceph-rm-blacklist-decode.patch: Fix issue with checking for
    string in a byte type object under Python 3 (LP: #1883112).

 -- Liam Young <email address hidden> Tue, 15 Mar 2022 10:37:28 +0000

Changed in ceph-iscsi (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.