rbd-target-{api,gw}.service doesn't start after a reboot

[Nobuto Murata]

ceph-iscsi quincy/stable 28

$ apt policy ceph-iscsi
ceph-iscsi:
  Installed: 3.5-2ubuntu1
  Candidate: 3.5-2ubuntu1
  Version table:
 *** 3.5-2ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages
        100 /var/lib/dpkg/status

How to reproduce:
once the ceph-iscsi charm is deployed with 2 units per the documentation, reboot one of the units.

# systemctl status rbd-target-gw.service
○ rbd-target-gw.service - Setup system to export rbd images through LIO
     Loaded: loaded (/lib/systemd/system/rbd-target-gw.service; disabled; vendor preset: enabled)
     Active: inactive (dead)

# systemctl status rbd-target-api.service
○ rbd-target-api.service - Ceph iscsi target configuration API
     Loaded: loaded (/lib/systemd/system/rbd-target-api.service; disabled; vendor preset: enabled)
     Active: inactive (dead)

This is probably because the systemd services are disabled.

# systemctl is-enabled rbd-target-{api,gw}.service
disabled
disabled

A possible workaround is to enable those services by hand.

[Nobuto Murata (nobuto)]

# journalctl --list-boots
-1 c2af49397d224372b583743e9b6aaeb4 Tue 2023-12-05 10:57:48 UTC—Thu 2023-12-07 01:30:33 UTC
 0 5e3a9649dd7f4cd590a939a8e24c5512 Thu 2023-12-07 01:30:52 UTC—Thu 2023-12-07 01:49:00 UTC

[Nobuto Murata (nobuto)]

Subscribing ~field-critical as it's blocking a customer acceptance in a resiliency testing.

[Nobuto Murata (nobuto)]

In the debian/rules, there is an override so I suppose not starting it by default is on purpose since the service doesn't work if there is no config written.

[debian/rules]
override_dh_installinit:
        dh_installinit -pceph-iscsi --name rbd-target-api
        dh_installinit -pceph-iscsi --name rbd-target-gw

[Nobuto Murata (nobuto)]

> In the debian/rules, there is an override so I suppose not starting it by default is on purpose since the service doesn't work if there is no config written.
>
> [debian/rules]
> override_dh_installinit:
> dh_installinit -pceph-iscsi --name rbd-target-api
> dh_installinit -pceph-iscsi --name rbd-target-gw

Hmm, the package structure looks weird to me. The override is to use a service name different from the package name, which makes sense. Then it will render the postinst script as:

====
invoke-rc.d --skip-systemd-native rbd-target-api $_dh_action || exit 1
...
invoke-rc.d --skip-systemd-native rbd-target-gw $_dh_action || exit 1
====

To defer it to systemd. However, the package doesn't leverage dh_installsystemd with --no-enable and --no-start. Instead, it uses debian/ceph-iscsi.install to manually copy usr/lib/systemd/system/*.service to the system.

[Nobuto Murata (nobuto)]

> To defer it to systemd. However, the package doesn't leverage dh_installsystemd with --no-enable and --no-start. Instead, it uses debian/ceph-iscsi.install to manually copy usr/lib/systemd/system/*.service to the system.

I think this is based on the previous upstream documentation.
https://github.com/ceph/ceph-iscsi/blob/97f5b02107ce891360a9741e068a2ec564185154/README#L123-L128

And the new README.md refers to systemctl enable.
https://github.com/ceph/ceph-iscsi/blob/97f5b02107ce891360a9741e068a2ec564185154/README.md?plain=1#L52-L64

In any case, the charm can make sure it's enabled after writing the config.

[Nobuto Murata (nobuto)]

I've confirmed the patch attached fixed the issue, but I wasn't able to have time to finish the unit test mocking.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "charm-ceph-iscsi.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-iscsi/+/902867

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-iscsi/+/902867
Committed: https://opendev.org/openstack/charm-ceph-iscsi/commit/657840476b61adc227ea900cd73ac29fd120a1c3
Submitter: "Zuul (22348)"
Branch: master

commit 657840476b61adc227ea900cd73ac29fd120a1c3
Author: Peter Sabaini <email address hidden>
Date: Thu Dec 7 09:04:20 2023 +0100

    Fix: enable GW services once we have config

    Closes-Bug: #2045828
    Change-Id: Ia473b2793a60172645713b722cbab80dcdbf93b6

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-iscsi/+/903275

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-iscsi/+/903275
Committed: https://opendev.org/openstack/charm-ceph-iscsi/commit/45ba9db2e56786a81c0958397adf1a1030b7b3d8
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit 45ba9db2e56786a81c0958397adf1a1030b7b3d8
Author: Peter Sabaini <email address hidden>
Date: Thu Dec 7 09:04:20 2023 +0100

    Fix: enable GW services once we have config

    Closes-Bug: #2045828
    Change-Id: Ia473b2793a60172645713b722cbab80dcdbf93b6

[Nobuto Murata (nobuto)]

Unsubscribing ~field-critical. The mitigation is implemented in the charm, and the packaging task is not urgent any longer.