Copy/paste of commands to check before/after:
... Reproducer (15-second startup delay in tgt.service)
FILE=/etc/systemd/system/tgt.service.d/start-delay.conf mkdir -p $(dirname $FILE) cat <<EOF > $FILE [Service] ExecStartPre=$(which sleep) 15 EOF systemctl daemon-reload
Before: ---
... Restart both services at the same time:
# date; systemctl restart cinder-volume.service tgt.service; date Mon Oct 16 21:57:12 UTC 2023 Mon Oct 16 21:57:27 UTC 2023
... Notice that cinder-volume.service is Started _BEFORE_ tgt.service
# journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -3 Oct 16 21:57:12 cinder-mantic systemd[1]: Started cinder-volume.service - OpenStack Cinder Volume. Oct 16 21:57:12 cinder-mantic systemd[1]: Starting tgt.service - (i)SCSI target daemon... Oct 16 21:57:27 cinder-mantic systemd[1]: Started tgt.service - (i)SCSI target daemon.
.. Log error:
# grep 'ERROR cinder.volume.manager' /var/log/cinder/cinder-volume.log ... 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager [None req-ce03264a-6765-41de-8016-a6f27d2685e4 - - - - - -] Failed to re-export volume, setting to ERROR.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. ... 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager Command: tgtadm --lld iscsi --op show --mode target 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager Exit code: 107 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager Stdout: '' 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager Stderr: 'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager
... Volume error:
# cinder list +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | 17c4f736-058b-4cfc-9864-a64ab0995957 | error | test-volume | 1 | True | __DEFAULT__ | false | | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
... Undo error state:
# systemctl restart cinder-volume.service # cinder reset-state --state in-use test-volume # cinder list
+--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | 17c4f736-058b-4cfc-9864-a64ab0995957 | in-use | test-volume | 1 | True | __DEFAULT__ | false | | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
After: ---
... Patched with Wants=
# systemctl show cinder-volume.service | grep Wants= Wants=network-online.target tgt.service
# cinder list +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | 17c4f736-058b-4cfc-9864-a64ab0995957 | in-use | test-volume | 1 | True | __DEFAULT__ | false | | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
# date; systemctl restart cinder-volume.service tgt.service; date Wed Oct 18 15:29:01 UTC 2023 Wed Oct 18 15:29:17 UTC 2023
... Notice that cinder-volume.service is Started _AFTER_ tgt.service
# journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -3 Oct 18 15:29:01 cinder-mantic systemd[1]: Starting tgt.service - (i)SCSI target daemon... Oct 18 15:29:17 cinder-mantic systemd[1]: Started tgt.service - (i)SCSI target daemon. Oct 18 15:29:17 cinder-mantic systemd[1]: Started cinder-volume.service - OpenStack Cinder Volume.
... Volume not in error:
... Without tgt.service installed:
# apt remove --purge --yes tgt
# systemctl status tgt.service Unit tgt.service could not be found.
# systemctl show cinder-volume.service | grep Wants= Wants=tgt.service network-online.target
# systemctl restart cinder-volume.service # echo $? 0
# systemctl status cinder-volume.service | grep Active: Active: active (running) since Wed 2023-10-18 16:47:14 UTC; 11s ago
# journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -1 Oct 18 16:47:14 cinder-mantic systemd[1]: Started cinder-volume.service - OpenStack Cinder Volume.
Copy/paste of commands to check before/after:
... Reproducer (15-second startup delay in tgt.service)
FILE=/ etc/systemd/ system/ tgt.service. d/start- delay.conf $(which sleep) 15
mkdir -p $(dirname $FILE)
cat <<EOF > $FILE
[Service]
ExecStartPre=
EOF
systemctl daemon-reload
Before:
---
... Restart both services at the same time:
# date; systemctl restart cinder- volume. service tgt.service; date
Mon Oct 16 21:57:12 UTC 2023
Mon Oct 16 21:57:27 UTC 2023
... Notice that cinder- volume. service is Started _BEFORE_ tgt.service
# journalctl -b -u cinder- volume. service -u tgt.service | grep Start | tail -3 volume. service - OpenStack Cinder Volume.
Oct 16 21:57:12 cinder-mantic systemd[1]: Started cinder-
Oct 16 21:57:12 cinder-mantic systemd[1]: Starting tgt.service - (i)SCSI target daemon...
Oct 16 21:57:27 cinder-mantic systemd[1]: Started tgt.service - (i)SCSI target daemon.
.. Log error:
# grep 'ERROR cinder. volume. manager' /var/log/ cinder/ cinder- volume. log volume. manager [None req-ce03264a- 6765-41de- 8016-a6f27d2685 e4 - - - - - -] Failed to re-export volume, setting to ERROR.: oslo_concurrenc y.processutils. ProcessExecutio nError: Unexpected error while running command. volume. manager oslo_concurrenc y.processutils. ProcessExecutio nError: Unexpected error while running command. volume. manager Command: tgtadm --lld iscsi --op show --mode target volume. manager Exit code: 107 volume. manager Stdout: '' volume. manager Stderr: 'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' volume. manager
...
2023-10-16 21:57:18.301 1658 ERROR cinder.
...
2023-10-16 21:57:18.301 1658 ERROR cinder.
2023-10-16 21:57:18.301 1658 ERROR cinder.
2023-10-16 21:57:18.301 1658 ERROR cinder.
2023-10-16 21:57:18.301 1658 ERROR cinder.
2023-10-16 21:57:18.301 1658 ERROR cinder.
2023-10-16 21:57:18.301 1658 ERROR cinder.
... Volume error:
# cinder list ------- ------- ------- ------- -----+- ------- +------ ------- +------ +------ ------- ---+--- ------- ---+--- ------- +------ ------- + ------- ------- ------- ------- -----+- ------- +------ ------- +------ +------ ------- ---+--- ------- ---+--- ------- +------ ------- + 058b-4cfc- 9864-a64ab09959 57 | error | test-volume | 1 | True | __DEFAULT__ | false | | ------- ------- ------- ------- -----+- ------- +------ ------- +------ +------ ------- ---+--- ------- ---+--- ------- +------ ------- +
+-----
| ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to |
+-----
| 17c4f736-
+-----
... Undo error state:
# systemctl restart cinder- volume. service
# cinder reset-state --state in-use test-volume
# cinder list
+----- ------- ------- ------- ------- -----+- ------- +------ ------- +------ +------ ------- ---+--- ------- ---+--- ------- +------ ------- + ------- ------- ------- ------- -----+- ------- +------ ------- +------ +------ ------- ---+--- ------- ---+--- ------- +------ ------- + 058b-4cfc- 9864-a64ab09959 57 | in-use | test-volume | 1 | True | __DEFAULT__ | false | | ------- ------- ------- ------- -----+- ------- +------ ------- +------ +------ ------- ---+--- ------- ---+--- ------- +------ ------- +
| ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to |
+-----
| 17c4f736-
+-----
After:
---
... Patched with Wants=
# systemctl show cinder- volume. service | grep Wants= network- online. target tgt.service
Wants=
# cinder list ------- ------- ------- ------- -----+- ------- +------ ------- +------ +------ ------- ---+--- ------- ---+--- ------- +------ ------- + ------- ------- ------- ------- -----+- ------- +------ ------- +------ +------ ------- ---+--- ------- ---+--- ------- +------ ------- + 058b-4cfc- 9864-a64ab09959 57 | in-use | test-volume | 1 | True | __DEFAULT__ | false | | ------- ------- ------- ------- -----+- ------- +------ ------- +------ +------ ------- ---+--- ------- ---+--- ------- +------ ------- +
+-----
| ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to |
+-----
| 17c4f736-
+-----
... Restart both services at the same time:
# date; systemctl restart cinder- volume. service tgt.service; date
Wed Oct 18 15:29:01 UTC 2023
Wed Oct 18 15:29:17 UTC 2023
... Notice that cinder- volume. service is Started _AFTER_ tgt.service
# journalctl -b -u cinder- volume. service -u tgt.service | grep Start | tail -3 volume. service - OpenStack Cinder Volume.
Oct 18 15:29:01 cinder-mantic systemd[1]: Starting tgt.service - (i)SCSI target daemon...
Oct 18 15:29:17 cinder-mantic systemd[1]: Started tgt.service - (i)SCSI target daemon.
Oct 18 15:29:17 cinder-mantic systemd[1]: Started cinder-
... Volume not in error:
# cinder list ------- ------- ------- ------- -----+- ------- +------ ------- +------ +------ ------- ---+--- ------- ---+--- ------- +------ ------- + ------- ------- ------- ------- -----+- ------- +------ ------- +------ +------ ------- ---+--- ------- ---+--- ------- +------ ------- + 058b-4cfc- 9864-a64ab09959 57 | in-use | test-volume | 1 | True | __DEFAULT__ | false | | ------- ------- ------- ------- -----+- ------- +------ ------- +------ +------ ------- ---+--- ------- ---+--- ------- +------ ------- +
+-----
| ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to |
+-----
| 17c4f736-
+-----
... Without tgt.service installed:
# apt remove --purge --yes tgt
# systemctl status tgt.service
Unit tgt.service could not be found.
# systemctl show cinder- volume. service | grep Wants= online. target
Wants=tgt.service network-
# systemctl restart cinder- volume. service
# echo $?
0
# systemctl status cinder- volume. service | grep Active:
Active: active (running) since Wed 2023-10-18 16:47:14 UTC; 11s ago
# journalctl -b -u cinder- volume. service -u tgt.service | grep Start | tail -1 volume. service - OpenStack Cinder Volume.
Oct 18 16:47:14 cinder-mantic systemd[1]: Started cinder-