Activity log for bug #1863094

Date Who What changed Old value New value Message
2020-02-13 12:12:41 Simon Andrejko bug added bug
2020-02-13 12:46:54 Radosław Piliszek kolla-ansible: status New Incomplete
2020-02-13 15:00:42 Simon Andrejko description Creating volume backup when using LVM backend fails. Module iscsi_tcp is not inserted. Error message from cinder_backup log (Debug mode ON): "libkmod: kmod_module_insert_module: could not find module by name='iscsi_tcp'\niscsiadm: Could not insert module tcp. Kmod error -38\niscsiadm: Could not load transport tcp.Dropping interface default." Expected behavior is to create volume backup using LVM backend successfully. After inserting iscsi_tcp module, creation of volume backup was without any issue. Best Regards, Šimon Description =========== Backup of created cinder volume with backend set as LVM fails. During backup iscsi do discovery and requests iscsi_tcp module. Cinder backup could not provide such module and fails. Steps to reproduce ================== Actions: 1. $ openstack volume create --image cirros-0.3.5 --size 3 test_volume -> Volume is created. 2. $ openstack volume backup create --name test_backup test_volume -> Backup of the volume fails. Expected result =============== After step (2) backup of the volume should be created successfully. Logs ==== 2020-02-12 23:44:20.717 39 DEBUG oslo_concurrency.processutils [-] u'iscsiadm -m discovery -t sendtargets -I default -p 9.9.9.11:3260' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:457 2020-02-12 23:44:20.718 39 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[140264042973136]: Unexpected error while running command. Command: iscsiadm -m discovery -t sendtargets -I default -p 9.9.9.11:3260 Exit code: 21 Stdout: u'' Stderr: u"libkmod: kmod_module_insert_module: could not find module by name='iscsi_tcp'\niscsiadm: Could not insert module tcp. Kmod error -38\niscsiadm: Could not load transport tcp.Dropping interface default.\n" loop /usr/lib/python2.7 /site-packages/oslo_privsep/daemon.py:449 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 445, in loop reply = self._process_cmd(*msg) File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 428, in _process_cmd ret = func(*f_args, **f_kwargs) File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 209, in _wrap return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 194, in execute_root return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 143, in custom_execute on_completion=on_completion, *cmd, **kwargs) File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute cmd=sanitized_cmd) ProcessExecutionError: Unexpected error while running command. Command: iscsiadm -m discovery -t sendtargets -I default -p 9.9.9.11:3260 Exit code: 21 Stdout: u'' Stderr: u"libkmod: kmod_module_insert_module: could not find module by name='iscsi_tcp'\niscsiadm: Could not insert module tcp. Kmod error -38\niscsiadm: Could not load transport tcp.Dropping interface default.\n" Additional ========== Root cause is probably that volume /lib/modules is not mounted in cinder_backup. After inserting iscsi_tcp module ($ modprobe iscsi_tcp) creating of backup worked. Best Regards, Šimon
2020-02-13 15:20:54 Simon Andrejko description Description =========== Backup of created cinder volume with backend set as LVM fails. During backup iscsi do discovery and requests iscsi_tcp module. Cinder backup could not provide such module and fails. Steps to reproduce ================== Actions: 1. $ openstack volume create --image cirros-0.3.5 --size 3 test_volume -> Volume is created. 2. $ openstack volume backup create --name test_backup test_volume -> Backup of the volume fails. Expected result =============== After step (2) backup of the volume should be created successfully. Logs ==== 2020-02-12 23:44:20.717 39 DEBUG oslo_concurrency.processutils [-] u'iscsiadm -m discovery -t sendtargets -I default -p 9.9.9.11:3260' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:457 2020-02-12 23:44:20.718 39 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[140264042973136]: Unexpected error while running command. Command: iscsiadm -m discovery -t sendtargets -I default -p 9.9.9.11:3260 Exit code: 21 Stdout: u'' Stderr: u"libkmod: kmod_module_insert_module: could not find module by name='iscsi_tcp'\niscsiadm: Could not insert module tcp. Kmod error -38\niscsiadm: Could not load transport tcp.Dropping interface default.\n" loop /usr/lib/python2.7 /site-packages/oslo_privsep/daemon.py:449 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 445, in loop reply = self._process_cmd(*msg) File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 428, in _process_cmd ret = func(*f_args, **f_kwargs) File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 209, in _wrap return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 194, in execute_root return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 143, in custom_execute on_completion=on_completion, *cmd, **kwargs) File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute cmd=sanitized_cmd) ProcessExecutionError: Unexpected error while running command. Command: iscsiadm -m discovery -t sendtargets -I default -p 9.9.9.11:3260 Exit code: 21 Stdout: u'' Stderr: u"libkmod: kmod_module_insert_module: could not find module by name='iscsi_tcp'\niscsiadm: Could not insert module tcp. Kmod error -38\niscsiadm: Could not load transport tcp.Dropping interface default.\n" Additional ========== Root cause is probably that volume /lib/modules is not mounted in cinder_backup. After inserting iscsi_tcp module ($ modprobe iscsi_tcp) creating of backup worked. Best Regards, Šimon Description =========== Backup of created cinder volume with backend set as LVM fails in case iscsi_tcp kernel module is not inserted on storage node. During backup iscsi do discovery and requests iscsi_tcp module. cinder_backup container could not insert such module and fails. Steps to reproduce ================== Actions: 1. $ openstack volume create --image cirros-0.3.5 --size 3 test_volume -> Volume is created. 2. $ openstack volume backup create --name test_backup test_volume -> Backup of the volume fails. Expected result =============== After step (2) backup of the volume should be created successfully. Logs ==== 2020-02-12 23:44:20.717 39 DEBUG oslo_concurrency.processutils [-] u'iscsiadm -m discovery -t sendtargets -I default -p 9.9.9.11:3260' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:457 2020-02-12 23:44:20.718 39 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[140264042973136]: Unexpected error while running command. Command: iscsiadm -m discovery -t sendtargets -I default -p 9.9.9.11:3260 Exit code: 21 Stdout: u'' Stderr: u"libkmod: kmod_module_insert_module: could not find module by name='iscsi_tcp'\niscsiadm: Could not insert module tcp. Kmod error -38\niscsiadm: Could not load transport tcp.Dropping interface default.\n" loop /usr/lib/python2.7 /site-packages/oslo_privsep/daemon.py:449 Traceback (most recent call last):   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 445, in loop     reply = self._process_cmd(*msg)   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 428, in _process_cmd     ret = func(*f_args, **f_kwargs)   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 209, in _wrap     return func(*args, **kwargs)   File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 194, in execute_root     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)   File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 143, in custom_execute     on_completion=on_completion, *cmd, **kwargs)   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute     cmd=sanitized_cmd) ProcessExecutionError: Unexpected error while running command. Command: iscsiadm -m discovery -t sendtargets -I default -p 9.9.9.11:3260 Exit code: 21 Stdout: u'' Stderr: u"libkmod: kmod_module_insert_module: could not find module by name='iscsi_tcp'\niscsiadm: Could not insert module tcp. Kmod error -38\niscsiadm: Could not load transport tcp.Dropping interface default.\n" Additional ========== Root cause is probably that volume /lib/modules is not mounted in cinder_backup. After inserting iscsi_tcp module ($ modprobe iscsi_tcp) creating of backup worked. Best Regards, Šimon
2020-02-13 17:08:28 Radosław Piliszek kolla-ansible: status Incomplete New
2020-02-13 20:15:44 Radosław Piliszek kolla-ansible: status New Triaged
2020-02-14 05:18:14 Chason Chan kolla-ansible: assignee Chason Chan (chen-xing)
2020-02-14 07:55:08 OpenStack Infra kolla-ansible: status Triaged In Progress
2020-02-14 07:55:08 OpenStack Infra kolla-ansible: assignee Chason Chan (chen-xing) Radosław Piliszek (yoctozepto)
2020-02-14 11:42:29 Radosław Piliszek nominated for series kolla-ansible/rocky
2020-02-14 11:42:29 Radosław Piliszek bug task added kolla-ansible/rocky
2020-02-14 11:42:29 Radosław Piliszek nominated for series kolla-ansible/stein
2020-02-14 11:42:29 Radosław Piliszek bug task added kolla-ansible/stein
2020-02-14 11:42:29 Radosław Piliszek nominated for series kolla-ansible/ussuri
2020-02-14 11:42:29 Radosław Piliszek bug task added kolla-ansible/ussuri
2020-02-14 11:42:29 Radosław Piliszek nominated for series kolla-ansible/train
2020-02-14 11:42:29 Radosław Piliszek bug task added kolla-ansible/train
2020-02-14 11:42:35 Radosław Piliszek kolla-ansible/train: status New Triaged
2020-02-14 11:42:36 Radosław Piliszek kolla-ansible/stein: status New Triaged
2020-02-14 11:42:39 Radosław Piliszek kolla-ansible/rocky: status New Triaged
2020-02-14 11:42:45 Radosław Piliszek kolla-ansible/ussuri: importance Undecided Medium
2020-02-14 11:42:52 Radosław Piliszek kolla-ansible/train: importance Undecided Medium
2020-02-14 11:42:54 Radosław Piliszek kolla-ansible/stein: importance Undecided Medium
2020-02-14 11:42:59 Radosław Piliszek kolla-ansible/rocky: importance Undecided Medium
2020-02-14 11:43:04 Radosław Piliszek kolla-ansible/ussuri: milestone 10.0.0
2020-02-14 13:06:06 OpenStack Infra kolla-ansible: assignee Radosław Piliszek (yoctozepto) Chason Chan (chen-xing)
2020-02-14 13:28:44 OpenStack Infra kolla-ansible: status In Progress Fix Released
2020-02-14 15:37:37 Radosław Piliszek kolla-ansible/ussuri: assignee Chason Chan (chen-xing) Radosław Piliszek (yoctozepto)
2020-02-14 15:47:32 OpenStack Infra kolla-ansible/train: status Triaged In Progress
2020-02-14 15:47:32 OpenStack Infra kolla-ansible/train: assignee Radosław Piliszek (yoctozepto)
2020-02-14 15:52:01 OpenStack Infra kolla-ansible/stein: status Triaged In Progress
2020-02-14 15:52:01 OpenStack Infra kolla-ansible/stein: assignee Radosław Piliszek (yoctozepto)
2020-02-14 15:53:59 OpenStack Infra kolla-ansible/rocky: status Triaged In Progress
2020-02-14 15:53:59 OpenStack Infra kolla-ansible/rocky: assignee Radosław Piliszek (yoctozepto)
2020-02-14 15:54:40 Radosław Piliszek kolla-ansible/train: milestone 9.0.2
2020-02-14 15:54:42 Radosław Piliszek kolla-ansible/stein: milestone 8.1.1
2020-02-14 15:54:44 Radosław Piliszek kolla-ansible/rocky: milestone 7.2.1
2020-02-14 22:01:29 OpenStack Infra kolla-ansible/stein: status In Progress Fix Committed
2020-02-14 22:17:49 OpenStack Infra kolla-ansible/train: status In Progress Fix Committed
2020-02-14 22:17:58 OpenStack Infra kolla-ansible/rocky: status In Progress Fix Committed
2020-03-24 11:15:40 Mark Goddard kolla-ansible/rocky: status Fix Committed Fix Released
2020-05-05 10:18:46 Mark Goddard kolla-ansible/stein: status Fix Committed Fix Released
2020-05-05 10:18:48 Mark Goddard kolla-ansible/train: status Fix Committed Fix Released