Register rados module path failed

[Piotr]

OS: Centos 8 stream
openstack-ansible setup-openstack.yml or os-cinder-install.yml
24.1.0.dev.184

TASK [ceph_client : Register rados module path] **************************************************************************************************************************************************************
fatal: [c2]: FAILED! => {"changed": false, "cmd": ["/usr/bin/python3", "-c", "import rados; print(rados.__file__)"], "delta": "0:00:00.016340", "end": "2022-05-10 13:01:38.656841", "msg": "non-zero return code", "rc": 1, "start": "2022-05-10 13:01:38.640501", "stderr": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\nModuleNotFoundError: No module named 'rados'", "stderr_lines": ["Traceback (most recent call last):", " File \"<string>\", line 1, in <module>", "ModuleNotFoundError: No module named 'rados'"], "stdout": "", "stdout_lines": []}

TASK [python_venv_build : gather build target facts] *********************************************************************************************************************************************************
ok: [c2 -> c2_repo_container-46fbf376(10.32.0.134)] => (item=c2_repo_container-46fbf376)

TASK [python_venv_build : Fail if wheels are not built for multiple hosts] ***********************************************************************************************************************************

TASK [python_venv_build : include_tasks] *********************************************************************************************************************************************************************
included: /etc/ansible/roles/python_venv_build/tasks/python_venv_wheel_build.yml for c2

TASK [python_venv_build : Install distro packages for wheel build] *******************************************************************************************************************************************
ok: [c2 -> c2_repo_container-46fbf376(10.32.0.134)]

TASK [python_venv_build : Clean up paths and files if venv_rebuild is enabled] *******************************************************************************************************************************
skipping: [c2] => (item=/var/www/repo/os-releases/24.1.0.dev184/centos-8-x86_64/requirements/cinder-24.1.0.dev184-requirements.txt)
skipping: [c2] => (item=/var/www/repo/os-releases/24.1.0.dev184/centos-8-x86_64/requirements/cinder-24.1.0.dev184-global-constraints.txt)
skipping: [c2] => (item=/var/www/repo/os-releases/24.1.0.dev184/centos-8-x86_64/requirements/cinder-24.1.0.dev184-source-constraints.txt)
skipping: [c2] => (item=/var/www/repo/os-releases/24.1.0.dev184/centos-8-x86_64/requirements/cinder-24.1.0.dev184-constraints.txt)

TASK [python_venv_build : Create wheel directory on the build host] ******************************************************************************************************************************************
ok: [c2 -> c2_repo_container-46fbf376(10.32.0.134)]

TASK [python_venv_build : Create requirements/constraints file directory on the build host] ******************************************************************************************************************
ok: [c2 -> c2_repo_container-46fbf376(10.32.0.134)]

TASK [python_venv_build : Create the wheel build virtualenv (if it does not exist)] **************************************************************************************************************************
ok: [c2 -> c2_repo_container-46fbf376(10.32.0.134)]

TASK [python_venv_build : Build requirements file for the venv] **********************************************************************************************************************************************
ok: [c2 -> c2_repo_container-46fbf376(10.32.0.134)]

TASK [python_venv_build : Build global constraints file for the venv] ****************************************************************************************************************************************
ok: [c2 -> c2_repo_container-46fbf376(10.32.0.134)]

TASK [python_venv_build : Build constraints file for the venv] ***********************************************************************************************************************************************
ok: [c2 -> c2_repo_container-46fbf376(10.32.0.134)]

TASK [python_venv_build : Upgrade the wheel build virtualenv pip/setuptools/wheel to the versions we want] ***************************************************************************************************
ok: [c2 -> c2_repo_container-46fbf376(10.32.0.134)]

TASK [python_venv_build : Clean up temporary wheel build path] ***********************************************************************************************************************************************

TASK [python_venv_build : Build wheels for the packages to be installed into the venv] ***********************************************************************************************************************
skipping: [c2]

TASK [python_venv_build : Index built wheels] ****************************************************************************************************************************************************************
skipping: [c2]

TASK [python_venv_build : Set wheel permissions] *************************************************************************************************************************************************************

TASK [python_venv_build : Move built wheels to common wheel path] ********************************************************************************************************************************************

TASK [python_venv_build : Build constraints file for installation purposes] **********************************************************************************************************************************

TASK [python_venv_build : Slurp up the constraints file for later re-deployment] *****************************************************************************************************************************
ok: [c2 -> c2_repo_container-46fbf376(10.32.0.134)]

TASK [python_venv_build : Install distro packages for venv build] ********************************************************************************************************************************************
ok: [c2]

TASK [python_venv_build : Ensure a fresh venv_install_destination_path if venv_rebuild is enabled] ***********************************************************************************************************

TASK [python_venv_build : Create the venv_install_destination_path parent directory] *************************************************************************************************************************
ok: [c2]

TASK [python_venv_build : Create the virtualenv (if it does not exist)] **************************************************************************************************************************************
ok: [c2]

TASK [python_venv_build : Create the requirements file contents] *********************************************************************************************************************************************
ok: [c2]

TASK [python_venv_build : Build requirements file for the venv] **********************************************************************************************************************************************
ok: [c2]

TASK [python_venv_build : Build global constraints file for the venv] ****************************************************************************************************************************************
ok: [c2]

TASK [python_venv_build : Build constraints file for the venv] ***********************************************************************************************************************************************
ok: [c2]

TASK [python_venv_build : Upgrade pip/setuptools/wheel to the versions we want] ******************************************************************************************************************************
ok: [c2]

TASK [python_venv_build : Install python packages into the venv] *********************************************************************************************************************************************
skipping: [c2]

TASK [python_venv_build : Add symlinks from distribution python packages] ************************************************************************************************************************************

TASK [python_venv_build : Ensure local facts folder exists] **************************************************************************************************************************************************
ok: [c2]

TASK [python_venv_build : Record the necessary facts] ********************************************************************************************************************************************************
skipping: [c2] => (item={'section': 'cinder', 'option': 'need_service_restart', 'value': True})
skipping: [c2] => (item={'section': 'cinder', 'option': 'need_online_data_migrations', 'value': True})
skipping: [c2] => (item={'section': 'cinder', 'option': 'venv_tag', 'value': '24.1.0.dev184'})

TASK [os_cinder : Link in the os-brick rootwrap filters] *****************************************************************************************************************************************************
ok: [c2]

TASK [os_cinder : Copy cinder rootwrap filters] **************************************************************************************************************************************************************
ok: [c2]

TASK [systemd_service : Pull systemd version] ****************************************************************************************************************************************************************
ok: [c2]

TASK [systemd_service : Set systemd version] *****************************************************************************************************************************************************************
ok: [c2]

TASK [systemd_service : Create TEMP run dir] *****************************************************************************************************************************************************************
ok: [c2] => (item={'group': 'cinder_volume', 'service_name': 'cinder-volume', 'start_order': 2, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-volume', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})
ok: [c2] => (item={'group': 'cinder_backup', 'service_name': 'cinder-backup', 'condition': True, 'start_order': 3, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-backup', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-backup', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})

TASK [systemd_service : Create TEMP service lock dir] ********************************************************************************************************************************************************
ok: [c2] => (item={'group': 'cinder_volume', 'service_name': 'cinder-volume', 'start_order': 2, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-volume', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})
ok: [c2] => (item={'group': 'cinder_backup', 'service_name': 'cinder-backup', 'condition': True, 'start_order': 3, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-backup', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-backup', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})

TASK [systemd_service : Create service.d overrides dir] ******************************************************************************************************************************************************
skipping: [c2] => (item={'group': 'cinder_volume', 'service_name': 'cinder-volume', 'start_order': 2, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-volume', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})
skipping: [c2] => (item={'group': 'cinder_backup', 'service_name': 'cinder-backup', 'condition': True, 'start_order': 3, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-backup', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-backup', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})

TASK [systemd_service : Create tmpfiles.d entry] *************************************************************************************************************************************************************
ok: [c2] => (item={'group': 'cinder_volume', 'service_name': 'cinder-volume', 'start_order': 2, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-volume', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})
ok: [c2] => (item={'group': 'cinder_backup', 'service_name': 'cinder-backup', 'condition': True, 'start_order': 3, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-backup', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-backup', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})

TASK [systemd_service : Place the systemd service] ***********************************************************************************************************************************************************
ok: [c2] => (item={'group': 'cinder_volume', 'service_name': 'cinder-volume', 'start_order': 2, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-volume', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})
ok: [c2] => (item={'group': 'cinder_backup', 'service_name': 'cinder-backup', 'condition': True, 'start_order': 3, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-backup', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-backup', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})

TASK [systemd_service : Place the systemd override] **********************************************************************************************************************************************************
skipping: [c2] => (item={'group': 'cinder_volume', 'service_name': 'cinder-volume', 'start_order': 2, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-volume', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})
skipping: [c2] => (item={'group': 'cinder_backup', 'service_name': 'cinder-backup', 'condition': True, 'start_order': 3, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-backup', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-backup', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})

TASK [systemd_service : Place the systemd timer] *************************************************************************************************************************************************************
skipping: [c2] => (item={'group': 'cinder_volume', 'service_name': 'cinder-volume', 'start_order': 2, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-volume', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})
skipping: [c2] => (item={'group': 'cinder_backup', 'service_name': 'cinder-backup', 'condition': True, 'start_order': 3, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-backup', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-backup', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})

TASK [systemd_service : Place the systemd socket] ************************************************************************************************************************************************************

TASK [systemd_service : Reload systemd on unit change] *******************************************************************************************************************************************************

TASK [systemd_service : include_tasks] ***********************************************************************************************************************************************************************
included: /etc/ansible/roles/systemd_service/tasks/systemd_load.yml for c2 => (item={'group': 'cinder_volume', 'service_name': 'cinder-volume', 'start_order': 2, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-volume', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})
included: /etc/ansible/roles/systemd_service/tasks/systemd_load.yml for c2 => (item={'group': 'cinder_backup', 'service_name': 'cinder-backup', 'condition': True, 'start_order': 3, 'execstarts': '/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-backup', 'execreloads': '/bin/kill -HUP $MAINPID', 'service_key': 'cinder-backup', 'enabled': 'yes', 'state': 'started', 'config_overrides': {'Service': {'Environment': 'PATH=/openstack/venvs/cinder-24.1.0.dev184/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'}}})

TASK [systemd_service : Load service cinder-volume] **********************************************************************************************************************************************************
ok: [c2] => (item=)

TASK [systemd_service : Load timer cinder-volume] ************************************************************************************************************************************************************
skipping: [c2] => (item=)

TASK [systemd_service : Load service cinder-backup] **********************************************************************************************************************************************************
ok: [c2] => (item=)

TASK [systemd_service : Load timer cinder-backup] ************************************************************************************************************************************************************
skipping: [c2] => (item=)

TASK [systemd_service : Load socket] *************************************************************************************************************************************************************************

TASK [os_cinder : Set the upgrade facts] *********************************************************************************************************************************************************************
skipping: [c2] => (item=need_online_data_migrations)
skipping: [c2] => (item=need_service_restart)

TASK [os_cinder : refresh local facts] ***********************************************************************************************************************************************************************
ok: [c2]

TASK [os_cinder : include_tasks] *****************************************************************************************************************************************************************************

TASK [os_cinder : Create aux cinder dir] *********************************************************************************************************************************************************************
ok: [c2]

TASK [os_cinder : Generate cinder config] ********************************************************************************************************************************************************************
ok: [c2]

TASK [os_cinder : Copy cinder configs] ***********************************************************************************************************************************************************************
ok: [c2] => (item={'src': 'resource_filters.json.j2', 'dest': '/etc/cinder/resource_filters.json', 'config_overrides': {}, 'config_type': 'json'})

TASK [os_cinder : Implement policy.yaml if there are overrides configured] ***********************************************************************************************************************************

TASK [os_cinder : Remove legacy policy.yaml file] ************************************************************************************************************************************************************
ok: [c2]

TASK [os_cinder : Preserve original configuration file(s)] ***************************************************************************************************************************************************
ok: [c2] => (item={'tmp_f': '/tmp/api-paste.ini', 'target_f': '/etc/cinder/api-paste.ini', 'config_overrides': {}, 'config_type': 'ini'})
ok: [c2] => (item={'tmp_f': '/tmp/rootwrap.conf', 'target_f': '/etc/cinder/rootwrap.conf', 'config_overrides': {'DEFAULT': {'filters_path': '/etc/cinder/rootwrap.d,/usr/share/cinder/rootwrap', 'exec_dirs': '/openstack/venvs/cinder-24.1.0.dev184/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin'}}, 'config_type': 'ini', 'owner': 'root', 'group': 'cinder', 'mode': '0640'})
ok: [c2] => (item={'tmp_f': '/tmp/resource_filters.json', 'target_f': '/etc/cinder/resource_filters.json', 'config_overrides': {}, 'config_type': 'json', 'owner': 'root', 'group': 'cinder', 'mode': '0640'})

TASK [os_cinder : Fetch override files] **********************************************************************************************************************************************************************
ok: [c2] => (item={'tmp_f': '/tmp/api-paste.ini', 'target_f': '/etc/cinder/api-paste.ini', 'config_overrides': {}, 'config_type': 'ini'})
ok: [c2] => (item={'tmp_f': '/tmp/rootwrap.conf', 'target_f': '/etc/cinder/rootwrap.conf', 'config_overrides': {'DEFAULT': {'filters_path': '/etc/cinder/rootwrap.d,/usr/share/cinder/rootwrap', 'exec_dirs': '/openstack/venvs/cinder-24.1.0.dev184/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin'}}, 'config_type': 'ini', 'owner': 'root', 'group': 'cinder', 'mode': '0640'})
ok: [c2] => (item={'tmp_f': '/tmp/resource_filters.json', 'target_f': '/etc/cinder/resource_filters.json', 'config_overrides': {}, 'config_type': 'json', 'owner': 'root', 'group': 'cinder', 'mode': '0640'})

TASK [os_cinder : Copy common config] ************************************************************************************************************************************************************************
ok: [c2] => (item={'tmp_f': '/tmp/api-paste.ini', 'target_f': '/etc/cinder/api-paste.ini', 'config_overrides': {}, 'config_type': 'ini'})
ok: [c2] => (item={'tmp_f': '/tmp/rootwrap.conf', 'target_f': '/etc/cinder/rootwrap.conf', 'config_overrides': {'DEFAULT': {'filters_path': '/etc/cinder/rootwrap.d,/usr/share/cinder/rootwrap', 'exec_dirs': '/openstack/venvs/cinder-24.1.0.dev184/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin'}}, 'config_type': 'ini', 'owner': 'root', 'group': 'cinder', 'mode': '0640'})
ok: [c2] => (item={'tmp_f': '/tmp/resource_filters.json', 'target_f': '/etc/cinder/resource_filters.json', 'config_overrides': {}, 'config_type': 'json', 'owner': 'root', 'group': 'cinder', 'mode': '0640'})

TASK [os_cinder : Cleanup fetched temp files] ****************************************************************************************************************************************************************
ok: [c2 -> localhost] => (item={'tmp_f': '/tmp/api-paste.ini', 'target_f': '/etc/cinder/api-paste.ini', 'config_overrides': {}, 'config_type': 'ini'})
ok: [c2 -> localhost] => (item={'tmp_f': '/tmp/rootwrap.conf', 'target_f': '/etc/cinder/rootwrap.conf', 'config_overrides': {'DEFAULT': {'filters_path': '/etc/cinder/rootwrap.d,/usr/share/cinder/rootwrap', 'exec_dirs': '/openstack/venvs/cinder-24.1.0.dev184/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin'}}, 'config_type': 'ini', 'owner': 'root', 'group': 'cinder', 'mode': '0640'})
ok: [c2 -> localhost] => (item={'tmp_f': '/tmp/resource_filters.json', 'target_f': '/etc/cinder/resource_filters.json', 'config_overrides': {}, 'config_type': 'json', 'owner': 'root', 'group': 'cinder', 'mode': '0640'})

TASK [os_cinder : Set rootwrap.d permissions] ****************************************************************************************************************************************************************
ok: [c2]

TASK [os_cinder : Ensure cinder tgt include] *****************************************************************************************************************************************************************

TASK [os_cinder : Create nfs shares export file] *************************************************************************************************************************************************************
skipping: [c2] => (item={'key': 'RBD', 'value': {'volume_driver': 'cinder.volume.drivers.rbd.RBDDriver', 'rbd_pool': 'volumes', 'rbd_ceph_conf': '/etc/ceph/ceph.conf', 'rbd_store_chunk_size': 8, 'volume_backend_name': 'rbddriver', 'rbd_user': 'cinder', 'rbd_secret_uuid': '4fc20fcd-fcab-4c56-ac1d-5c9cf5a13934', 'report_discard_supported': True}})
ok: [c2] => (item={'key': 'NFS', 'value': {'volume_driver': 'cinder.volume.drivers.nfs.NfsDriver', 'volume_backend_name': 'NFS', 'nfs_shares_config': '/etc/cinder/nfs_shares', 'nfs_mount_options': 'rsize=65535,wsize=65535,timeo=1200,actimeo=120', 'shares': [{'ip': '10.32.0.5', 'share': '/nfs4'}]}})
skipping: [c2] => (item={'key': 'LVM', 'value': {'volume_group': 'cinder-volumes', 'volume_driver': 'cinder.volume.drivers.lvm.LVMVolumeDriver', 'volume_backend_name': 'LVM', 'iscsi_ip_address': '10.32.0.4', 'iscsi_helper': 'tgtadm'}})

TASK [os_cinder : Drop sudoers file] *************************************************************************************************************************************************************************
ok: [c2]

TASK [systemd_service : Pull systemd version] ****************************************************************************************************************************************************************
ok: [c2]

TASK [systemd_service : Set systemd version] *****************************************************************************************************************************************************************
ok: [c2]

TASK [systemd_service : Create TEMP run dir] *****************************************************************************************************************************************************************
ok: [c2] => (item={'service_name': 'cinder-volume-usage-audit', 'execstarts': ['/bin/bash -c "/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume-usage-audit --start_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00\\s-d\\s-1hour)\\" --end_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00)\\" --send_actions"'], 'timer': {'state': 'stopped', 'options': {'OnBootSec': '30min', 'OnCalendar': 'hourly', 'Persistent': True}}})

TASK [systemd_service : Create TEMP service lock dir] ********************************************************************************************************************************************************
ok: [c2] => (item={'service_name': 'cinder-volume-usage-audit', 'execstarts': ['/bin/bash -c "/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume-usage-audit --start_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00\\s-d\\s-1hour)\\" --end_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00)\\" --send_actions"'], 'timer': {'state': 'stopped', 'options': {'OnBootSec': '30min', 'OnCalendar': 'hourly', 'Persistent': True}}})

TASK [systemd_service : Create service.d overrides dir] ******************************************************************************************************************************************************
skipping: [c2] => (item={'service_name': 'cinder-volume-usage-audit', 'execstarts': ['/bin/bash -c "/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume-usage-audit --start_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00\\s-d\\s-1hour)\\" --end_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00)\\" --send_actions"'], 'timer': {'state': 'stopped', 'options': {'OnBootSec': '30min', 'OnCalendar': 'hourly', 'Persistent': True}}})

TASK [systemd_service : Create tmpfiles.d entry] *************************************************************************************************************************************************************
ok: [c2] => (item={'service_name': 'cinder-volume-usage-audit', 'execstarts': ['/bin/bash -c "/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume-usage-audit --start_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00\\s-d\\s-1hour)\\" --end_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00)\\" --send_actions"'], 'timer': {'state': 'stopped', 'options': {'OnBootSec': '30min', 'OnCalendar': 'hourly', 'Persistent': True}}})

TASK [systemd_service : Place the systemd service] ***********************************************************************************************************************************************************
ok: [c2] => (item={'service_name': 'cinder-volume-usage-audit', 'execstarts': ['/bin/bash -c "/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume-usage-audit --start_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00\\s-d\\s-1hour)\\" --end_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00)\\" --send_actions"'], 'timer': {'state': 'stopped', 'options': {'OnBootSec': '30min', 'OnCalendar': 'hourly', 'Persistent': True}}})

TASK [systemd_service : Place the systemd override] **********************************************************************************************************************************************************
skipping: [c2] => (item={'service_name': 'cinder-volume-usage-audit', 'execstarts': ['/bin/bash -c "/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume-usage-audit --start_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00\\s-d\\s-1hour)\\" --end_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00)\\" --send_actions"'], 'timer': {'state': 'stopped', 'options': {'OnBootSec': '30min', 'OnCalendar': 'hourly', 'Persistent': True}}})

TASK [systemd_service : Place the systemd timer] *************************************************************************************************************************************************************
ok: [c2] => (item={'service_name': 'cinder-volume-usage-audit', 'execstarts': ['/bin/bash -c "/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume-usage-audit --start_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00\\s-d\\s-1hour)\\" --end_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00)\\" --send_actions"'], 'timer': {'state': 'stopped', 'options': {'OnBootSec': '30min', 'OnCalendar': 'hourly', 'Persistent': True}}})

TASK [systemd_service : Place the systemd socket] ************************************************************************************************************************************************************

TASK [systemd_service : Reload systemd on unit change] *******************************************************************************************************************************************************

TASK [systemd_service : include_tasks] ***********************************************************************************************************************************************************************
included: /etc/ansible/roles/systemd_service/tasks/systemd_load.yml for c2 => (item={'service_name': 'cinder-volume-usage-audit', 'execstarts': ['/bin/bash -c "/openstack/venvs/cinder-24.1.0.dev184/bin/cinder-volume-usage-audit --start_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00\\s-d\\s-1hour)\\" --end_time \\"$$(date\\s+%%Y-%%m-%%d\\\\\\s%%H:00:00)\\" --send_actions"'], 'timer': {'state': 'stopped', 'options': {'OnBootSec': '30min', 'OnCalendar': 'hourly', 'Persistent': True}}})

TASK [systemd_service : Load service cinder-volume-usage-audit] **********************************************************************************************************************************************
ok: [c2] => (item=)

TASK [systemd_service : Load timer cinder-volume-usage-audit] ************************************************************************************************************************************************
ok: [c2] => (item=)

TASK [systemd_service : Load socket] *************************************************************************************************************************************************************************

TASK [os_cinder : Perform a cinder DB sync] ******************************************************************************************************************************************************************

TASK [os_cinder : Perform online data migrations] ************************************************************************************************************************************************************
skipping: [c2]

TASK [os_cinder : Disable the online migrations requirement] *************************************************************************************************************************************************

TASK [uwsgi : Gather variables for each operating system] ****************************************************************************************************************************************************

TASK [uwsgi : Gather variables for installation method] ******************************************************************************************************************************************************

TASK [uwsgi : Create uWSGI system group] *********************************************************************************************************************************************************************

TASK [uwsgi : Create uWSGI system user] **********************************************************************************************************************************************************************

TASK [uwsgi : Install distro packages] ***********************************************************************************************************************************************************************
skipping: [c2]

TASK [python_venv_build : Gather variables for each operating system] ****************************************************************************************************************************************

TASK [python_venv_build : gather build target facts] *********************************************************************************************************************************************************
skipping: [c2] => (item=c2_repo_container-46fbf376)

TASK [python_venv_build : Fail if wheels are not built for multiple hosts] ***********************************************************************************************************************************

TASK [python_venv_build : include_tasks] *********************************************************************************************************************************************************************

TASK [python_venv_build : Slurp up the constraints file for later re-deployment] *****************************************************************************************************************************
skipping: [c2]

TASK [python_venv_build : Install distro packages for venv build] ********************************************************************************************************************************************
skipping: [c2]

TASK [python_venv_build : Ensure a fresh venv_install_destination_path if venv_rebuild is enabled] ***********************************************************************************************************

TASK [python_venv_build : Create the venv_install_destination_path parent directory] *************************************************************************************************************************

TASK [python_venv_build : Create the virtualenv (if it does not exist)] **************************************************************************************************************************************

TASK [python_venv_build : Create the requirements file contents] *********************************************************************************************************************************************

TASK [python_venv_build : Build requirements file for the venv] **********************************************************************************************************************************************
skipping: [c2]

TASK [python_venv_build : Build global constraints file for the venv] ****************************************************************************************************************************************
skipping: [c2]

TASK [python_venv_build : Build constraints file for the venv] ***********************************************************************************************************************************************
skipping: [c2]

TASK [python_venv_build : Upgrade pip/setuptools/wheel to the versions we want] ******************************************************************************************************************************
skipping: [c2]

TASK [python_venv_build : Install python packages into the venv] *********************************************************************************************************************************************
skipping: [c2]

TASK [python_venv_build : Add symlinks from distribution python packages] ************************************************************************************************************************************

TASK [python_venv_build : Ensure local facts folder exists] **************************************************************************************************************************************************

TASK [python_venv_build : Record the necessary facts] ********************************************************************************************************************************************************
skipping: [c2] => (item={'section': 'uwsgi', 'option': 'venv_tag', 'value': '24.1.0.dev184'})

TASK [uwsgi : Ensure uWSGI directory exists] *****************************************************************************************************************************************************************

TASK [uwsgi : Deploy uWSGI configuration files] **************************************************************************************************************************************************************

TASK [Run the systemd service role] **************************************************************************************************************************************************************************

TASK [include_role : openstack.osa.service_setup] ************************************************************************************************************************************************************

TASK [apt_package_pinning : Add apt pin preferences] *********************************************************************************************************************************************************

TASK [ceph_client : End playbook if disabled] ****************************************************************************************************************************************************************
skipping: [c2]

TASK [ceph_client : Gather variables for each operating system] **********************************************************************************************************************************************
ok: [c2]

TASK [ceph_client : Fail if ceph_pkg_source uses an incorrect parameter] *************************************************************************************************************************************

TASK [ceph_client : include_tasks] ***************************************************************************************************************************************************************************
included: /etc/ansible/roles/ceph_client/tasks/ceph_preinstall_dnf.yml for c2

TASK [ceph_client : Copy EPEL gpg keyfile to the key location] ***********************************************************************************************************************************************
ok: [c2] => (item={'key': '/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8'})

TASK [ceph_client : Install EPEL gpg keys] *******************************************************************************************************************************************************************
[WARNING]: Using a variable for a task's 'args' is unsafe in some situations (see https://docs.ansible.com/ansible/devel/reference_appendices/faq.html#argsplat-unsafe)
ok: [c2] => (item={'key': '/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8'})

TASK [ceph_client : Install the EPEL repository] *************************************************************************************************************************************************************
ok: [c2]

TASK [ceph_client : Copy Ceph gpg keyfile to the key location] ***********************************************************************************************************************************************
ok: [c2] => (item={'key': '/etc/pki/rpm-gpg/ceph_com_keys_release'})

TASK [ceph_client : Install Ceph gpg keys] *******************************************************************************************************************************************************************
[WARNING]: Using a variable for a task's 'args' is unsafe in some situations (see https://docs.ansible.com/ansible/devel/reference_appendices/faq.html#argsplat-unsafe)
ok: [c2] => (item={'key': '/etc/pki/rpm-gpg/ceph_com_keys_release'})

TASK [ceph_client : Add ceph repo] ***************************************************************************************************************************************************************************
ok: [c2]

TASK [ceph_client : Add ceph noarch repo] ********************************************************************************************************************************************************************
ok: [c2]

TASK [ceph_client : include_tasks] ***************************************************************************************************************************************************************************
included: /etc/ansible/roles/ceph_client/tasks/ceph_install.yml for c2

TASK [ceph_client : Install ceph packages] *******************************************************************************************************************************************************************
ok: [c2]

TASK [ceph_client : include_tasks] ***************************************************************************************************************************************************************************
included: /etc/ansible/roles/ceph_client/tasks/ceph_install_python_libs.yml for c2

TASK [ceph_client : Find the venv's python version] **********************************************************************************************************************************************************
ok: [c2]

TASK [ceph_client : Set python venvs details] ****************************************************************************************************************************************************************
ok: [c2]

TASK [ceph_client : Register rados module path] **************************************************************************************************************************************************************
fatal: [c2]: FAILED! => {"changed": false, "cmd": ["/usr/bin/python3", "-c", "import rados; print(rados.__file__)"], "delta": "0:00:00.016340", "end": "2022-05-10 13:01:38.656841", "msg": "non-zero return code", "rc": 1, "start": "2022-05-10 13:01:38.640501", "stderr": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\nModuleNotFoundError: No module named 'rados'", "stderr_lines": ["Traceback (most recent call last):", " File \"<string>\", line 1, in <module>", "ModuleNotFoundError: No module named 'rados'"], "stdout": "", "stdout_lines": []}

PLAY RECAP ***************************************************************************************************************************************************************************************************
c2 : ok=87 changed=1 unreachable=0 failed=1 skipped=31 rescued=0 ignored=0
c3 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
c4 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
c5 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

EXIT NOTICE [Playbook execution failure] **************************************
===============================================================================

[Piotr (pszmydki)]

c2 - controller + compute + swift
c3 - compute + ceph
c4 - compute + cinder-lvm
c5 - compute + cinder_nfs

[Piotr (pszmydki)]

ceph status - ok
firewall -disabled

[Dmitriy Rabotyagov (noonedeadpunk)]

Can you please provide out put of following commands:

1. /usr/bin/python3 --version
2. /openstack/venvs/cinder-24.1.0.dev184/bin/python3 --version

[Piotr (pszmydki)]

[root@c2 /]# /usr/bin/python3 --version
Python 3.9.7

[root@c2 /]# /openstack/venvs/cinder-24.1.0.dev184/bin/python3 --version
Python 3.9.7

[Dmitriy Rabotyagov (noonedeadpunk)]

ok, that's interesting. I assumed that it's 3.8 tops in CentOS Stream.
At the same time, I guess that python3-cephfs/python3-rados/python3-rbd are available only for Python 3.6.

So I believe that in order for ceph libraries to be available and usable, you must use python3.6 as a default python3. You can run `alternatives --config python3` to change default python version.

Other way around would be building python3-ceph packages against python3.9 and maintaining a repo for these.

[Piotr (pszmydki)]

ERROR: Ansible requires Python 3.8 or newer on the controller. Current version: 3.6.8 (default, Mar 25 2022, 11:15:52) [GCC 8.5.0 20210514 (Red Hat 8.5.0-10)]

[Dmitriy Rabotyagov (noonedeadpunk)]

Um, are you installing from master? As on Xena we're using ansible-core==2.11.6 and on Wallaby we use ansible-base==2.10.10 which both work on py3.6 nicely.

On master though, we have a workaround in place, that does install python38 and use it, but only for ansible virtualenv:
https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/bootstrap-ansible.sh#L83

With that there should be no need to adjust default python version on hosts, as we explicitly passing python version while bootstrapping ansible venv.

Also I'm not sure where from you have pasted error?

[Piotr (pszmydki)]

The best solution is to deploy from another host?

[Dmitriy Rabotyagov (noonedeadpunk)]

Sorry, I'm not really understanding your question.

What I said is that both on Xena and Wallaby there's no requirement for python38 for ansible.
And on master we have a workaround already in code that will install and use required python version on deploy host. It should not hurt if deploy host and controller are same hosts though.

Nevertheless, default python version should be kept to default one, which is 3.6, since required python bindings are missing for all other versions.

[Piotr (pszmydki)]

I deploy openstack cluster from C2 host (controller),and i think the better solution is to deploy from outside

[Piotr (pszmydki)]

this is the same env but i deploy it from external host and when i run openstack-ansible setup-openstack.yml i have error: a secret with UUID 4fc20fcd-fcab-4c56-ac1d-5c9cf5a13934 already defined for use with client.cinder secret

TASK [ceph_client : Define libvirt nova secret] **************************************************************************************************************************************************************
fatal: [c3]: FAILED! => {"changed": true, "cmd": ["virsh", "secret-define", "--file", "/tmp/nova-secret.xml"], "delta": "0:00:00.040550", "end": "2022-05-10 21:06:21.291577", "msg": "non-zero return code", "rc": 1, "start": "2022-05-10 21:06:21.251027", "stderr": "error: Failed to set attributes from /tmp/nova-secret.xml\nerror: internal error: a secret with UUID 4fc20fcd-fcab-4c56-ac1d-5c9cf5a13934 already defined for use with client.cinder secret", "stderr_lines": ["error: Failed to set attributes from /tmp/nova-secret.xml", "error: internal error: a secret with UUID 4fc20fcd-fcab-4c56-ac1d-5c9cf5a13934 already defined for use with client.cinder secret"], "stdout": "", "stdout_lines": []}
fatal: [c5]: FAILED! => {"changed": true, "cmd": ["virsh", "secret-define", "--file", "/tmp/nova-secret.xml"], "delta": "0:00:00.020133", "end": "2022-05-10 21:06:21.436198", "msg": "non-zero return code", "rc": 1, "start": "2022-05-10 21:06:21.416065", "stderr": "error: Failed to set attributes from /tmp/nova-secret.xml\nerror: internal error: a secret with UUID 4fc20fcd-fcab-4c56-ac1d-5c9cf5a13934 already defined for use with client.cinder secret", "stderr_lines": ["error: Failed to set attributes from /tmp/nova-secret.xml", "error: internal error: a secret with UUID 4fc20fcd-fcab-4c56-ac1d-5c9cf5a13934 already defined for use with client.cinder secret"], "stdout": "", "stdout_lines": []}
fatal: [c4]: FAILED! => {"changed": true, "cmd": ["virsh", "secret-define", "--file", "/tmp/nova-secret.xml"], "delta": "0:00:00.041083", "end": "2022-05-10 21:06:21.488796", "msg": "non-zero return code", "rc": 1, "start": "2022-05-10 21:06:21.447713", "stderr": "error: Failed to set attributes from /tmp/nova-secret.xml\nerror: internal error: a secret with UUID 4fc20fcd-fcab-4c56-ac1d-5c9cf5a13934 already defined for use with client.cinder secret", "stderr_lines": ["error: Failed to set attributes from /tmp/nova-secret.xml", "error: internal error: a secret with UUID 4fc20fcd-fcab-4c56-ac1d-5c9cf5a13934 already defined for use with client.cinder secret"], "stdout": "", "stdout_lines": []}

[Dmitriy Rabotyagov (noonedeadpunk)]

I guess you have fully copied content of /etc/openstack_deploy?

We also store some facts in /etc/ansible, like flags that things are deployed and should not run more then once.
You should check it and likely copy to your new deploy host.

[Piotr (pszmydki)]

this is a new install: a new git clone on other host, new OS c2, lxc-destroy all containers.

[Piotr (pszmydki)]

only the same version: 24.1.0.dev184

[Piotr (pszmydki)]

- name: Provide xml file to create the secret
  template:
    src: secret.xml.j2
    dest: /tmp/nova-secret.xml
    mode: "0600"
  with_items:
    - secret_uuid: "{{ nova_ceph_client_uuid }}"
      client_name: "{{ nova_ceph_client }}"

Maybe ansible use the same uuid for two users (nova and cinder) ?

[Piotr (pszmydki)]

ceph of course :-)

[Piotr (pszmydki)]

Hello,
could you check my ansible settings for my lab cluster?
I want to have 3 cinder driver: ceph, lvm and nfs.
I have 4 servers: c2 ( with ip .2) sda - OS, sdb,sdc - Swift
c3 (with ip .3) sda- OS, sdb,sdc,sdd - CEPH
c4 (.4) sda- OS, sdb,sdc,sdd - LVM (cinder-volume)
c5 (.5) sda- OS, sdb,sdc,sdd - NFS ( for cindre nfs driver)
Every time when i deploy from ansible only ceph is working, but i see
configuration of lvm and nfs in cinder.conf and in dashboard.
Here is my ansible config: