Description:
After StarlingX config repo sync, bootstrap playbook failed during persist-config step
TASK [persist-config : debug] *********************************************************************************************************************************************************
ok: [wc68] => {
"populate_result": {
"changed": true,
"failed": false,
"failed_when_result": false,
"msg": "non-zero return code",
"rc": 1,
"stderr": "Shared connection to 128.224.151.38 closed.\r\n",
"stderr_lines": [
"Shared connection to 128.224.151.38 closed."
],
"stdout": "\r\nPopulating system config...\r\nPopulating load config...\r\nPopulating management network...\r\nPopulating pxeboot network...\r\nPopulating oam network...\r\nPopulating multicast network...\r\nPopulating cluster host network...\r\nPopulating cluster pod network...\r\nPopulating cluster service network...\r\nNetwork config completed.\r\nPopulating DNS config...\r\nPopulating docker config...\r\nManagement mac = 00:00:00:00:00:00\r\nRoot fs device = /dev/disk/by-path/pci-0000:00:1f.2-ata-1.0\r\nBoot device = /dev/disk/by-path/pci-0000:00:1f.2-ata-1.0\r\nConsole = ttyS0,115200n8\r\nTboot = false\r\nInstall output = text\r\nHost values = {'tboot': 'false', 'install_output': 'text', 'rootfs_device': '/dev/disk/by-path/pci-0000:00:1f.2-ata-1.0', 'boot_device': '/dev/disk/by-path/pci-0000:00:1f.2-ata-1.0', 'availability': 'offline', 'mgmt_mac': '00:00:00:00:00:00', 'console': 'ttyS0,115200n8', 'mgmt_ip': '192.168.204.3', 'hostname': 'controller-0', 'operational': 'disabled', 'invprovision': 'provisioning', 'administrative': 'locked', 'personality': 'controller'}\r\nNo handlers could be found for logger \"cgtsclient.common.http\"\r\nFailed to update the initial system config.\r\nTraceback (most recent call last):\r\n File \"/tmp/.ansible-wrsroot/tmp/ansible-tmp-1559924824.42-23280222972029/populate_initial_config.py\", line 780, in <module>\r\n controller = populate_controller_config(client)\r\n File \"/tmp/.ansible-wrsroot/tmp/ansible-tmp-1559924824.42-23280222972029/populate_initial_config.py\", line 672, in populate_controller_config\r\n controller = client.sysinv.ihost.create(**values)\r\n File \"/usr/lib64/python2.7/site-packages/cgtsclient/v1/ihost.py\", line 74, in create\r\n return self._create(self._path(), new)\r\n File \"/usr/lib64/python2.7/site-packages/cgtsclient/common/base.py\", line 51, in _create\r\n _, body = self.api.json_request('POST', url, body=body)\r\n File \"/usr/lib64/python2.7/site-packages/cgtsclient/common/http.py\", line 243, in json_request\r\n method, **kwargs)\r\n File \"/usr/lib64/python2.7/site-packages/cgtsclient/common/http.py\", line 219, in _cs_request\r\n error_json.get('debuginfo'), *args)\r\ncgtsclient.exc.HTTPInternalServerError: 'module' object has no attribute 'is_openstack_installed'\r\nTraceback (most recent call last):\r\n\r\n File \"/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py\", line 438, in _process_data\r\n **args)\r\n\r\n File \"/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py\", line 172, in dispatch\r\n result = getattr(proxyobj, method)(ctxt, **kwargs)\r\n\r\n File \"/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py\", line 1686, in configure_ihost\r\n self._puppet.update_system_config()\r\n\r\n File \"/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py\", line 30, in _wrapper\r\n func(self, *args, **kwargs)\r\n\r\n File \"/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py\", line 115, in update_system_config\r\n config.update(puppet_plugin.obj.get_system_config())\r\n\r\n File \"/usr/lib64/python2.7/site-packages/sysinv/puppet/keystone.py\", line 130, in get_system_config\r\n if utils.is_openstack_installed(self.dbapi):\r\n\r\nAttributeError: 'module' object has no attribute 'is_openstack_installed'\r\n\r\n",
"stdout_lines": [
"",
"Populating system config...",
"Populating load config...",
"Populating management network...",
"Populating pxeboot network...",
"Populating oam network...",
"Populating multicast network...",
"Populating cluster host network...",
"Populating cluster pod network...",
"Populating cluster service network...",
"Network config completed.",
"Populating DNS config...",
"Populating docker config...",
"Management mac = 00:00:00:00:00:00",
"Root fs device = /dev/disk/by-path/pci-0000:00:1f.2-ata-1.0",
"Boot device = /dev/disk/by-path/pci-0000:00:1f.2-ata-1.0",
"Console = ttyS0,115200n8",
"Tboot = false",
"Install output = text",
"Host values = {'tboot': 'false', 'install_output': 'text', 'rootfs_device': '/dev/disk/by-path/pci-0000:00:1f.2-ata-1.0', 'boot_device': '/dev/disk/by-path/pci-0000:00:1f.2-ata-1.0', 'availability': 'offline', 'mgmt_mac': '00:00:00:00:00:00', 'console': 'ttyS0,115200n8', 'mgmt_ip': '192.168.204.3', 'hostname': 'controller-0', 'operational': 'disabled', 'invprovision': 'provisioning', 'administrative': 'locked', 'personality': 'controller'}",
"No handlers could be found for logger \"cgtsclient.common.http\"",
"Failed to update the initial system config.",
"Traceback (most recent call last):",
" File \"/tmp/.ansible-wrsroot/tmp/ansible-tmp-1559924824.42-23280222972029/populate_initial_config.py\", line 780, in <module>",
" controller = populate_controller_config(client)",
" File \"/tmp/.ansible-wrsroot/tmp/ansible-tmp-1559924824.42-23280222972029/populate_initial_config.py\", line 672, in populate_controller_config",
" controller = client.sysinv.ihost.create(**values)",
" File \"/usr/lib64/python2.7/site-packages/cgtsclient/v1/ihost.py\", line 74, in create",
" return self._create(self._path(), new)",
" File \"/usr/lib64/python2.7/site-packages/cgtsclient/common/base.py\", line 51, in _create",
" _, body = self.api.json_request('POST', url, body=body)",
" File \"/usr/lib64/python2.7/site-packages/cgtsclient/common/http.py\", line 243, in json_request",
" method, **kwargs)",
" File \"/usr/lib64/python2.7/site-packages/cgtsclient/common/http.py\", line 219, in _cs_request",
" error_json.get('debuginfo'), *args)",
"cgtsclient.exc.HTTPInternalServerError: 'module' object has no attribute 'is_openstack_installed'",
"Traceback (most recent call last):",
"",
" File \"/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py\", line 438, in _process_data",
" **args)",
"",
" File \"/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py\", line 172, in dispatch",
" result = getattr(proxyobj, method)(ctxt, **kwargs)",
"",
" File \"/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py\", line 1686, in configure_ihost",
" self._puppet.update_system_config()",
"",
" File \"/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py\", line 30, in _wrapper",
" func(self, *args, **kwargs)",
"",
" File \"/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py\", line 115, in update_system_config",
" config.update(puppet_plugin.obj.get_system_config())",
"",
" File \"/usr/lib64/python2.7/site-packages/sysinv/puppet/keystone.py\", line 130, in get_system_config",
" if utils.is_openstack_installed(self.dbapi):",
"",
"AttributeError: 'module' object has no attribute 'is_openstack_installed'",
""
]
}
}
TASK [persist-config : Fail if populate config script throws an exception] ************************************************************************************************************
fatal: [wc68]: FAILED! => {"changed": false, "msg": "Failed to provision initial system configuration."}
PLAY RECAP ****************************************************************************************************************************************************************************
wc68 : ok=104 changed=36 unreachable=0 failed=1
Issue is caused by
Commit [1]which just merged this morning was using a method that was renamed by Commit [2]
[1] https://review.opendev.org/#/c/657383/
[2] https://review.opendev.org/#/c/658193/
Severity:
Critical
Steps to reproduce:
Sync config repo to latest
Make a build
Install and run bootstrap playbook
Expected behavior:
Bootstrap completes successfully
Actual behavior:
Controller-0 failed to bootstrap due to the issue above
Reproducibility:
100%
Repo/Branch/Pull time:
config/master/May 7th around 12pm EST
Marking as release gating; critical priority given system deployments are failing