s037 updates: Failed to update nodes - Controller
While fixing
https://launchpad.net/bugs/1783399 --> TAGS regression which stopped updates to run in ci jobs
and
https://launchpad.net/bugs/1783857 --> false positive that reports SUCCESS for failed jobs/playbooks
Found this issue:
fs037 updates job is consistently failing at the same point with the same error.
***Not sure this was introduced when gate was reporting false positives.
However, for some reason this is a successful update from patchset3:
http://logs.openstack.org/28/585528/3/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/2e91dbe/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz
and it is also getting success here in this other patch:
http://logs.openstack.org/35/586235/2/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/f9ee659/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz
2018-07-26 16:37:22 | u'TASK [Set docker_startup_configs_with_default fact] ****************************',
2018-07-26 16:37:22 | u'Thursday 26 July 2018 16:37:20 +0000 (0:00:00.220) 0:02:55.340 ********* ',
2018-07-26 16:37:22 | u'An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ^',
2018-07-26 16:37:22 | u'fatal: [centos-7-rax-ord-0000988567]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}',
http://logs.openstack.org/28/585528/5/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/ee17474/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz#_2018-07-26_16_37_22
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack Traceback (most recent call last):
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack result = cmd.run(parsed_args)
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack super(Command, self).run(parsed_args)
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack return super(Command, self).run(parsed_args)
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack return_code = self.take_action(parsed_args) or 0
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_update.py", line 190, in take_action
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack parsed_args.ssh_user)
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/utils.py", line 959, in run_update_ansible_action
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack node_user=ssh_user, skip_tags=skip_tags)
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/package_update.py", line 98, in update_ansible
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack raise RuntimeError('Update failed with: {}'.format(payload))
2018-07-26 16:37:22 | 2018-07-26 16:37:22.981 8878 ERROR openstack RuntimeError: Update failed with: {u'status': u'FAILED', u'message': u'Failed to update nodes - Controller, please see the logs.', u'execution': {u'name': u'tripleo.package_update.v1.update_nodes', u'created_at': u'2018-07-26 16:34:18', u'updated_at': u'2018-07-26 16:37:21', u'id': u'fc5288d5-f461-40d0-af2f-902074d85561', u'params': {u'namespace': u'', u'env': {}}, u'input': {u'inventory_file': u'Undercloud:\n hosts:\n undercloud: {}\n vars:\n ansible_connection: local\n ansible_host: localhost\n ansible_remote_tmp: /tmp/ansible-${USER}\n auth_url: http://192.168.24.1:5000/\n cacert: null\n os_auth_token: gAAAAABbWfgEt5Gy1D4jmAOx9nRndzad7UBzCGebdpNnzjZTq8LqJJvY9nCr0cuIFaR12Fakg74uU7FJ3C2uDc3PoSR70U9CKaZ-tVU9jRRayMbhLOMPWLSXLcGliXlMDoULswbQnovB35mtuITvRMrYwHHgl-cQKZdqkmynG5_tHXAPpFB3-S0\n overcloud_admin_password: xmWu48EB1GLLPHUheidfbFtN8\n overcloud_horizon_url: http://192.168.24.14:80/dashboard\n overcloud_keystone_url: http://192.168.24.14:5000\n plan: overcloud\n project_name: admin\n undercloud_service_list: [openstack-nova-compute, openstack-heat-engine, openstack-ironic-conductor,\n openstack-swift-container, openstack-swift-object, openstack-mistral-engine]\n undercloud_swift_url: http://192.168.24.1:8080/v1/AUTH_efa64b61052648c182fe8a05fec844f8\n username: admin\nController:\n hosts:\n centos-7-rax-ord-0000988567:\n ansible_host: 192.168.24.3\n ctlplane_ip: 192.168.24.3\n deploy_server_id: 448f84fc-62a9-4cbb-815c-835b5607989c\n enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n tenant]\n external_ip: 192.168.24.3\n internal_api_ip: 192.168.24.3\n management_ip: 192.168.24.3\n storage_ip: 192.168.24.3\n storage_mgmt_ip: 192.168.24.3\n tenant_ip: 192.168.24.3\n vars: {ansible_ssh_user: tripleo-admin, bootstrap_server_id: 448f84fc-62a9-4cbb-815c-835b5607989c,\n tripleo_role_name: Controller}\novercloud:\n children:\n Controller: {}\n vars: {ctlplane_vip: 192.168.24.14, external_vip: 192.168.24.14, internal_api_vip: 192.168.24.14,\n redis_vip: 192.168.24.6, storage_mgmt_vip: 192.168.24.14, storage_vip: 192.168.24.14}\nhaproxy:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\nkernel:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\nsshd:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\ntripleo_firewall:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\noslo_messaging_notify:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\nmysql_client:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\nntp:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\nclustercheck:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\nsnmp:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\nkeystone:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\ntripleo_packages:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\npacemaker:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\nmysql:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\nca_certs:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\ntimezone:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\ndocker:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\nmemcached:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\noslo_messaging_rpc:\n children:\n Controller: {}\n vars: {ansible_ssh_user: tripleo-admin}\n', u'work_dir': u'/var/lib/mistral', u'verbosity': 1, u'skip_tags': u'', u'playbook': u'update_steps_playbook.yaml', u'ansible_extra_env_variables': {u'ANSIBLE_HOST_KEY_CHECKING': u'False', u'ANSIBLE_LOG_PATH': u'/var/log/mistral/package_update.log'}, u'module_path': u'/usr/share/ansible-modules', u'nodes': u'Controller', u'node_user': u'tripleo-admin', u'ansible_queue_name': u'update'}, u'spec': {u'tasks': {u'get_private_key': {u'name': u'get_private_key', u'on-success': u'node_update', u'publish': {u'private_key': u'<% task().result %>'}, u'version': u'2.0', u'action': u'tripleo.validations.get_privkey', u'type': u'direct'}, u'node_update_failed': {u'version': u'2.0', u'type': u'direct', u'name': u'node_update_failed', u'publish': {u'status': u'FAILED', u'message': u'Failed to update nodes - <% $.nodes %>, please see the logs.'}, u'on-success': u'send_message'}, u'node_update_passed': {u'version': u'2.0', u'type': u'direct', u'name': u'node_update_passed', u'publish': {u'status': u'SUCCESS', u'message': u'Updated nodes - <% $.nodes %>'}, u'on-success': u'send_message'}, u'send_message': {u'input': {u'status': u"<% $.get('status', 'SUCCESS') %>", u'message': u"<% $.get('message', '') %>", u'queue_name': u'<% $.ansible_queue_name %>', u'type': u'<% execution().name %>', u'execution': u'<% execution() %>'}, u'version': u'2.0', u'type': u'direct', u'name': u'send_message', u'workflow': u'tripleo.messaging.v1.send'}, u'node_update': {u'name': u'node_update', u'on-error': u'node_update_failed', u'on-success': [{u'node_update_passed': u'<% task().result.returncode = 0 %>'}, {u'node_update_failed': u'<% task().result.returncode != 0 %>'}], u'publish': {u'output': u'<% task().result %>'}, u'version': u'2.0', u'action': u'tripleo.ansible-playbook', u'input': {u'remote_user': u'<% $.node_user %>', u'limit_hosts': u'<% $.nodes %>', u'become_user': u'root', u'verbosity': u'<% $.verbosity %>', u'queue_name': u'<% $.ansible_queue_name %>', u'extra_env_variables': u'<% $.ansible_extra_env_variables %>', u'skip_tags': u'<% $.skip_tags %>', u'inventory': u'<% $.inventory_file %>', u'become': True, u'module_path': u'<% $.module_path %>', u'playbook': u'<% $.work_dir %>/<% execution().id %>/<% $.playbook %>', u'trash_output': True, u'execution_id': u'<% execution().id %>', u'ssh_private_key': u'<% $.private_key %>'}, u'type': u'direct'}, u'download_config': {u'name': u'download_config', u'on-error': u'node_update_failed', u'on-success': u'get_private_key', u'version': u'2.0', u'action': u'tripleo.config.download_config', u'input': {u'work_dir': u'<% $.work_dir %>/<% execution().id %>'}, u'type': u'direct'}}, u'description': u'Take a container and perform an update nodes by nodes', u'tags': [u'tripleo-common-managed'], u'version': u'2.0', u'input': [{u'node_user': u'tripleo-admin'}, u'nodes', u'playbook', u'inventory_file', {u'ansible_queue_name': u'tripleo'}, {u'module_path': u'/usr/share/ansible-modules'}, {u'ansible_extra_env_variables': {u'ANSIBLE_HOST_KEY_CHECKING': u'False', u'ANSIBLE_LOG_PATH': u'/var/log/mistral/package_update.log'}}, {u'verbosity': 1}, {u'work_dir': u'/var/lib/mistral'}, {u'skip_tags': u''}], u'name': u'update_nodes'}}, u'plan_name': None, u'execution_id': u'fc5288d5-f461-40d0-af2f-902074d85561', u'deployment_status': None}
@rlandy found this:
https:/
start_order: 1
user: root
'start_order' looks misplaced...
a successful job has:
https:/