periodic fedora 28 job failing with "/bin/sh: line 1: exit: null: numeric argument required" in Run async deployment StandalonePostDeployment step

Bug #1812837 reported by Arx Cruz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Emilien Macchi

Bug Description

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-centos-7-containers-standalone-master/54b8e6b/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

2019-01-22 07:16:35 | TASK [Run deployment StandalonePostDeployment] *********************************
2019-01-22 07:16:36 | fatal: [upstream-fedora-28-rdo-cloud-tripleo-0000410235]: FAILED! => {"changed": true, "cmd": "/usr/libexec/os-refresh-config/configure.d/55-heat-config\n exit $(jq .deploy_status_code /var/lib/heat-config/deployed/66e37616-2aff-44c6-9458-0d9e1558eb44.notify.json)", "delta": "0:00:00.835319", "end": "2019-01-22 07:16:36.923527", "msg": "non-zero return code", "rc": 2, "start": "2019-01-22 07:16:36.088208", "stderr": "[2019-01-22 07:16:36,154] (heat-config) [DEBUG] Running /usr/libexec/heat-config/hooks/ansible < /var/lib/heat-config/deployed/66e37616-2aff-44c6-9458-0d9e1558eb44.json\n[2019-01-22 07:16:36,197] (heat-config) [INFO] !!binary \"\"\n\n[2019-01-22 07:16:36,198] (heat-config) [DEBUG] b'[2019-01-22 07:16:36,192] (heat-config) [DEBUG] Running ansible-playbook -i localhost, /var/lib/heat-config/heat-config-ansible/66e37616-2aff-44c6-9458-0d9e1558eb44_playbook.yaml --extra-vars @/var/lib/heat-config/heat-config-ansible/66e37616-2aff-44c6-9458-0d9e1558eb44_variables.json\\n[2019-01-22 07:16:36,193] (heat-config) [WARNING] ansible not installed yet\\n'\n[2019-01-22 07:16:36,198] (heat-config) [INFO] Completed /usr/libexec/heat-config/hooks/ansible\n[2019-01-22 07:16:36,198] (heat-config) [DEBUG] Running heat-config-notify /var/lib/heat-config/deployed/66e37616-2aff-44c6-9458-0d9e1558eb44.json < /var/lib/heat-config/deployed/66e37616-2aff-44c6-9458-0d9e1558eb44.notify.json\n[2019-01-22 07:16:36,912] (heat-config) [INFO] b''\n[2019-01-22 07:16:36,912] (heat-config) [DEBUG] b''\n/bin/sh: line 1: exit: null: numeric argument required", "stderr_lines": ["[2019-01-22 07:16:36,154] (heat-config) [DEBUG] Running /usr/libexec/heat-config/hooks/ansible < /var/lib/heat-config/deployed/66e37616-2aff-44c6-9458-0d9e1558eb44.json", "[2019-01-22 07:16:36,197] (heat-config) [INFO] !!binary \"\"", "", "[2019-01-22 07:16:36,198] (heat-config) [DEBUG] b'[2019-01-22 07:16:36,192] (heat-config) [DEBUG] Running ansible-playbook -i localhost, /var/lib/heat-config/heat-config-ansible/66e37616-2aff-44c6-9458-0d9e1558eb44_playbook.yaml --extra-vars @/var/lib/heat-config/heat-config-ansible/66e37616-2aff-44c6-9458-0d9e1558eb44_variables.json\\n[2019-01-22 07:16:36,193] (heat-config) [WARNING] ansible not installed yet\\n'", "[2019-01-22 07:16:36,198] (heat-config) [INFO] Completed /usr/libexec/heat-config/hooks/ansible", "[2019-01-22 07:16:36,198] (heat-config) [DEBUG] Running heat-config-notify /var/lib/heat-config/deployed/66e37616-2aff-44c6-9458-0d9e1558eb44.json < /var/lib/heat-config/deployed/66e37616-2aff-44c6-9458-0d9e1558eb44.notify.json", "[2019-01-22 07:16:36,912] (heat-config) [INFO] b''", "[2019-01-22 07:16:36,912] (heat-config) [DEBUG] b''", "/bin/sh: line 1: exit: null: numeric argument required"], "stdout": "", "stdout_lines": []}
2019-01-22 07:16:36 | ...ignoring
2019-01-22 07:16:36 |
2019-01-22 07:16:36 | TASK [Run async deployment StandalonePostDeployment] ***************************
2019-01-22 07:16:36 | skipping: [upstream-fedora-28-rdo-cloud-tripleo-0000410235]
2019-01-22 07:16:37 |
2019-01-22 07:16:37 | TASK [Output for sync deployment StandalonePostDeployment] *********************
2019-01-22 07:16:37 | fatal: [upstream-fedora-28-rdo-cloud-tripleo-0000410235]: FAILED! => {
2019-01-22 07:16:37 | "msg": [
2019-01-22 07:16:37 | {
2019-01-22 07:16:37 | "stderr": [
2019-01-22 07:16:37 | "[2019-01-22 07:16:36,154] (heat-config) [DEBUG] Running /usr/libexec/heat-config/hooks/ansible < /var/lib/heat-config/deployed/66e37616-2aff-44c6-9458-0d9e1558eb44.json",
2019-01-22 07:16:37 | "[2019-01-22 07:16:36,197] (heat-config) [INFO] !!binary \"\"",
2019-01-22 07:16:37 | "",
2019-01-22 07:16:37 | "[2019-01-22 07:16:36,198] (heat-config) [DEBUG] b'[2019-01-22 07:16:36,192] (heat-config) [DEBUG] Running ansible-playbook -i localhost, /var/lib/heat-config/heat-config-ansible/66e37616-2aff-44c6-9458-0d9e1558eb44_playbook.yaml --extra-vars @/var/lib/heat-config/heat-config-ansible/66e37616-2aff-44c6-9458-0d9e1558eb44_variables.json\\n[2019-01-22 07:16:36,193] (heat-config) [WARNING] ansible not installed yet\\n'",
2019-01-22 07:16:37 | "[2019-01-22 07:16:36,198] (heat-config) [INFO] Completed /usr/libexec/heat-config/hooks/ansible",
2019-01-22 07:16:37 | "[2019-01-22 07:16:36,198] (heat-config) [DEBUG] Running heat-config-notify /var/lib/heat-config/deployed/66e37616-2aff-44c6-9458-0d9e1558eb44.json < /var/lib/heat-config/deployed/66e37616-2aff-44c6-9458-0d9e1558eb44.notify.json",
2019-01-22 07:16:37 | "[2019-01-22 07:16:36,912] (heat-config) [INFO] b''",
2019-01-22 07:16:37 | "[2019-01-22 07:16:36,912] (heat-config) [DEBUG] b''",
2019-01-22 07:16:37 | "/bin/sh: line 1: exit: null: numeric argument required"
2019-01-22 07:16:37 | ]
2019-01-22 07:16:37 | },
2019-01-22 07:16:37 | {
2019-01-22 07:16:37 | "status_code": "2"
2019-01-22 07:16:37 | }
2019-01-22 07:16:37 | ]
2019-01-22 07:16:37 | }
2019-01-22 07:16:37 |
2019-01-22 07:16:37 | NO MORE HOSTS LEFT *************************************************************

First time this is happening

Arx Cruz (arxcruz)
tags: removed: alert
Changed in tripleo:
importance: Undecided → High
Revision history for this message
wes hayutin (weshayutin) wrote :
Changed in tripleo:
status: Triaged → Fix Released
Revision history for this message
Alex Schultz (alex-schultz) wrote :

We need to revert the proposed change because it breaks fedora packaging. https://review.openstack.org/#/c/632588/

Changed in tripleo:
status: Fix Released → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-tripleoclient (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/632756

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-puppet-elements (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/632766

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/632768

Changed in tripleo:
status: Triaged → In Progress
assignee: nobody → Emilien Macchi (emilienm)
Revision history for this message
wes hayutin (weshayutin) wrote :
Changed in tripleo:
status: In Progress → Fix Released
tags: removed: ci promotion-blocker
Changed in tripleo:
status: Fix Released → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/632768
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7d3c7b16bb62da7c5d184cfdff8484a85fca6e8e
Submitter: Zuul
Branch: master

commit 7d3c7b16bb62da7c5d184cfdff8484a85fca6e8e
Author: Emilien Macchi <email address hidden>
Date: Wed Jan 23 10:55:19 2019 -0500

    deployed-servers: symlink ansible-playbook

    In RHEL8, we'll have /usr/bin/ansible-playbook provided by ansible package.
    Everywhere in TripleO we rely on /usr/bin/ansible-playbook but the
    ansible-python3 package in Fedora 28 doesn't provide it.

    Until it does, we have a temporary workaround that will create a symlink only
    if the file doesn't exist and if ansible-playbook3 is here.

    Change-Id: I520bf0d665fa9a331fe94168595d3b09dbf1795c
    Related-Bug: #1812837

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-puppet-elements (master)

Reviewed: https://review.openstack.org/632766
Committed: https://git.openstack.org/cgit/openstack/tripleo-puppet-elements/commit/?id=5e6c7d96e361452977cd53cede6e2bb4ff966c21
Submitter: Zuul
Branch: master

commit 5e6c7d96e361452977cd53cede6e2bb4ff966c21
Author: Emilien Macchi <email address hidden>
Date: Wed Jan 23 10:51:11 2019 -0500

    overcloud-agent: create ansible-playbook symlinks in post-install

    overcloud-agent element takes care of deploying the heat-agents that
    depend on Ansible.

    This temporary workaround makes sure:
    - ansible-playbook exists if ansible-playbook-3 only is installed (f28)
    - ansible-playbook-3 exists if ansible-playbook only is installed (el8)

    Change-Id: Iad05160afb700a18caf5d3584c2826857412a3fa
    Related-Bug: #1812837

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to python-tripleoclient (master)

Reviewed: https://review.openstack.org/632756
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=9e500285f0f31fc122ddbf7d4cef7617c728f7e4
Submitter: Zuul
Branch: master

commit 9e500285f0f31fc122ddbf7d4cef7617c728f7e4
Author: Emilien Macchi <email address hidden>
Date: Wed Jan 23 10:25:31 2019 -0500

    tripleo_deploy: symlink ansible-playbook

    In RHEL8, we'll have /usr/bin/ansible-playbook provided by ansible
    package.
    Everywhere in TripleO we rely on /usr/bin/ansible-playbook but the
    ansible-python3 package in Fedora 28 doesn't provide it.
    Until it does, we have a temporary workaround that will create a symlink
    only if the file doesn't exist and if ansible-playbook3 is here.

    Change-Id: I040345ffb76a14035e6100c4e61a9aea07b690c6
    Related-Bug: #1812837

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.