Layered Build: Applying puppet ansible_bootstrap manifest failed

Bug #1877035 reported by chendongqi
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Critical
Austin Sun

Bug Description

Brief Description
-----------------
ansible apply puppet failed

Severity
--------
Critical:

Steps to Reproduce
------------------
Following the steps for installing StarlingX it fails at the point where we want to run ansible-playbook

Expected Behavior
------------------
Execute ansible-playbook pass

Actual Behavior
----------------
Fails to ansible-playbook

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Baremetal One node system

Branch/Pull Time/Commit
-----------------------
ISO="20200506T013246Z"

Timestamp/Logs
--------------
Please see attached:
 - puppet.log
 - sysinv.log
 - ansible.log
 - localhost.yml

Test Activity
-------------
Sanity

Workaround
----------

Revision history for this message
chendongqi (chen-dq) wrote :
Revision history for this message
chendongqi (chen-dq) wrote :
Revision history for this message
chendongqi (chen-dq) wrote :
Revision history for this message
chendongqi (chen-dq) wrote :
Revision history for this message
chendongqi (chen-dq) wrote :
Revision history for this message
Austin Sun (sunausti) wrote :

from puppet log
2020-05-06T09:23:08.861 ^[[0;36mDebug: 2020-05-06 09:23:08 +0000 Executing: '/usr/bin/openstack --os-interface internal role list --quiet --format csv'^[[0m
2020-05-06T09:23:09.653 ^[[0;36mDebug: 2020-05-06 09:23:09 +0000 Puppet::Provider::Openstack: Non-fatal error: 'Execution of '/usr/bin/openstack --os-interface internal role list --quiet --format csv' returned 1: Missing value auth-url required for auth plugin password'. Retrying for 169 more seconds^[[0m

2020-05-06T09:25:59.806 ^[[1;31mError: 2020-05-06 09:25:59 +0000 Failed to apply catalog: Execution of '/usr/bin/openstack --os-interface internal role list --quiet --format csv' returned 1: Missing value auth-url required for auth plugin password (tried 44, for a total of 170 seconds)

Revision history for this message
Nicolae Jascanu (njascanu-intel) wrote :

We confirm that the build is failing with the same error:

build.info
OS="centos"
SW_VERSION="20.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20200506T013246Z"

JOB="STX_build_layer_flock_master_master"
<email address hidden>"
BUILD_NUMBER="102"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2020-05-06 01:32:46 +0000"

FLOCK_OS="centos"
FLOCK_JOB="STX_build_layer_flock_master_master"
<email address hidden>"
FLOCK_BUILD_NUMBER="102"
FLOCK_BUILD_HOST="starlingx_mirror"
FLOCK_BUILD_DATE="2020-05-06 01:32:46 +0000"

from ansible.log:
2020-05-06 05:46:26,181 p=14108 u=sysadmin | fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/usr/local/bin/puppet-manifest-apply.sh", "/tmp/hieradata", "192.168.204.2", "controller", "ansible_bootstrap", ">", "/tmp/apply_manifest.log"], "delta": "0:04:19.184824", "end": "2020-05-06 05:46:26.166978", "msg": "non-zero return code", "rc": 1, "start": "2020-05-06 05:42:06.982154", "stderr": "cp: cannot stat ‘/tmp/hieradata/192.168.204.2.yaml’: No such file or directory\ncp: cannot stat ‘/tmp/hieradata/system.yaml’: No such file or directory\ncp: cannot stat ‘/tmp/hieradata/secure_system.yaml’: No such file or directory\ncp: cannot stat ‘>’: No such file or directory", "stderr_lines": ["cp: cannot stat ‘/tmp/hieradata/192.168.204.2.yaml’: No such file or directory", "cp: cannot stat ‘/tmp/hieradata/system.yaml’: No such file or directory", "cp: cannot stat ‘/tmp/hieradata/secure_system.yaml’: No such file or directory", "cp: cannot stat ‘>’: No such file or directory"], "stdout": "Applying puppet ansible_bootstrap manifest...\n[WARNING]\nWarnings found. See /var/log/puppet/2020-05-06-05-42-06_controller/puppet.log for details", "stdout_lines": ["Applying puppet ansible_bootstrap manifest...", "[WARNING]", "Warnings found. See /var/log/puppet/2020-05-06-05-42-06_controller/puppet.log for details"]}
2020-05-06 05:46:26,183 p=14108 u=sysadmin | PLAY RECAP *********************************************************************
2020-05-06 05:46:26,183 p=14108 u=sysadmin | localhost : ok=115 changed=30 unreachable=0 failed=1

from puppet.log
2020-05-06-05-42-06_controller/puppet.log:2020-05-06T05:46:22.461 Error: 2020-05-06 05:46:22 +0000 Failed to apply catalog: Execution of '/usr/bin/openstack --os-interface internal role list --quiet --format csv' returned 1: Missing value auth-url required for auth plugin password (tried 47, for a total of 170 seconds)
latest/puppet.log:2020-05-06T05:46:22.461 Error: 2020-05-06 05:46:22 +0000 Failed to apply catalog: Execution of '/usr/bin/openstack --os-interface internal role list --quiet --format csv' returned 1: Missing value auth-url required for auth plugin password (tried 47, for a total of 170 seconds)

Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Critical
tags: added: stx.4.0 stx.distro.openstack stx.sanity
Ghada Khalil (gkhalil)
Changed in starlingx:
status: New → Triaged
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Marking as critical given this is rendering the latest build unusable.

From the error above, it appears that the issue is related to openstack which was recently upversioned on the host from stein to train:
https://review.opendev.org/#/q/topic:train_upgrade+(status:open+OR+status:merged)

Changed in starlingx:
assignee: nobody → Austin Sun (sunausti)
Revision history for this message
Ghada Khalil (gkhalil) wrote :

As discussed in the community call on May 6, Austin will continue this investigation with Yan Chen who merged the openstack train upversion.

Revision history for this message
Yan Chen (ychen2u) wrote :

This issue should have been fixed by this patch: https://review.opendev.org/#/c/716842/
We will check if anything wrong with the layered build.

Revision history for this message
Nicolae Jascanu (njascanu-intel) wrote :

The SANITY baremetal on SIMPLEX failed with the same error:

OS="centos"
SW_VERSION="20.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20200507T015208Z"

JOB="STX_build_layer_flock_master_master"
<email address hidden>"
BUILD_NUMBER="103"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2020-05-07 01:52:08 +0000"

FLOCK_OS="centos"
FLOCK_JOB="STX_build_layer_flock_master_master"
<email address hidden>"
FLOCK_BUILD_NUMBER="103"
FLOCK_BUILD_HOST="starlingx_mirror"
FLOCK_BUILD_DATE="2020-05-07 01:52:08 +0000"

from ansible.log:
2020-05-07 03:39:11,993 p=14110 u=sysadmin | fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/usr/local/bin/puppet-manifest-apply.sh", "/tmp/hieradata", "192.168.204.2", "controller", "ansible_bootstrap", ">", "/tmp/apply_manifest.log"], "delta": "0:04:13.680889", "end": "2020-05-07 03:39:11.978494", "msg": "non-zero return code", "rc": 1, "start": "2020-05-07 03:34:58.297605", "stderr": "cp: cannot stat ‘/tmp/hieradata/192.168.204.2.yaml’: No such file or directory\ncp: cannot stat ‘/tmp/hieradata/system.yaml’: No such file or directory\ncp: cannot stat ‘/tmp/hieradata/secure_system.yaml’: No such file or directory\ncp: cannot stat ‘>’: No such file or directory", "stderr_lines": ["cp: cannot stat ‘/tmp/hieradata/192.168.204.2.yaml’: No such file or directory", "cp: cannot stat ‘/tmp/hieradata/system.yaml’: No such file or directory", "cp: cannot stat ‘/tmp/hieradata/secure_system.yaml’: No such file or directory", "cp: cannot stat ‘>’: No such file or directory"], "stdout": "Applying puppet ansible_bootstrap manifest...\n[WARNING]\nWarnings found. See /var/log/puppet/2020-05-07-03-34-58_controller/puppet.log for details", "stdout_lines": ["Applying puppet ansible_bootstrap manifest...", "[WARNING]", "Warnings found. See /var/log/puppet/2020-05-07-03-34-58_controller/puppet.log for details"]}
2020-05-07 03:39:11,994 p=14110 u=sysadmin | PLAY RECAP *********************************************************************
2020-05-07 03:39:11,995 p=14110 u=sysadmin | localhost : ok=115 changed=30 unreachable=0 failed=1

from /var/log/puppet/2020-05-07-03-34-58_controller/puppet.log
2020-05-07T03:39:08.093 Error: 2020-05-07 03:39:08 +0000 Failed to apply catalog: Execution of '/usr/bin/openstack --os-interface internal role list --quiet --format csv' returned 1: Missing value auth-url required for auth plugin password (tried 47, for a total of 170 seconds)

Revision history for this message
Austin Sun (sunausti) wrote :

@Ghada:
  As just sent to mail list abort this issue , need build team check cengn build env.

====================================================================

Hi All:
After further analysis this issue with Yan. Suspect there is bug hiding in cengn layer build.

Scott : would you like to check cengn build env ?

The details analysis :
Latest puppet-keystone rpm built from cengn should include change [1] which was merged by April 28th.
From distro layer build 0505 [3] and latest distro output , puppet-keystone includes change [1].
But if we extract puppet-keystone in 0506 iso built from layer build [2] or latest iso [5] , it is surprised this change [1] was not included. so bug#1877035 shows-up based on 0506 and 0507 iso.
If we extract puppet-keystone from 0506 iso built from monolithic [4], the change [1] was included in puppet-keystone, and ansible provision pass it.

BTW: layer build in my local setup (cross build flock or flock layer only build ) did not show this issue, so it should not be related to layer build script.

[1] https://review.opendev.org/#/c/716842/
[2] http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20200506T013246Z/outputs/iso/bootimage.iso
[3] http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/20200505T130946Z/outputs/RPMS/std/puppet-keystone-11.3.0-1.el7.tis.9.noarch.rpm
[4] http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/monolithic/20200506T080014Z/outputs/iso/bootimage.iso
[5] http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20200507T015208Z/outputs/iso/bootimage.iso
[6] http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/std/puppet-keystone-11.3.0-1.el7.tis.9.noarch.rpm

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to integ (master)

Reviewed: https://review.opendev.org/726219
Committed: https://git.openstack.org/cgit/starlingx/integ/commit/?id=16ec1ab35e94a1dfa7a66c876ea7537d53a2c691
Submitter: Zuul
Branch: master

commit 16ec1ab35e94a1dfa7a66c876ea7537d53a2c691
Author: Saul Wold <email address hidden>
Date: Thu May 7 09:57:42 2020 -0700

    Fix TIS_PATCH_VER for keystone and openstacklib

    Packages were not getting versioned correctly after they were updated
    because the TIS_PATCH_VER was not updated. When we update a package
    the TIS_PATCH_VER needs to be updated otherwise the build system will
    not see the update correctly.

    Build packages, ISO verified updated packages installed and system ran
    ansible-playbook correctly.

    Closes-Bug: 1877035
    Change-Id: Ifd38b42b0403796bb2445e433cef43100a5434ef
    Signed-off-by: Saul Wold <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Ghada Khalil (gkhalil)
tags: added: stx.build
Ghada Khalil (gkhalil)
summary: - applying puppet ansible_bootstrap manifest failed
+ Layered Build: Applying puppet ansible_bootstrap manifest failed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to integ (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/729834

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to integ (f/centos8)
Download full text (15.2 KiB)

Reviewed: https://review.opendev.org/729834
Committed: https://git.openstack.org/cgit/starlingx/integ/commit/?id=e4d12decc4c702e1e908d2430c7b4bc524c31c07
Submitter: Zuul
Branch: f/centos8

commit 5bb777d3725a48bc18431daedb6fd67198cd053a
Author: SidneyAn <email address hidden>
Date: Wed May 20 23:22:06 2020 +0800

    Add python-daemon to srpm list

    add python-daemon-2.2.3-7.el8.src.rpm to srpm list
    for pkg python3-daemon building.

    Change-Id: I0ad60d1083222130e72f935e08f97a8608b75880
    Story: 2007106
    Task: 39291
    Signed-off-by: SidneyAn <email address hidden>

commit fc125a7a24c00850aafd4a791a63e8e627b5ee1e
Author: Ran An <email address hidden>
Date: Thu May 14 11:41:50 2020 +0000

    Revert "Add python3-daemon required by logmgmt"

    This reverts commit 97cd7ea5c1037dd22488793ea9271462fedc4c7a.

    Change-Id: I3f09054c1546252493f8eb29dc70806829324a52

commit 97cd7ea5c1037dd22488793ea9271462fedc4c7a
Author: SidneyAn <email address hidden>
Date: Fri Apr 3 15:48:09 2020 +0800

    Add python3-daemon required by logmgmt

    pkg logmgmt upgraded to python3 requires python3 model "daemon",
    and no pkgs in Centos7 offical repo provide it.

    this patch refer to the python3-daemon pkg build by rdo
    for CentOS 8: python-daemon-2.2.3-7.el8.src.rpm

    disable the rpm check part which is not required in stx to
    reduce python3 dependencies that not supported by CentOS 7

    Depends-on: https://review.opendev.org/#/c/727657/
    Depends-on: https://review.opendev.org/#/c/727662/
    Change-Id: Ie08ea9c7adf830ad4e8e924fa69352fb2a923a6f
    Story: 2007106
    Task: 39291
    Signed-off-by: SidneyAn <email address hidden>

commit e2dc5c2dd0042788697ade268ac5c24fe9dc2f8c
Author: Steven Webster <email address hidden>
Date: Tue May 12 10:32:21 2020 -0400

    Fix sriov device plugin image build

    Previous commit d204f10ab5 introduced a build script to assist
    in building the SR-IOV device plugin.

    The script utilizes a Makefile to do build the plugin binary,
    then the image.

    Building the binary depends on go being present on the host. If it
    is not, the build will fail.

    Building the binary is actually not required, as it will be also
    done in a container as part of the 'make image', rather than copying
    the binary from the host.

    Closes-Bug: #1878224
    Change-Id: I4499ea2bbef4b3da8a154c69a07b415574517500
    Signed-off-by: Steven Webster <email address hidden>

commit d204f10ab53414dd46d5eb51fd99950d3ab70fa8
Author: Steven Webster <email address hidden>
Date: Fri Apr 24 10:59:59 2020 -0400

    Uprev the SR-IOV device plugin to the latest version

    This is intended primarily to pick up support for SR-IOV
    accelerators.

    The builder has been changed to a script model, as the
    device plugin's Dockerfile has been moved to a separate
    directory. The build-stx-images script does find this file,
    but the docker build will fail as the device plugin's
    source directory is no longer where the builder expects
    it to be. Instead, use the existing Makefile to assist
    in building the bi...

tags: added: in-f-centos8
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.