controller-1 unlock failed by "Failed to generate bootstrap token"

Bug #1876755 reported by Peng Peng
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
High
John Kung

Bug Description

Brief Description
-----------------
During system initial, controller-1 unlock failed by " Failed to generate bootstrap token"

Severity
--------
Major

Steps to Reproduce
------------------
setup multi nodes system, host-unlock controller-1

Expected Behavior
------------------
host-unlock successfully

Actual Behavior
----------------
host-unlock failed

Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor

System Configuration
--------------------
Multi-node system

Lab-name: wcp_71-75

Branch/Pull Time/Commit
-----------------------
2020-05-03_20-00-00

Last Pass
---------
2020-05-01_20-00-00

Timestamp/Logs
--------------
system host-unlock controller-1
Remote error: SysinvException Failed to generate bootstrap token
[u'Traceback (most recent call last):\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data\n **args)\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n result = getattr(proxyobj, method)(ctxt, **kwargs)\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1690, in configure_ihost\n self._configure_controller_host(context, host)\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1324, in _configure_controller_host\n self._puppet.update_host_config(host)\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 31, in _wrapper\n func(self, *args, **kwargs)\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 148, in update_host_config\n config.update(puppet_plugin.obj.get_host_config(host))\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/puppet/kubernetes.py", line 75, in get_host_config\n config.update(self._get_host_join_command(host))\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/puppet/kubernetes.py", line 141, in _get_host_join_command\n join_cmd = self._get_kubernetes_join_cmd(host)\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/puppet/kubernetes.py", line 196, in _get_kubernetes_join_cmd\n \'Failed to generate bootstrap token\')\n', u'SysinvException: Failed to generate bootstrap token\n'].

Test Activity
-------------
Installation

Revision history for this message
Peng Peng (ppeng) wrote :
tags: added: stx.retestneeded
Revision history for this message
Ghada Khalil (gkhalil) wrote :

stx.4.0 / high priority - unlock fails and is not recoverable

tags: added: stx.4.0 stx.containers
Changed in starlingx:
importance: Undecided → High
status: New → Triaged
assignee: nobody → John Kung (john-kung)
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Assigning to John Kung to do initial triage

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/725401

Changed in starlingx:
status: Triaged → In Progress
Revision history for this message
Ghada Khalil (gkhalil) wrote :

This issue was introduced by the fix for https://bugs.launchpad.net/starlingx/+bug/1873916 which was merged on 2020-04-29

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (master)

Reviewed: https://review.opendev.org/725401
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=40463adf9aba476ba44b0dd89d4c30c9343b43b4
Submitter: Zuul
Branch: master

commit 40463adf9aba476ba44b0dd89d4c30c9343b43b4
Author: John Kung <email address hidden>
Date: Mon May 4 17:13:59 2020 -0400

    Fix certificate-key to 64 characters

    Update get_secure_static_config() to fix the
    kubernetes::kubeadm::certificate-key, to the 64 characters
    expected by kubernetes.

    Change-Id: I366e6eb1dc4e764425ef2a82a493db47a080f49a
    Closes-bug: 1876755
    Signed-off-by: John Kung <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
Peng Peng (ppeng) wrote :

Issue reproduced on
Issue was reproduced on
Lab: SM_3
Load: 2020-05-03_20-00-00

First initial unlock during installation was successfully
unlock during test was failed as below

log:
Details: CLI 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock controller-0' failed to execute. Output: Remote error: SysinvException Failed to generate bootstrap token
E [u'Traceback (most recent call last):\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data\n **args)\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n result = getattr(proxyobj, method)(ctxt, **kwargs)\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1690, in configure_ihost\n self._configure_controller_host(context, host)\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1324, in _configure_controller_host\n self._puppet.update_host_config(host)\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 31, in _wrapper\n func(self, *args, **kwargs)\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 148, in update_host_config\n config.update(puppet_plugin.obj.get_host_config(host))\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/puppet/kubernetes.py", line 75, in get_host_config\n config.update(self._get_host_join_command(host))\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/puppet/kubernetes.py", line 141, in _get_host_join_command\n join_cmd = self._get_kubernetes_join_cmd(host)\n', u' File "/usr/lib64/python2.7/site-packages/sysinv/puppet/kubernetes.py", line 196, in _get_kubernetes_join_cmd\n \'Failed to generate bootstrap token\')\n', u'SysinvException: Failed to generate bootstrap token\n'].

utils/cli.py:152: CLIRejected
[2020-05-05 07:18:20,516] 266 INFO MainThread conftest.testcase_log::

log added:
https://files.starlingx.kube.cengn.ca/launchpad/1876755

Revision history for this message
John Kung (john-kung) wrote :

Fix for this issue with the incorrect certificate-key was merged on May 5: https://review.opendev.org/#/c/725401/

Revision history for this message
Peng Peng (ppeng) wrote :

Verified on
Lab: WCP_71_75
Load: 2020-05-05_20-29-49

tags: removed: stx.retestneeded
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/729812

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (f/centos8)
Download full text (37.5 KiB)

Reviewed: https://review.opendev.org/729812
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=539d476456277c22d0dcbc3cbbc832e623242264
Submitter: Zuul
Branch: f/centos8

commit 320cc40de8518787c2be234d7fdf88ec0a462df2
Author: Don Penney <email address hidden>
Date: Wed May 13 13:06:11 2020 -0400

    Add auto-versioning to starlingx/config packages

    This update makes use of the PKG_GITREVCOUNT variable to auto-version
    the packages in this repo.

    Change-Id: I3a2c8caeb4b4647608978b1f2ccfcf0661508803
    Depends-On: https://review.opendev.org/727837
    Story: 2006166
    Task: 39766
    Signed-off-by: Don Penney <email address hidden>

commit d9f2aea0fb228ed69eb9c9262e29041eedabc15d
Author: Sharath Kumar K <email address hidden>
Date: Wed Apr 22 16:22:22 2020 +0200

    De-branding in starlingx/config: CGCS -> StarlingX

    1. Rename CGCS to StarlingX for .spec files

    Test:
    After the de-brand change, bootimage.iso has been built in the flock
    Layer and installed on the dev machine to validate the changes.

    Please note, doing de-brand changes in batches, this is batch9 changes.

    Story: 2006387
    Task: 39524

    Change-Id: Ia1fe0f2baafb78c974551100f16e6a7d99882f15
    Signed-off-by: Sharath Kumar K <email address hidden>

    De-branding in starlingx/config: CGCS -> StarlingX

    1. Rename CGCS to StarlingX for .spec file
    2. Rename TIS to StarlingX for .service files

    Test:
    After the de-brand change, bootimage.iso has been built in the flock
    Layer and installed on the dev machine to validate the changes.

    Please note, doing de-brand changes in batches, this is batch10 changes.

    Story: 2006387
    Task: 36202

    Change-Id: I404ce0da2621495175ad31489e9ad6f7b0211e26
    Signed-off-by: Sharath Kumar K <email address hidden>

commit d141e954fa6bbf688929ec90d1b6604a97792c43
Author: Teresa Ho <email address hidden>
Date: Tue Mar 31 10:08:57 2020 -0400

    Sysinv extensions for FPGA support

    This update adds cli and restapi to support FPGA device
    programming.

    CLI commands:
    system device-image-apply
    system device-image-create
    system device-image-delete
    system device-image-list
    system device-image-remove
    system device-image-show
    system device-image-state-list
    system device-label-list
    system host-device-image-update
    system host-device-image-update-abort
    system host-device-label-assign
    system host-device-label-list
    system host-device-label-remove

    Story: 2006740
    Task: 39498

    Change-Id: I556c2e7a51b3931b5a66ab27b67f51e3a8aebd9f
    Signed-off-by: Teresa Ho <email address hidden>

commit 491cca42ed854d2cb3ee3646b93c56a4f45f563c
Author: Elena Taivan <email address hidden>
Date: Wed Apr 29 11:25:26 2020 +0000

    Qcow2 conversion to raw can be done using 'image-conversion' filesystem

    1. Conversion filesystem can be added before/after
       stx-openstack is applied
    2. If conversion filesystem is added after stx-openstack
       is applied, changes to stx-openstack will only take effec...

tags: added: in-f-centos8
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.