StarlingX

barbican service not functional until after initial controller unlock

Bug #1834670 reported by Allain Legacy on 2019-06-28

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	Fix Released	Medium	Alexander Kozyrev

Bug Description

Brief Description
-----------------
The barbican service is not functional after bootstrapping and before the initial controller-0 unlock. This prevents configuring a BMC username and password on controller-0 prior to the unlock.

The problem appears to be related to the Barbican service not being functional prior to unlocking controller-0. See sysinv.log below which show failures to store the BMC password prior to the unlock.

Also, prior to the unlock, the "openstack secrets" command which are backed by Barbican do not work (but does work after the unlock):

[sysadmin@controller-0 ~(keystone_admin)]$ openstack secret list
5xx Server error: Service Unavailable
Service Unavailable

...even though barbican shows up in the openstack endpoint list:

Severity
--------
Major, controller-0 cannot be configured for BMC access until after the system has been configured.

Steps to Reproduce
------------------
1. bootstrap the system using Ansible
2. configured controller-0 as per the wiki instructions
3. configure a bmc type, user, password on controller-0
4. unlock controller-0
5. observe that there is an alarm raised against the BMC access on controller-0

Expected Behavior
------------------
The end user should be able to configure a BMC user/password prior to unlocking controller-0.

Actual Behavior
----------------
An alarm is raised and BMC access is not functional.

Reproducibility
---------------
100%

System Configuration
--------------------
Any system with physical BMC devices.

Branch/Pull Time/Commit
-----------------------
20190628T013000Z

Last Pass
---------
Unknown

Timestamp/Logs
--------------
This alarm will be raised if the BMC is configured prior to unlocking controller-0:

controller-0 access to board management module has failed.

This appears to be as a result of failing to access barbican prior to the unlock as is shown by these sysinv.log logs.

2019-06-28 06:44:09.610 104162 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2019-06-28-06-44-09 patch
2019-06-28 06:44:09.610 104162 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['bm_username', 'bm_ip', 'bm_type']
2019-06-28 06:44:09.610 104162 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['bm_username', 'bm_ip', 'bm_type']
2019-06-28 06:44:09.611 104162 INFO sysinv.api.controllers.v1.host [-] bm_ip in delta=set(['bm_username', 'bm_ip', 'bm_type']) obm_ip= nbm_ip=128.224.64.171
2019-06-28 06:44:09.611 104162 INFO sysinv.api.controllers.v1.host [-] Updating bm_type from bmc to bmc
2019-06-28 06:44:10.044 90155 ERROR barbicanclient.client [-] 5xx Server error: Service Unavailable
2019-06-28 06:44:10.045 90155 ERROR sysinv.conductor.openstack [req-3923eb1c-9465-4ee3-b766-e432b6df392c admin admin] Unable to find Barbican secret 9c630c87-f605-4d9a-82c5-ba49744a3806
2019-06-28 06:44:10.055 90155 ERROR barbicanclient.client [-] 5xx Server error: <html>
<head>
  <title>503 Service Unavailable</title>
</head>
<body>
  <h1>503 Service Unavailable</h1>
  The server is currently unavailable. Please try again at a later time.<br /><br />
The Keystone service is temporarily unavailable.

</body>
</html>
2019-06-28 06:44:10.055 90155 ERROR sysinv.conductor.openstack [req-3923eb1c-9465-4ee3-b766-e432b6df392c admin admin] Unable to create Barbican secret 9c630c87-f605-4d9a-82c5-ba49744a3806
2019-06-28 06:44:10.061 104162 INFO sysinv.api.controllers.v1.host [-] controller-0 bm semantic checks for user_agent gophercloud/2.0.0 passed
2019-06-28 06:44:10.061 104162 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=None notify_vim=False notify_mtc=True skip_notify_mtce=False
2019-06-28 06:44:10.062 104162 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'bm_type': 'bmc'}
2019-06-28 06:44:10.078 104162 INFO sysinv.api.controllers.v1.host [-] controller-0 Action none perform notify_mtce
2019-06-28 06:44:10.081 104162 INFO sysinv.api.controllers.v1.mtce_api [-] number of calls to rest_api_request=1 (max_retry=3)
2019-06-28 06:44:10.081 104162 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:2112/v1/hosts/9c630c87-f605-4d9a-82c5-ba49744a3806 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"tboot": "false", "ttys_dcd": null, "subfunctions": "controller", "bm_ip": "128.224.64.171", "install_state": null, "rootfs_device": "/dev/disk/by-path/pci-0000:00:1f.2-ata-1.0", "ihost_action": null, "bm_username": "root", "operation": "modify", "serialid": null, "id": 1, "vim_progress_status": null, "console": "ttyS0,115200n8", "uuid": "9c630c87-f605-4d9a-82c5-ba49744a3806", "mgmt_ip": "192.168.204.3", "software_load": "19.01", "config_status": null, "hostname": "controller-0", "iscsi_initiator_name": null, "capabilities": {"stor_function": "monitor"}, "install_output": "text", "location": {}, "availability": "online", "invprovision": "provisioned", "peer_id": null, "administrative": "locked", "personality": "controller", "recordtype": "standard", "bm_mac": null, "mtce_info": null, "isystem_uuid": "e22eac3b-b6d8-4e93-b68c-29ec4e0ac6c4", "boot_device": "/dev/disk/by-path/pci-0000:00:1f.2-ata-1.0", "install_state_info": null, "mgmt_mac": "00:00:00:00:00:00", "subfunction_oper": "disabled", "task": "", "target_load": "19.01", "vsc_controllers": null, "operational": "disabled", "subfunction_avail": "not-installed", "action": "none", "bm_type": "bmc"}
2019-06-28 06:44:10.083 104162 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'status': u'pass'}

Test Activity
-------------
Sanity testing

See original description

Tags:

Ghada Khalil (gkhalil) on 2019-06-28

description:	updated
Changed in starlingx:
assignee:	nobody → Alex Kozyrev (akozyrev)
tags:	added: stx.config

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2019-06-28:

Currently, Barbican is not started as part of the bootstrap. It only gets started on the unlock by SM. However, it appears that the endpoints are registered. We'll need to look at whether Barbican can get started earlier since sysinv has a dependency on it.

Changed in starlingx:
status:	New → Incomplete
importance:	Undecided → Medium
tags:	added: stx.2.0
Changed in starlingx:
status:	Incomplete → Triaged

Revision history for this message

Alexander Kozyrev (akozyrev) wrote on 2019-07-12:

Keystone token is wrong for Barbican during bootstrap:

2019-07-11 11:07:55.993 1910656 ERROR keystonemiddleware.auth_token [-] Bad response code while validating token: 400 Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-6a9b3282-1ae2-4e4a-a4de-8cb9fe66b61b): BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-6a9b3282-1ae2-4e4a-a4de-8cb9fe66b61b)
2019-07-11 11:07:55.994 1910656 WARNING keystonemiddleware.auth_token [-] Identity response: {"error":{"code":400,"message":"Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.","title":"Bad Request"}}
: BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-6a9b3282-1ae2-4e4a-a4de-8cb9fe66b61b)
2019-07-11 11:07:55.994 1910656 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Failed to fetch token data from identity server: ServiceError: Failed to fetch token data from identity server

The following settings are missing in /etc/barbican/barbican.conf
project_domain_name=Default
user_domain_name=Default

Changed in starlingx:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-15: Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/670846

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-16: Fix merged to config (master)

Reviewed: https://review.opendev.org/670846
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=96ecfd25426d50d8edbae579d156a5eebd40bcd8
Submitter: Zuul
Branch: master

commit 96ecfd25426d50d8edbae579d156a5eebd40bcd8
Author: Alex Kozyrev <email address hidden>
Date: Fri Jul 12 06:12:49 2019 -0400

Fix domain setting for Barbican during bootstrap

    Barbican returns "503 Service Unavailable" during bootstrap
    phase of StarlingX. This happens because Keystone auth token
    lacks domain details for Barbican. Need to explicitly specify
    project_domain_name and user_domain_name in Barbican config.

    Change-Id: I4bf6b275c1eb271b62a2e7a1bc72c049f193afc4
    Closes-bug: 1834670
    Signed-off-by: Alex Kozyrev <email address hidden>

Changed in starlingx:
status:	In Progress → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.