BMC node in periodic OVB jobs fails to retrieve metadata information
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Critical
|
Gabriele Cerami |
Bug Description
Many periodic jobs are failing during node registration:
e.g.
shows:
FAILED', u'message': [{u'result': u'Node eb42fa7e-
For all the nodes.
Looking at the bottom of BMC console logs at
we can see node fails to retrieve metedata information with these errors:
cloud-init[1335]: 2019-05-30 01:51:09,851 - url_helper.
so either the network or the metadata service is not working properly
Changed in tripleo: | |
milestone: | train-1 → train-2 |
Today no failure but the job times out... I think it might still be introspection related, at least that doesn't look quite right:
http:// logs.rdoproject .org/openstack- periodic- master/ opendev. org/openstack/ tripleo- ci/master/ periodic- tripleo- ci-centos- 7-ovb-3ctlr_ 1comp-featurese t001-master/ d6746ac/ logs/undercloud /home/zuul/ overcloud_ prep_images. log.txt. gz
2019-06-03 02:08:21 | Introspection of node 2c6e169f- 5d90-4d9d- 8b1c-b1ecd7aa71 b2 completed. Status:SUCCESS. Errors:None bb52-4291- b4e3-b5065e1165 07 timed out. f28f-43c4- a02f-55c9b7ba9f fa completed. Status:SUCCESS. Errors:None aa2a-4257- a31b-97aa8c1aea 6d completed. Status:SUCCESS. Errors:None
2019-06-03 02:08:21 | Introspection of node d17a4ffc-
2019-06-03 02:08:21 | Introspection of node 78c030a9-
2019-06-03 02:08:21 | Introspection of node 2233542e-
2019-06-03 02:08:21 | Successfully introspected 4 node(s).
2019-06-03 02:08:21 |
2019-06-03 02:08:21 | Introspection completed.
2019-06-03 02:08:21 | + openstack overcloud node provide --all-manageable
2019-06-03 02:08:24 | Waiting for messages on queue 'tripleo' with no timeout.
one of those nodes times out and then it waits indefinitely.. nothing further in log
from bmc logs i see
http:// logs.rdoproject .org/openstack- periodic- master/ opendev. org/openstack/ tripleo- ci/master/ periodic- tripleo- ci-centos- 7-ovb-3ctlr_ 1comp-featurese t001-master/ d6746ac/ logs/bmc- console. log
[ 93.777302] cloud-init[2063]: parse error: Invalid numeric literal at line 1, column 15
not sure yet if this is same bug or different one