periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master - which includes idempotency check - is timing out during introspection
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Critical
|
Bogdan Dobrelya |
Bug Description
periodic-
fs020 should run with the options as fs001 with the following differences:
- nodeset 1ctrl_1comp
- runs the undercloud idempotent check
- runs full tempest
The current fs020 file has fallen behind in updates and https:/
https:/
fs020 passes introspection in the Rocky promotion pipeline
tags: | added: alert promotion-blocker |
Changed in tripleo: | |
milestone: | none → stein-2 |
importance: | Undecided → Critical |
status: | New → Triaged |
tags: | added: pike-backport-potential queens-backport-potential rocky-backport-potential |
The error reported is:
u'failed_ introspection' : [u'446a0047- 6853-45c4- 920b-92622c20e1 14', u'49affcd3- 4cad-4015- 89f3-325add9760 1c'], u'result': u'Failure caused by error in tasks: send_message\n\n send_message [task_ex_ id=62909211- fd12-45f1- bf3e-269f29211f 6a] -> Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection\n
Looking at the ironic-inspector logs:
https:/ /logs.rdoprojec t.org/openstack -periodic/ git.openstack. org/openstack- infra/tripleo- ci/master/ periodic- tripleo- ci-centos- 7-ovb-1ctlr_ 1comp-featurese t020-master/ d77f650/ logs/undercloud /var/log/ containers/ ironic- inspector/ ironic- inspector. log.txt. gz?level= ERROR
2019-01-05 12:09:02.761 9 ERROR oslo_db. sqlalchemy. engines [-] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql. err.Operational Error) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http:// sqlalche. me/e/e3q8)
2019-01-05 12:09:05.771 9 ERROR ironic_ inspector. conductor. manager [-] Periodic clean up of node cache failed: DBConnectionError: (pymysql. err.Operational Error) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http:// sqlalche. me/e/e3q8)
and
2019-01-05 16:26:11.299 8 ERROR ironic. drivers. modules. ipmitool [req-9699d061- d9e8-4a48- 9dfa-6f7b924067 71 - - - - -] IPMI Error while attempting "ipmitool -I lanplus -H 192.168.100.216 -L ADMINISTRATOR -U admin -R 12 -N 5 -f /tmp/tmpDVbPcz power status" for node 446a0047- 6853-45c4- 920b-92622c20e1 14. Error: Unexpected error while running command.
Command: ipmitool -I lanplus -H 192.168.100.216 -L ADMINISTRATOR -U admin -R 12 -N 5 -f /tmp/tmpDVbPcz power status
in
https:/ /logs.rdoprojec t.org/openstack -periodic/ git.openstack. org/openstack- infra/tripleo- ci/master/ periodic- tripleo- ci-centos- 7-ovb-1ctlr_ 1comp-featurese t020-master/ d77f650/ logs/undercloud /var/log/ containers/ ironic/ ironic- conductor. log.txt. gz?level= ERROR
This seems similar to a lot of what was discussed in https:/ /bugs.launchpad .net/tripleo/ +bug/1784847 but this error is reproducible with:
- master
- 1cont_1comp
- idempotent check