u'failed_introspection': [u'446a0047-6853-45c4-920b-92622c20e114', u'49affcd3-4cad-4015-89f3-325add97601c'], u'result': u'Failure caused by error in tasks: send_message\n\n send_message [task_ex_id=62909211-fd12-45f1-bf3e-269f29211f6a] -> Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection\n
2019-01-05 12:09:02.761 9 ERROR oslo_db.sqlalchemy.engines [-] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-05 12:09:05.771 9 ERROR ironic_inspector.conductor.manager [-] Periodic clean up of node cache failed: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
and
2019-01-05 16:26:11.299 8 ERROR ironic.drivers.modules.ipmitool [req-9699d061-d9e8-4a48-9dfa-6f7b92406771 - - - - -] IPMI Error while attempting "ipmitool -I lanplus -H 192.168.100.216 -L ADMINISTRATOR -U admin -R 12 -N 5 -f /tmp/tmpDVbPcz power status" for node 446a0047-6853-45c4-920b-92622c20e114. Error: Unexpected error while running command.
Command: ipmitool -I lanplus -H 192.168.100.216 -L ADMINISTRATOR -U admin -R 12 -N 5 -f /tmp/tmpDVbPcz power status
The error reported is:
u'failed_ introspection' : [u'446a0047- 6853-45c4- 920b-92622c20e1 14', u'49affcd3- 4cad-4015- 89f3-325add9760 1c'], u'result': u'Failure caused by error in tasks: send_message\n\n send_message [task_ex_ id=62909211- fd12-45f1- bf3e-269f29211f 6a] -> Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection\n
Looking at the ironic-inspector logs:
https:/ /logs.rdoprojec t.org/openstack -periodic/ git.openstack. org/openstack- infra/tripleo- ci/master/ periodic- tripleo- ci-centos- 7-ovb-1ctlr_ 1comp-featurese t020-master/ d77f650/ logs/undercloud /var/log/ containers/ ironic- inspector/ ironic- inspector. log.txt. gz?level= ERROR
2019-01-05 12:09:02.761 9 ERROR oslo_db. sqlalchemy. engines [-] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql. err.Operational Error) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http:// sqlalche. me/e/e3q8)
2019-01-05 12:09:05.771 9 ERROR ironic_ inspector. conductor. manager [-] Periodic clean up of node cache failed: DBConnectionError: (pymysql. err.Operational Error) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http:// sqlalche. me/e/e3q8)
and
2019-01-05 16:26:11.299 8 ERROR ironic. drivers. modules. ipmitool [req-9699d061- d9e8-4a48- 9dfa-6f7b924067 71 - - - - -] IPMI Error while attempting "ipmitool -I lanplus -H 192.168.100.216 -L ADMINISTRATOR -U admin -R 12 -N 5 -f /tmp/tmpDVbPcz power status" for node 446a0047- 6853-45c4- 920b-92622c20e1 14. Error: Unexpected error while running command.
Command: ipmitool -I lanplus -H 192.168.100.216 -L ADMINISTRATOR -U admin -R 12 -N 5 -f /tmp/tmpDVbPcz power status
in
https:/ /logs.rdoprojec t.org/openstack -periodic/ git.openstack. org/openstack- infra/tripleo- ci/master/ periodic- tripleo- ci-centos- 7-ovb-1ctlr_ 1comp-featurese t020-master/ d77f650/ logs/undercloud /var/log/ containers/ ironic/ ironic- conductor. log.txt. gz?level= ERROR
This seems similar to a lot of what was discussed in https:/ /bugs.launchpad .net/tripleo/ +bug/1784847 but this error is reproducible with:
- master
- 1cont_1comp
- idempotent check