The last operation from subnode-2 is the following:
May 12 06:56:56 centos-7-2-node-rax-ord-8800008-577002 os-collect-config: [2017-05-12 06:56:56,622] (os-refresh-config) [INFO] Completed phase post-configure
On the undercloud we see the following:
From heat-api-cfn.log:
2017-05-12 06:56:56.594 4047 INFO eventlet.wsgi.server [req-e2a159fe-3806-45a2-9972-4d510300d84a 47c98f069b0d4633a7a17156c6ba04e1 4af916985a214dd5955f7ed87971413c - 6e59edb5d4cf4033b30791e96cfcb654 6e59edb5d4cf4033b30791e96cfcb654] 192.168.24.3 - - [12/May/2017 06:56:56] "POST /v1/signal/arn%3Aopenstack%3Aheat%3A%3A70e0eeedf2db4f6790f3bbb50c76e223%3Astacks%2Fovercloud-Controller-varysnuo6nyx-0-whpm4maqfizz-Controller-ppc2knqolux2%2F04dc8360-364d-4647-a7c2-90965a94140e%2Fresources%2FInstanceIdDeployment?Timestamp=2017-05-12T06%3A56%3A29Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=a58bdc75277f472aa7690c18184b164f&SignatureVersion=2&Signature=vQ2ZL0avTlmW%2Fzoba4%2BaSQR1i0PofnnpKbCScL6ZBZw%3D HTTP/1.1" 200 115 3.394451
An error I did notice is the following:
heat-engine.log.txt.gz:2017-05-12 06:56:23.397 4058 DEBUG glanceclient.common.http [req-f1f9166e-9152-4039-930a-702f30fa869a da857f112c104b7b82c0e80eb58c648c 70e0eeedf2db4f6790f3bbb50c76e223 - - -] Request returned failure status 404. _handle_response /usr/lib/python2.7/site-packages/glanceclient/common/http.py:106
Potentially we have some glance/swift issues?
api.log.txt.gz:2017-05-12 06:53:16.014 3560 INFO swiftclient [req-881cc954-1238-4240-b363-fb3191f199c9 da857f112c104b7b82c0e80eb58c648c 70e0eeedf2db4f6790f3bbb50c76e223 - default default] RESP STATUS: 404 Not Found
api.log.txt.gz:2017-05-12 06:53:16.015 3560 ERROR swiftclient [req-881cc954-1238-4240-b363-fb3191f199c9 da857f112c104b7b82c0e80eb58c648c 70e0eeedf2db4f6790f3bbb50c76e223 - default default] Container HEAD failed: http://192.168.24.1:8080/v1/AUTH_8737cddce65d4cd18483a8785bf517bd/glance 404 Not Found
api.log.txt.gz:2017-05-12 06:53:16.015 3560 ERROR swiftclient ClientException: Container HEAD failed: http://192.168.24.1:8080/v1/AUTH_8737cddce65d4cd18483a8785bf517bd/glance 404 Not Found
api.log.txt.gz:2017-05-12 06:54:16.195 3561 INFO eventlet.wsgi.server [req-4519b27d-8381-4c8c-af76-7001c5316ca3 da857f112c104b7b82c0e80eb58c648c 70e0eeedf2db4f6790f3bbb50c76e223 - default default] 192.168.24.1 - - [12/May/2017 06:54:16] "HEAD /v1/images/bm-deploy-kernel HTTP/1.1" 404 213 0.641636
api.log.txt.gz:2017-05-12 06:54:16.225 3561 INFO eventlet.wsgi.server [req-cd136535-2a9e-4a5c-80c0-ad35813575fa da857f112c104b7b82c0e80eb58c648c 70e0eeedf2db4f6790f3bbb50c76e223 - default default] 192.168.24.1 - - [12/May/2017 06:54:16] "HEAD /v1/images/bm-deploy-kernel HTTP/1.1" 404 213 0.025524
api.log.txt.gz:2017-05-12 06:54:16.315 3561 INFO eventlet.wsgi.server [req-fa4bb63b-6119-42b5-bcd0-db422ca28d51 da857f112c104b7b82c0e80eb58c648c 70e0eeedf2db4f6790f3bbb50c76e223 - default default] 192.168.24.1 - - [12/May/2017 06:54:16] "HEAD /v1/images/bm-deploy-ramdisk HTTP/1.1" 404 213 0.026485
api.log.txt.gz:2017-05-12 06:54:16.347 3561 INFO eventlet.wsgi.server [req-654f8c67-718f-43fc-8b7f-4dd9ac59885f da857f112c104b7b82c0e80eb58c648c 70e0eeedf2db4f6790f3bbb50c76e223 - default default] 192.168.24.1 - - [12/May/2017 06:54:16] "HEAD /v1/images/bm-deploy-ramdisk HTTP/1.1" 404 213 0.028838
api.log.txt.gz:2017-05-12 06:56:23.395 3560 INFO eventlet.wsgi.server [req-09b61bbf-b265-4b51-8621-a43696ddfbcd da857f112c104b7b82c0e80eb58c648c 70e0eeedf2db4f6790f3bbb50c76e223 - default default] 192.168.24.1 - - [12/May/2017 06:56:23] "HEAD /v1/images/overcloud-full HTTP/1.1" 404 213 0.315360
Looking at: org/85/ 463985/ 2/check/ gate-tripleo- ci-centos- 7-nonha- multinode- oooq/1067ca3/ logs
logs.openstack.
The last operation from subnode-2 is the following: 7-2-node- rax-ord- 8800008- 577002 os-collect-config: [2017-05-12 06:56:56,622] (os-refresh-config) [INFO] Completed phase post-configure
May 12 06:56:56 centos-
May 12 06:56:53 centos- 7-2-node- rax-ord- 8800008- 577002 os-collect-config: ++ curl -s -w '%{http_code}' -X POST -H 'Content-Type: application/json' -o /tmp/tmp.sj2FHanUyh --data-binary '{"deploy_stdout": "os-apply-config deployment 0e65c5de- e457-440a- 9c09-4a51723bf5 32 completed", "deploy_ status_ code": "0"}' 'http:// 192.168. 24.1:8000/ v1/signal/ arn%3Aopenstack %3Aheat% 3A%3A70e0eeedf2 db4f6790f3bbb50 c76e223% 3Astacks% 2Fovercloud- Controller- varysnuo6nyx- 0-whpm4maqfizz- Controller- ppc2knqolux2% 2F04dc8360- 364d-4647- a7c2-90965a9414 0e%2Fresources% 2FInstanceIdDep loyment? Timestamp= 2017-05- 12T06%3A56% 3A29Z&Signature Method= HmacSHA256& AWSAccessKeyId= a58bdc75277f472 aa7690c18184b16 4f&SignatureVer sion=2& Signature= vQ2ZL0avTlmW% 2Fzoba4% 2BaSQR1i0Pofnnp KbCScL6ZBZw% 3D' 7-2-node- rax-ord- 8800008- 577002 os-collect-config: + status=200
May 12 06:56:56 centos-
On the undercloud we see the following: wsgi.server [req-e2a159fe- 3806-45a2- 9972-4d510300d8 4a 47c98f069b0d463 3a7a17156c6ba04 e1 4af916985a214dd 5955f7ed8797141 3c - 6e59edb5d4cf403 3b30791e96cfcb6 54 6e59edb5d4cf403 3b30791e96cfcb6 54] 192.168.24.3 - - [12/May/2017 06:56:56] "POST /v1/signal/ arn%3Aopenstack %3Aheat% 3A%3A70e0eeedf2 db4f6790f3bbb50 c76e223% 3Astacks% 2Fovercloud- Controller- varysnuo6nyx- 0-whpm4maqfizz- Controller- ppc2knqolux2% 2F04dc8360- 364d-4647- a7c2-90965a9414 0e%2Fresources% 2FInstanceIdDep loyment? Timestamp= 2017-05- 12T06%3A56% 3A29Z&Signature Method= HmacSHA256& AWSAccessKeyId= a58bdc75277f472 aa7690c18184b16 4f&SignatureVer sion=2& Signature= vQ2ZL0avTlmW% 2Fzoba4% 2BaSQR1i0Pofnnp KbCScL6ZBZw% 3D HTTP/1.1" 200 115 3.394451
From heat-api-cfn.log:
2017-05-12 06:56:56.594 4047 INFO eventlet.
If I keep tracing that request (req-e2a159fe- 3806-45a2- 9972-4d510300d8 4a) through heat services I see: var/log/ heat/heat- engine. log.txt. gz:2017- 05-12 06:56:55.068 4058 DEBUG heat.engine. clients. keystoneclient [req-e2a159fe- 3806-45a2- 9972-4d510300d8 4a - 70e0eeedf2db4f6 790f3bbb50c76e2 23 - - -] Using stack domain heat_stack __init__ /usr/lib/ python2. 7/site- packages/ heat/engine/ clients/ os/keystone/ heat_keystonecl ient.py: 98 var/log/ heat/heat- engine. log.txt. gz:2017- 05-12 06:56:54.159 4058 WARNING oslo_config.cfg [req-e2a159fe- 3806-45a2- 9972-4d510300d8 4a - 70e0eeedf2db4f6 790f3bbb50c76e2 23 - - -] Option "auth_plugin" from group "trustee" is deprecated. Use option "auth_type" from group "trustee". var/log/ heat/heat- engine. log.txt. gz:2017- 05-12 06:56:56.589 4058 DEBUG oslo_messaging. _drivers. amqpdriver [req-e2a159fe- 3806-45a2- 9972-4d510300d8 4a - 70e0eeedf2db4f6 790f3bbb50c76e2 23 - - -] sending reply msg_id: 96c04daa60ea49f c956ddc2c81c232 5a reply queue: reply_f730340a2 9fd463aadc604f3 bb6d3445 time elapsed: 2.478203106s _send_reply /usr/lib/ python2. 7/site- packages/ oslo_messaging/ _drivers/ amqpdriver. py:73 var/log/ heat/heat- engine. log.txt. gz:2017- 05-12 06:56:56.591 4058 DEBUG heat.engine.service [req-e2a159fe- 3806-45a2- 9972-4d510300d8 4a - 70e0eeedf2db4f6 790f3bbb50c76e2 23 - - -] signaling resource overcloud- Controller- varysnuo6nyx- 0-whpm4maqfizz- Controller- ppc2knqolux2: InstanceIdDeplo yment _resource_signal /usr/lib/ python2. 7/site- packages/ heat/engine/ service. py:1807 var/log/ heat/heat- engine. log.txt. gz:2017- 05-12 06:56:56.642 4058 DEBUG oslo_messaging. _drivers. amqpdriver [req-e2a159fe- 3806-45a2- 9972-4d510300d8 4a - 70e0eeedf2db4f6 790f3bbb50c76e2 23 - - -] CALL msg_id: 86f08b9195a74c1 182567c7789485f 8b exchange 'heat' topic 'engine' _send /usr/lib/ python2. 7/site- packages/ oslo_messaging/ _drivers/ amqpdriver. py:448 var/log/ heat/heat- engine. log.txt. gz:2017- 05-12 06:56:56.925 4057 DEBUG oslo_messaging. _drivers. amqpdriver [req-e2a159fe- 3806-45a2- 9972-4d510300d8 4a - 70e0eeedf2db4f6 790f3bbb50c76e2 23 - - -] sending reply msg_id: 86f08b9195a74c1 182567c7789485f 8b reply queue: reply_dc1ca8984 ed1486194739f1e 78e14688 time elapsed: 0.0253046449998s _send_reply /usr/lib/ python2. 7/site- packages/ oslo_messaging/ _drivers/ amqpdriver. py:73 var/log/ heat/heat- engine. log.txt. gz:2017- 05-12 06:56:56.930 4058 DEBUG oslo_messaging. _drivers. amqpdriver [req-e2a159fe- 3806-45a2- 9972-4d510300d8 4a - 70e0eeedf2db4f6 790f3bbb50c76e2 23 - - -] CALL msg_id: 2e57fed65172405 da6edd1e9b8bfda 7d exchange 'heat' topic 'engine' _send /usr/lib/ python2. 7/site- packages/ oslo_messaging/ _drivers/ amqpdriver. py:448 var/log/ heat/heat- engine. log.txt. gz:2017- 05-12 06:56:56.965 4058 DEBUG oslo_messaging. _drivers. amqpdriver [req-e2a159fe- 3806-45a2- 9972-4d510300d8 4a - 70e0eeedf2db4f6 790f3bbb50c76e2 23 - - -] sending reply msg_id: 2e57fed65172405 da6edd1e9b8bfda 7d reply queue: reply_dc1ca8984 ed1486194739f1e 78e14688 time elapsed: 0.032641324s _send_reply /usr/lib/ python2. 7/site- packages/ oslo_messaging/ _drivers/ amqpdriver. py:73
undercloud/
undercloud/
undercloud/
undercloud/
undercloud/
undercloud/
undercloud/
undercloud/
An error I did notice is the following: log.txt. gz:2017- 05-12 06:56:23.397 4058 DEBUG glanceclient. common. http [req-f1f9166e- 9152-4039- 930a-702f30fa86 9a da857f112c104b7 b82c0e80eb58c64 8c 70e0eeedf2db4f6 790f3bbb50c76e2 23 - - -] Request returned failure status 404. _handle_response /usr/lib/ python2. 7/site- packages/ glanceclient/ common/ http.py: 106
heat-engine.
Potentially we have some glance/swift issues? txt.gz: 2017-05- 12 06:53:16.014 3560 INFO swiftclient [req-881cc954- 1238-4240- b363-fb3191f199 c9 da857f112c104b7 b82c0e80eb58c64 8c 70e0eeedf2db4f6 790f3bbb50c76e2 23 - default default] RESP STATUS: 404 Not Found txt.gz: 2017-05- 12 06:53:16.015 3560 ERROR swiftclient [req-881cc954- 1238-4240- b363-fb3191f199 c9 da857f112c104b7 b82c0e80eb58c64 8c 70e0eeedf2db4f6 790f3bbb50c76e2 23 - default default] Container HEAD failed: http:// 192.168. 24.1:8080/ v1/AUTH_ 8737cddce65d4cd 18483a8785bf517 bd/glance 404 Not Found txt.gz: 2017-05- 12 06:53:16.015 3560 ERROR swiftclient ClientException: Container HEAD failed: http:// 192.168. 24.1:8080/ v1/AUTH_ 8737cddce65d4cd 18483a8785bf517 bd/glance 404 Not Found txt.gz: 2017-05- 12 06:54:16.195 3561 INFO eventlet. wsgi.server [req-4519b27d- 8381-4c8c- af76-7001c5316c a3 da857f112c104b7 b82c0e80eb58c64 8c 70e0eeedf2db4f6 790f3bbb50c76e2 23 - default default] 192.168.24.1 - - [12/May/2017 06:54:16] "HEAD /v1/images/ bm-deploy- kernel HTTP/1.1" 404 213 0.641636 txt.gz: 2017-05- 12 06:54:16.225 3561 INFO eventlet. wsgi.server [req-cd136535- 2a9e-4a5c- 80c0-ad35813575 fa da857f112c104b7 b82c0e80eb58c64 8c 70e0eeedf2db4f6 790f3bbb50c76e2 23 - default default] 192.168.24.1 - - [12/May/2017 06:54:16] "HEAD /v1/images/ bm-deploy- kernel HTTP/1.1" 404 213 0.025524 txt.gz: 2017-05- 12 06:54:16.315 3561 INFO eventlet. wsgi.server [req-fa4bb63b- 6119-42b5- bcd0-db422ca28d 51 da857f112c104b7 b82c0e80eb58c64 8c 70e0eeedf2db4f6 790f3bbb50c76e2 23 - default default] 192.168.24.1 - - [12/May/2017 06:54:16] "HEAD /v1/images/ bm-deploy- ramdisk HTTP/1.1" 404 213 0.026485 txt.gz: 2017-05- 12 06:54:16.347 3561 INFO eventlet. wsgi.server [req-654f8c67- 718f-43fc- 8b7f-4dd9ac5988 5f da857f112c104b7 b82c0e80eb58c64 8c 70e0eeedf2db4f6 790f3bbb50c76e2 23 - default default] 192.168.24.1 - - [12/May/2017 06:54:16] "HEAD /v1/images/ bm-deploy- ramdisk HTTP/1.1" 404 213 0.028838 txt.gz: 2017-05- 12 06:56:23.395 3560 INFO eventlet. wsgi.server [req-09b61bbf- b265-4b51- 8621-a43696ddfb cd da857f112c104b7 b82c0e80eb58c64 8c 70e0eeedf2db4f6 790f3bbb50c76e2 23 - default default] 192.168.24.1 - - [12/May/2017 06:56:23] "HEAD /v1/images/ overcloud- full HTTP/1.1" 404 213 0.315360
api.log.
api.log.
api.log.
api.log.
api.log.
api.log.
api.log.
api.log.