Heat engine hangs during big heat stack creation

[Sergey Galkin]

Steps to reproduce
1. Deploy ccp with configs from https://review.openstack.org/#/c/451419/ on the 152 nodes
2. Try to run shaker (http://pyshaker.readthedocs.io/en/latest/) scenario openstack/full_l2

From time to time heat hangs without any error message.
Last messages in
kubectl -n ccp logs --tail 10 heat-engine-2936414888-8zjwc
is
2017-04-07 16:08:34.154 58 WARNING heat.engine.attributes [req-bd72fe71-dd07-41fb-bdb5-0276962eeaca 4ac37f41f1114d3c8e7c3ff789f0ed12 612fb3db46174cc8b3c38c800db78959 - - -] Attribute networks is not of type Map

screenshots attached

[Sergey Galkin (sgalkin)]

Heat logs

[Sergey Galkin (sgalkin)]

Full heat logs
Messages like '2017-04-07 16:01:22.691 58 ERROR heat.engine.resource ConnectFailure: Unable to establish connection to http://keystone.ccp.svc.cluster.local:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",))'
is described in
https://bugs.launchpad.net/fuel-ccp/+bug/1680430

[Sergey Galkin (sgalkin)]

And I can start delete this stack without problem
2017-04-07 16:08:29.936 28 WARNING heat.engine.attributes [req-7289fd58-9b28-4f82-92fe-dcb77b5807f5 4ac37f41f1114d3c8e7c3ff789f0ed12 612fb3db46174cc8b3c38c800db78959 - - -] Attribute networks is not of type Map
2017-04-07 16:08:32.002 54 WARNING heat.engine.attributes [req-96c48b60-63d7-499e-bb26-9fc919220739 4ac37f41f1114d3c8e7c3ff789f0ed12 612fb3db46174cc8b3c38c800db78959 - - -] Attribute networks is not of type Map
2017-04-07 16:08:34.154 58 WARNING heat.engine.attributes [req-bd72fe71-dd07-41fb-bdb5-0276962eeaca 4ac37f41f1114d3c8e7c3ff789f0ed12 612fb3db46174cc8b3c38c800db78959 - - -] Attribute networks is not of type Map
2017-04-11 04:53:30.458 31 INFO heat.engine.service [req-def37b0d-b9e9-4ff1-b62b-c3ae21255720 4ac37f41f1114d3c8e7c3ff789f0ed12 612fb3db46174cc8b3c38c800db78959 - - -] Deleting stack shaker_sukerf
2017-04-11 04:53:30.956 31 INFO heat.engine.stack [req-def37b0d-b9e9-4ff1-b62b-c3ae21255720 4ac37f41f1114d3c8e7c3ff789f0ed12 612fb3db46174cc8b3c38c800db78959 - - -] Stack CREATE FAILED (shaker_sukerf): Stack CREATE cancelled
2017-04-11 04:53:31.001 31 INFO heat.engine.worker [req-def37b0d-b9e9-4ff1-b62b-c3ae21255720 4ac37f41f1114d3c8e7c3ff789f0ed12 612fb3db46174cc8b3c38c800db78959 - - -] [shaker_sukerf(067633bc-7612-428f-aec4-950b51a7ff1f)] Stopped all active workers for stack CREATE
2017-04-11 04:53:31.077 31 INFO heat.engine.stack [req-def37b0d-b9e9-4ff1-b62b-c3ae21255720 4ac37f41f1114d3c8e7c3ff789f0ed12 612fb3db46174cc8b3c38c800db78959 - - -] Stack DELETE IN_PROGRESS (shaker_sukerf): Stack DELETE started