Liberty: heat engine does not kill all workers when stopped

[Tom Verdaat]

In the Liberty release, the heat engine does not seem to pick up the settings of the oslo_messaging_* section. In stead the log is flooded with:

ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.

This despite heat.conf having entirely different settings in the oslo_messaging_rabbit section, which functioned properly in Kilo. Tested a workaround by copying the rabbitmq parameters to the DEFAULT section but this didn't make a difference.

Experienced on Ubuntu 14.04 using the ubuntu-cloud-archive packages.

[Thomas Herve (therve)]

Can you link your config file?

[Tom Verdaat (tom-verdaat)]

Of course

[Tom Verdaat (tom-verdaat)]

Done some more tests and there might be more to this problem. Provisioned heat with puppet. Restarting heat-engine after changing the configuration file results in the report I submitted, but it looks like the restart doesn't kill all the workers, which means the configuration file isn't actually reloaded. Running this does the trick:

service heat-engine stop; killall -9 heat-engine; service heat-engine start

Not how it's supposed to work but its a start. Wondering if anybody can confirm this.

[Neetu Jain (nutshi)]

I got a couple of errors when i upgraded to liberty

1) ERROR: Remote error: UnsupportedVersion Endpoint does not support RPC version 1.8. Attempted method: list_stacks
2) ERROR: Remote error: OperationalError (OperationalError) (1054, "Unknown column 'raw_template.predecessor' in 'field list'") 'SELECT raw_template.created_at AS raw_template_created_at, raw_template.updated_at AS raw_template_updated_at, raw_template.id AS raw_template_id, raw_template.template AS raw_template_template, raw_template.files AS raw_template_files, raw_template.environment AS raw_template_environment, raw_template.predecessor AS raw_template_predecessor \nFROM raw_template \nWHERE raw_template.id = %s' (2L,)
[u'
3) {"changed": false, "cmd": "su -s /bin/sh -c \"/opt/heat/bin/heat-manage db_sync\" heat", "delta": "0:00:00.558782", "end": "2015-12-16 22:35:40.538291", "rc": 1, "start": "2015-12-16 22:35:39.979509", "warnings": []} AttributeError: 'TranslatorFactory' object has no attribute 'contextual_form'

but all of them got fixed when i restart heat-engine , heat-api, heat-api-cfn

[Tom Verdaat (tom-verdaat)]

@nutshi I think these are separate bugs. Don't think these are directly related to the AMQP settings problem which I filed originally.

[Željko Jagušt (zeljko-jagust)]

@tom-verdaat Your solution works for me, so I confirm, after stopping heat engine, leftover workers have to be killed manually.