A worker was found in dead state

[Jean-Philippe Evrard]

The translations job seem to regularily fail due to the following reason [1]: "A worker was found in dead state".

This started to appear when we moved to ansible 2.4, IIRC,
and doesn't seem to happen on my machine.

Further investigation is needed to ensure everything is well tested.

[1]: http://logs.openstack.org/periodic/git.openstack.org/openstack/openstack-ansible/master/openstack-ansible-deploy-translations-ubuntu-xenial/7f00866/job-output.txt.gz#_2018-02-06_07_54_53_970546

[Frank Kloeker (f-kloeker)]

It seems an Out of Memory issue on [2]. Last Host Sys Info said 7925 MB used, 57 MB free [3]

[2] http://logs.openstack.org/periodic/git.openstack.org/openstack/openstack-ansible/master/openstack-ansible-deploy-translations-ubuntu-xenial/7f00866/logs/openstack/aio1_octavia_server_container-120aa509/syslog.txt.gz#_Feb__6_07_54_59

[3] http://logs.openstack.org/periodic/git.openstack.org/openstack/openstack-ansible/master/openstack-ansible-deploy-translations-ubuntu-xenial/7f00866/logs/openstack/instance-info/host_system_info_07-54-59.log.txt.gz

[Marcin Dulak (marcin-dulak)]

This is an Ansible error reported in many places by many people, outside of openstack-ansible, and long before Ansible 2.4 https://github.com/ansible/ansible/issues/12239

I've never been able to trigger this error using a simple playbook, but give the openstack-ansible deployment host 512 MB RAM and you will most likely see it.

[Frank Kloeker (f-kloeker)]

Meanwhile I can confirm the error on a Ubuntu 16.04 VM with 8 GB of RAM. Will test with other Ansible versions.

[Frank Kloeker (f-kloeker)]

OSA master is using currenly ansible 2.4.2.0.
I had the same experience with 2.4.3.0
In ansible 2.5.0a1 I've got
ERROR! Unexpected Exception, this is probably a bug: module 'ansible.playbook.play_context' has no attribute 'MAGIC_VARIABLE_MAPPING'
+ ansible -m setup -a gather_subset=network,hardware,virtual hosts
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e @/etc/openstack_deploy/user_variables_translations.yml -e @/etc/openstack_deploy/user_variables.yml "
to see the full traceback, use -vvv
++ exit_fail 347 0
++ set +x
ERROR! Unexpected Exception, this is probably a bug: module 'ansible.playbook.play_context' has no attribute 'MAGIC_VARIABLE_MAPPING'

[Jean-Philippe Evrard (jean-philippe-evrard)]

Yeah we are not ready for 2.5 yet, but I didn't expect a RAM issue for our translations deploy until 2.4 appeared. That's what I meant. Probably worth digging deeper in why the memory is higher up for this deploy, and how can we reduce it.

[Marcin Dulak (marcin-dulak)]

There are some memory fixes promised in 2.4.4.0 https://github.com/ansible/ansible/issues/31673 but one project https://github.com/iiab/iiab/pull/671 reverted to 2.4.2.0, since it turned out 2.4.3.0 was aggravating memory problems

[Dmitriy Rabotyagov (noonedeadpunk)]

Doesn't happen now, so marking as resolved