Unable to use multiprocessing, this is normally caused by lack of access to /dev/shm

[James Bagwell]

Summary:
==================

Deployment of stein overcloud fails with Stderr: u'ERROR! Unable to use multiprocessing, this is normally caused by lack of access to /dev/shm: [Errno 13] Permission denied\n'

Steps to reproduce
==================

Deploy openstack stein containerized deployment

Expected result
===============
Overcloud deployed successfully.

Actual result
=============
Full deployment command: /usr/bin/python2 /usr/bin/openstack cbis overcloud deploy --templates templates/ --no-cleanup

Ansible Command: ansible-playbook-2 -vvvvv /tmp/ansible-mistral-actionoKf_6L/playbook.yaml --become --become-user root --inventory-file /tmp/ansible-mistral-actionoKf_6L/inventory.yaml
fails during the deployment with exit code 1.

================
Quick troubleshooting shows correct permissions of shm.

(undercloud) [stack@undercloud cbis]$ ls -lha /dev/ | grep shm
drwxrwxrwt. 2 root root 60 Dec 14 22:04 shm

(undercloud) [stack@undercloud cbis]$ ls -lha /dev/shm
total 84K
drwxrwxrwt. 2 root root 60 Dec 14 22:04 .
drwxr-xr-x. 19 root root 3.2K Dec 14 22:27 ..
-rw-------. 1 root root 81K Dec 14 20:35 libpod_lock

Environment
===========
 Stein with Centos 7.7
 3 Controllers , 2 ovs computes.

 ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)
 ansible 2.8.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/cbis-admin/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug 7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
 Which networking type did you use?
   Openvswitch

Logs & Configs
==============

Stdout: u"ansible-playbook-2 2.8.0\n config file = /tmp/ansible-mistral-actionoKf_6L/ansible.cfg\n configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules', u'/usr/share/openstack-tripleo-validations/library']\n ansible python module location = /usr/lib/python2.7/site-packages/ansible\n executable location = /bin/ansible-playbook-2\n python version = 2.7.5 (default, Aug 7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\nUsing /tmp/ansible-mistral-actionoKf_6L/ansible.cfg as config file\nsetting up inventory plugins\nhost_list declined parsing /tmp/ansible-mistral-actionoKf_6L/inventory.yaml as it did not pass it's verify_file() method\nscript declined parsing /tmp/ansible-mistral-actionoKf_6L/inventory.yaml as it did not pass it's verify_file() method\nParsed /tmp/ansible-mistral-actionoKf_6L/inventory.yaml inventory source with yaml plugin\n"
Stderr: u'ERROR! Unable to use multiprocessing, this is normally caused by lack of access to /dev/shm: [Errno 13] Permission denied\n'

[Przemyslaw Sasnal (psasnal)]

I'm facing exact same problem, which seems to be intermittent. I have noticed, that /dev/shm directory inside mistral containers is not writable (even by root on the container). I believe that's because it is mounted incorrectly to system partition (and blocked by SELinux).

On undercloud with the bug:
(inside mistral_executor) df -h
...
dev/vda10 219G 42G 177G 20% /dev/shm
...

On undercloud without the bug:
(inside mistral_executor) df -h
...
shm 63M 0 63M 0% /dev/shm
...

[kobig (kobi.ginon)]

thanks @psasnal
i m having the same issue , the randomization of the issue is not explainable.
in Queens version with mistral on containers we did not had this issue at all.
while in stein this happens 2 out of 3 deployments.
can someone take this issue ?

[wes hayutin (weshayutin)]

@Emilien, please have a look or help get it into the right hands.
Thanks