realpath() of /etc/uwsgi/nova-placement-uwsgi.ini failed

[Danilo Chilene]

Hello,

Trying to deploy OSA 15.1.6, below what I'm using and the errors:

OS: Ubuntu 16.04
user_variables.yml: https://gist.github.com/bicofino/6b4ca30658c05d2e07290336fb377db1
openstack_user_config.yml: https://gist.github.com/bicofino/8f738f83e0b5a6f973903a5343566a09

The result of openstack-ansible setup-openstack.yml or openstack-ansible os-nova-install.yml:

failed: [infra4] (item={'key': u'nova-compute', 'value': {u'init_config_overrides': {}, u'service_name': u'nova-compute', u'group': u'nova_compute'}}) => {"failed": true, "item": {"key": "nova-compute", "value": {"group": "nova_compute", "init_config_overrides": {}, "service_name": "nova-compute"}}, "msg": "Could not find the requested service nova-compute: cannot enable"}

failed: [infra2_nova_api_placement_container-83b3fc24] (item={'key': u'nova-placement-api', 'value': {u'program_override': u'/openstack/venvs/nova-15.1.6/bin/uwsgi --ini /etc/uwsgi/nova-placement-uwsgi.ini', u'group': u'nova_api_placement', u'service_name': u'nova-placement-api', u'log_string': u'--logto ', u'init_config_overrides': {}, u'condition': True}}) => {"failed": true, "item": {"key": "nova-placement-api", "value": {"condition": true, "group": "nova_api_placement", "init_config_overrides": {}, "log_string": "--logto ", "program_override": "/openstack/venvs/nova-15.1.6/bin/uwsgi --ini /etc/uwsgi/nova-placement-uwsgi.ini", "service_name": "nova-placement-api"}}, "msg": "Unable to start service nova-placement-api: Job for nova-placement-api.service failed because the control process exited with error code. See \"systemctl status nova-placement-api.service\" and \"journalctl -xe\" for details.\n"}

PLAY RECAP *********************************************************************
infra1 : ok=60 changed=0 unreachable=0 failed=0
infra1_nova_api_metadata_container-a50c905c : ok=62 changed=1 unreachable=0 failed=0
infra1_nova_api_os_compute_container-9cd98e7e : ok=71 changed=1 unreachable=0 failed=0
infra1_nova_api_placement_container-dcc8761c : ok=63 changed=2 unreachable=0 failed=0
infra1_nova_conductor_container-3cfc2e91 : ok=62 changed=1 unreachable=0 failed=0
infra1_nova_console_container-04fab19c : ok=81 changed=5 unreachable=0 failed=0
infra1_nova_scheduler_container-fa3f2ce1 : ok=62 changed=1 unreachable=0 failed=0
infra2 : ok=60 changed=0 unreachable=0 failed=0
infra2_nova_api_metadata_container-9a678a5b : ok=62 changed=1 unreachable=0 failed=0
infra2_nova_api_os_compute_container-ca25fbcf : ok=62 changed=1 unreachable=0 failed=0
infra2_nova_api_placement_container-83b3fc24 : ok=61 changed=1 unreachable=0 failed=1
infra2_nova_conductor_container-34c321fe : ok=62 changed=1 unreachable=0 failed=0
infra2_nova_console_container-9b76699d : ok=66 changed=3 unreachable=0 failed=0
infra2_nova_scheduler_container-abff7a55 : ok=62 changed=1 unreachable=0 failed=0
infra3 : ok=60 changed=0 unreachable=0 failed=0
infra3_nova_api_metadata_container-38399046 : ok=62 changed=1 unreachable=0 failed=0
infra3_nova_api_os_compute_container-ecef6493 : ok=62 changed=1 unreachable=0 failed=0
infra3_nova_api_placement_container-cd3a34fa : ok=63 changed=2 unreachable=0 failed=0
infra3_nova_conductor_container-d62f3561 : ok=62 changed=1 unreachable=0 failed=0
infra3_nova_console_container-83802bc8 : ok=66 changed=3 unreachable=0 failed=0
infra3_nova_scheduler_container-6daee036 : ok=62 changed=1 unreachable=0 failed=0
infra4 : ok=59 changed=0 unreachable=0 failed=1

The logs from the nova_api_placement container:

root@infra1-nova-api-placement-container-dcc8761c:/etc# tail -f /var/log/syslog
Jul 20 03:20:11 infra1-nova-api-placement-container-dcc8761c systemd[1]: Started nova openstack service.
Jul 20 03:20:11 infra1-nova-api-placement-container-dcc8761c uwsgi[6869]: realpath() of /etc/uwsgi/nova-placement-uwsgi.ini failed: No such file or directory [core/utils.c line 3618]
Jul 20 03:20:11 infra1-nova-api-placement-container-dcc8761c systemd[1]: nova-placement-api.service: Main process exited, code=exited, status=1/FAILURE
Jul 20 03:20:11 infra1-nova-api-placement-container-dcc8761c systemd[1]: nova-placement-api.service: Unit entered failed state.
Jul 20 03:20:11 infra1-nova-api-placement-container-dcc8761c systemd[1]: nova-placement-api.service: Failed with result 'exit-code'.
Jul 20 03:20:13 infra1-nova-api-placement-container-dcc8761c systemd[1]: nova-placement-api.service: Service hold-off time over, scheduling restart.
Jul 20 03:20:13 infra1-nova-api-placement-container-dcc8761c systemd[1]: Stopped nova openstack service.
Jul 20 03:20:13 infra1-nova-api-placement-container-dcc8761c systemd[1]: nova-placement-api.service: Start request repeated too quickly.
Jul 20 03:20:13 infra1-nova-api-placement-container-dcc8761c systemd[1]: Failed to start nova openstack service.
Jul 20 03:20:18 infra1-nova-api-placement-container-dcc8761c ansible-user: Invoked with comment=None ssh_key_bits=0 update_password=NOT_LOGGING_PASSWORD non_unique=False force=False ssh_key_type=rsa ssh_key_passphrase=NOT_LOGGING_PARAMETER createhome=True uid=None home=None append=True skeleton=None ssh_key_comment=ansible-generated on infra1-nova-api-placement-container-dcc8761c group=None system=False state=present shell=None expires=None ssh_key_file=None groups=nova move_home=False password=NOT_LOGGING_PARAMETER name=www-data seuser=None remove=False login_class=None generate_ssh_key=None
^C
root@infra1-nova-api-placement-container-dcc8761c:/etc# ls /uwsgi/
ls: cannot access '/uwsgi/': No such file or directory

All other steps run fine(setup-hosts, setup-infra).

Any idea how to handle this? I didn't get this on 1.15.5.

Best Regards,

Danilo

[Danilo Chilene (bicofino)]

Well,

I tried to reproduce the error again today, no error!!!

PLAY RECAP *********************************************************************
infra1 : ok=127 changed=4 unreachable=0 failed=0
infra1_nova_api_metadata_container-a50c905c : ok=81 changed=10 unreachable=0 failed=0
infra1_nova_api_os_compute_container-9cd98e7e : ok=92 changed=10 unreachable=0 failed=0
infra1_nova_api_placement_container-dcc8761c : ok=102 changed=25 unreachable=0 failed=0
infra1_nova_conductor_container-3cfc2e91 : ok=81 changed=10 unreachable=0 failed=0
infra1_nova_console_container-04fab19c : ok=100 changed=14 unreachable=0 failed=0
infra1_nova_scheduler_container-fa3f2ce1 : ok=81 changed=10 unreachable=0 failed=0
infra2 : ok=126 changed=4 unreachable=0 failed=0
infra2_nova_api_metadata_container-9a678a5b : ok=81 changed=10 unreachable=0 failed=0
infra2_nova_api_os_compute_container-ca25fbcf : ok=81 changed=10 unreachable=0 failed=0
infra2_nova_api_placement_container-83b3fc24 : ok=98 changed=21 unreachable=0 failed=0
infra2_nova_conductor_container-34c321fe : ok=81 changed=10 unreachable=0 failed=0
infra2_nova_console_container-9b76699d : ok=85 changed=12 unreachable=0 failed=0
infra2_nova_scheduler_container-abff7a55 : ok=81 changed=10 unreachable=0 failed=0
infra3 : ok=126 changed=4 unreachable=0 failed=0
infra3_nova_api_metadata_container-38399046 : ok=81 changed=10 unreachable=0 failed=0
infra3_nova_api_os_compute_container-ecef6493 : ok=81 changed=10 unreachable=0 failed=0
infra3_nova_api_placement_container-cd3a34fa : ok=98 changed=20 unreachable=0 failed=0
infra3_nova_conductor_container-d62f3561 : ok=81 changed=10 unreachable=0 failed=0
infra3_nova_console_container-83802bc8 : ok=85 changed=12 unreachable=0 failed=0
infra3_nova_scheduler_container-6daee036 : ok=81 changed=10 unreachable=0 failed=0
infra4 : ok=126 changed=5 unreachable=0 failed=0

Thursday 20 July 2017 13:58:57 +0000 (0:00:01.192) 0:09:15.484 *********
===============================================================================
os_nova : Install required pip packages -------------------------------- 22.22s
ceph_client : Install ceph packages ------------------------------------ 19.18s
ceph_client : Restart os services -------------------------------------- 16.50s
pip_install : Install UCA repository key ------------------------------- 16.28s
os_nova : Generate nova config ----------------------------------------- 16.24s
os_nova : Create nova dir ---------------------------------------------- 12.23s
pip_install : Install PIP ---------------------------------------------- 10.25s
ceph_client : Verify Ceph monitors are up ------------------------------- 9.81s
os_nova : Install nova role packages (apt) ------------------------------ 8.17s
os_nova : Copy nova rootwrap filter config ------------------------------ 7.88s
ceph_client : Set ceph_mon_host to an online monitor host --------------- 7.46s
ceph_client : Retrieve keyrings for openstack clients from c...

[Danilo Chilene (bicofino)]

I fixed infra4 restarting systemctl, systemctl daemon-reload

[Jean-Philippe Evrard (jean-philippe-evrard)]

Could you consider it as a temporarily host failure, and close this one up?
Or did you manage to re-create the failure?

[Danilo Chilene (bicofino)]

Hi,

I wasn't able to reproduce the error again but had to use the systemctl daemon-reload, looks like a race condition.

[Andy McCrae (andrew-mccrae)]

I think we can move the ordering of the wsgi service bits.
We'll need the changes to merge in master (https://review.openstack.org/#/c/451425/) this is related to fixing up the uwsgi bits in master.

Once that happens I'll make a manual backport of the reordering - that should resolve this I think.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/493942

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/493942
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=4db7eb15bd559a2d56919350a618658265835721
Submitter: Jenkins
Branch: stable/ocata

commit 4db7eb15bd559a2d56919350a618658265835721
Author: Andy McCrae <email address hidden>
Date: Tue Aug 15 17:18:28 2017 +0100

    Reorder the placement service setup

    We should ensure the placement service is all in place before we go
    about restarting it in the systemd tasks.

    This PR re-orders the nova_placement setup. In master we have moved to
    setup the services at the same time, so this is a manual backport of the
    reordering done in If6c30e00c1c753692c970457b75e3ae7f5cc066c.

    Change-Id: Ieec42b1521e80cc940c6ee48e4d8f198e5712839
    Closes-Bug: 1705374

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_nova 15.1.9 release.