Nova vnc proxy fails on redeploy

[Hugh Saunders]

When nova_console_novnc_install[1] runs, it removes /usr/share/novnc which is the working directory for novncproxy. After removal of this directory the proxy serves up empty responses.

root@infra01-nova-console-container-5d47ccdd:~# curl localhost:6080
curl: (52) Empty reply from server

Strace reveals it failing trying to open its working directory:
[pid 25915] getcwd(0x1ef1550, 1024) = -1 ENOENT (No such file or directory)

Restarting nova-novncproxy resolves the symptom as it will use the current /usr/share/novnc as its working directory.

I think the solution is to trigger a restart handler on the task that removes the directory, ensuring that novncproxy will always be restarted if that directory is removed.

[1]: https://github.com/openstack/openstack-ansible-os_nova/blob/97ecede740e830d5bacb3ad4ae7a3d789a373403/tasks/consoles/nova_console_novnc_install.yml#L16

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/539583

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/539583
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=c41103ba702cd6fd85d8e4d4fe86caef81623f44
Submitter: Zuul
Branch: master

commit c41103ba702cd6fd85d8e4d4fe86caef81623f44
Author: Jesse Pretorius <email address hidden>
Date: Wed Jan 31 15:49:44 2018 +0000

    Improve console install and restart services appropriately

    The current implementation wipes out the novnc/spice directory
    every single time the playbook executes. While it does replace
    it afterwards it does not fire a notifier to make the service
    restart, resulting in the service failing.

    This patch makes the install process idempotent - it will only
    wipe out the directory if it does not look the way it is
    expected to look. It also fires the notifier to restart services
    if the git repo changes.

    This should ensure that novnc/spice restart properly on upgrades
    and that there are no changes/restarts for the service if the git
    SHA or software does not change.

    Closes-Bug: #1746523
    Change-Id: Ic39f884016a6ba0498110267027783001a087a1c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/540328

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/540329

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/540331

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/540331
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=7cc81440be5d4834daef91130918a33fdc6eb3d7
Submitter: Zuul
Branch: stable/newton

commit 7cc81440be5d4834daef91130918a33fdc6eb3d7
Author: Jesse Pretorius <email address hidden>
Date: Wed Jan 31 15:49:44 2018 +0000

    Improve console install and restart services appropriately

    The current implementation wipes out the novnc/spice directory
    every single time the playbook executes. While it does replace
    it afterwards it does not fire a notifier to make the service
    restart, resulting in the service failing.

    This patch makes the install process idempotent - it will only
    wipe out the directory if it does not look the way it is
    expected to look. It also fires the notifier to restart services
    if the git repo changes.

    This should ensure that novnc/spice restart properly on upgrades
    and that there are no changes/restarts for the service if the git
    SHA or software does not change.

    Backport note:
    The 'Manage LB' handler notification is removed as it is not
    present in newton.

    Closes-Bug: #1746523
    Change-Id: Ic39f884016a6ba0498110267027783001a087a1c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/540328
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=802237e4b2af0011011cabf64f96d9ca12bdd17b
Submitter: Zuul
Branch: stable/pike

commit 802237e4b2af0011011cabf64f96d9ca12bdd17b
Author: Jesse Pretorius <email address hidden>
Date: Wed Jan 31 15:49:44 2018 +0000

    Improve console install and restart services appropriately

    The current implementation wipes out the novnc/spice directory
    every single time the playbook executes. While it does replace
    it afterwards it does not fire a notifier to make the service
    restart, resulting in the service failing.

    This patch makes the install process idempotent - it will only
    wipe out the directory if it does not look the way it is
    expected to look. It also fires the notifier to restart services
    if the git repo changes.

    This should ensure that novnc/spice restart properly on upgrades
    and that there are no changes/restarts for the service if the git
    SHA or software does not change.

    Backport note:
    The 'Manage LB' handler notification is removed as it is not
    present in pike.

    Closes-Bug: #1746523
    Change-Id: Ic39f884016a6ba0498110267027783001a087a1c
    (cherry picked from commit c41103ba702cd6fd85d8e4d4fe86caef81623f44)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_nova 17.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/540329
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=742f3875d65a9c19b7586e97af1887f3b533e8d5
Submitter: Zuul
Branch: stable/ocata

commit 742f3875d65a9c19b7586e97af1887f3b533e8d5
Author: Jesse Pretorius <email address hidden>
Date: Wed Jan 31 15:49:44 2018 +0000

    Improve console install and restart services appropriately

    The current implementation wipes out the novnc/spice directory
    every single time the playbook executes. While it does replace
    it afterwards it does not fire a notifier to make the service
    restart, resulting in the service failing.

    This patch makes the install process idempotent - it will only
    wipe out the directory if it does not look the way it is
    expected to look. It also fires the notifier to restart services
    if the git repo changes.

    This should ensure that novnc/spice restart properly on upgrades
    and that there are no changes/restarts for the service if the git
    SHA or software does not change.

    Backport note:
    The 'Manage LB' handler notification is removed as it is not
    present in ocata.

    Depends-On: https://review.openstack.org/540437
    Closes-Bug: #1746523
    Change-Id: Ic39f884016a6ba0498110267027783001a087a1c
    (cherry picked from commit c41103ba702cd6fd85d8e4d4fe86caef81623f44)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_nova 14.2.15 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_nova 16.0.8 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_nova 15.1.17 release.