Deployment progress bar doesn't appear, user must press F5

[Dan Trainor]

The progress bar in the UI used to indicate deployment progress occasionally does not update, and the user is forced to refresh the page with F5.

This is likely caused by some transient Zaqar WebSocket issues we've been seeing with some other actions as well, where progress update messages never make it to the client (browser).

[Dan Trainor (dtrainor)]

The Zaqar request originates from the client (browser), which then passes through haproxy (443/tcp for SSL connections), then to Apache's mod_proxy for the appropriate endpoint, in this case, /zaqar. The request is then passed to the Proxy backend from Apache.

The Zaqar WebSocket server is quick to close connections on a completed request. Apache assumes this connection is still open, and on subsequent requests attempts to reuse this connection that's already closed. Doing so results in random Zaqar 502 messages:

192.168.24.1 - - [12/Nov/2018:16:20:56 -0500] "GET /zaqar HTTP/1.1" 502 232 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"

mod_dumpio information about the request:

[...]
[Mon Nov 12 16:20:56.623698 2018] [dumpio:trace7] [pid 523] mod_dumpio.c(103): [remote 192.168.24.3:9000] mod_dumpio: dumpio_out (data-POOL): Upgrade: WebSocket\r\nConnection: Upgrade\r\n\r\n
[Mon Nov 12 16:20:56.623703 2018] [dumpio:trace7] [pid 523] mod_dumpio.c(63): [remote 192.168.24.3:9000] mod_dumpio: dumpio_out (metadata-FLUSH): 0 bytes
[Mon Nov 12 16:20:56.623808 2018] [proxy:error] [pid 523] (32)Broken pipe: [client 192.168.24.1:47986] AH01084: pass request body failed to 192.168.24.3:9000 (192.168.24.3)

Using mod_proxy's 'enablereuse=off' forces Apache to close those connections after data has been transmitted, and reopens a connection with subsequent requests. Testing has shown no performance impact.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/618019

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/618019
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=d4c98bd2c615cc3ac6546cea6814b113c352a8d3
Submitter: Zuul
Branch: master

commit d4c98bd2c615cc3ac6546cea6814b113c352a8d3
Author: Dan Trainor <email address hidden>
Date: Wed Nov 14 12:55:13 2018 -0700

    Enable disablereuse option in Zaqar proxy backend

    Modify the Zaqar mod_proxy backend to use use the parameter
    disablereuse=On, disabling the ability to reuse connections to the
    proxy backend. This is necessary to prevent reuse of connections by
    mod_proxy to the backend Zaqar WebSocket server when the Zaqar
    WebSocket server closes the socket on its own, that mod_proxy
    occasionally expects to be open. This creates a new connection with
    every request.

    Change-Id: Id5f22d8682805566eadd8205a5aa88524ce94b80
    Closes-Bug: #1803422

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/618539

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/618544

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/618732

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/618732
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=34bf798ec7e75b46d6c947523fd110ec490c8341
Submitter: Zuul
Branch: stable/rocky

commit 34bf798ec7e75b46d6c947523fd110ec490c8341
Author: Dan Trainor <email address hidden>
Date: Wed Nov 14 12:55:13 2018 -0700

    Enable disablereuse option in Zaqar proxy backend

    Modify the Zaqar mod_proxy backend to use use the parameter
    disablereuse=On, disabling the ability to reuse connections to the
    proxy backend. This is necessary to prevent reuse of connections by
    mod_proxy to the backend Zaqar WebSocket server when the Zaqar
    WebSocket server closes the socket on its own, that mod_proxy
    occasionally expects to be open. This creates a new connection with
    every request.

    Change-Id: Id5f22d8682805566eadd8205a5aa88524ce94b80
    Closes-Bug: #1803422
    (cherry picked from commit d4c98bd2c615cc3ac6546cea6814b113c352a8d3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/618544
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=6df4b027e88eef2c7bf020a3e9d82ff701652df0
Submitter: Zuul
Branch: stable/pike

commit 6df4b027e88eef2c7bf020a3e9d82ff701652df0
Author: Dan Trainor <email address hidden>
Date: Wed Nov 14 12:55:13 2018 -0700

    Enable disablereuse option in Zaqar proxy backend

    Modify the Zaqar mod_proxy backend to use use the parameter
    disablereuse=On, disabling the ability to reuse connections to the
    proxy backend. This is necessary to prevent reuse of connections by
    mod_proxy to the backend Zaqar WebSocket server when the Zaqar
    WebSocket server closes the socket on its own, that mod_proxy
    occasionally expects to be open. This creates a new connection with
    every request.

    Change-Id: Id5f22d8682805566eadd8205a5aa88524ce94b80
    Closes-Bug: #1803422
    (cherry picked from commit d4c98bd2c615cc3ac6546cea6814b113c352a8d3)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 9.4.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/618539
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=249d5a279155e0b2b32cc299fa022e0649acb499
Submitter: Zuul
Branch: stable/queens

commit 249d5a279155e0b2b32cc299fa022e0649acb499
Author: Dan Trainor <email address hidden>
Date: Wed Nov 14 12:55:13 2018 -0700

    Enable disablereuse option in Zaqar proxy backend

    Modify the Zaqar mod_proxy backend to use use the parameter
    disablereuse=On, disabling the ability to reuse connections to the
    proxy backend. This is necessary to prevent reuse of connections by
    mod_proxy to the backend Zaqar WebSocket server when the Zaqar
    WebSocket server closes the socket on its own, that mod_proxy
    occasionally expects to be open. This creates a new connection with
    every request.

    Change-Id: Id5f22d8682805566eadd8205a5aa88524ce94b80
    Closes-Bug: #1803422
    (cherry picked from commit d4c98bd2c615cc3ac6546cea6814b113c352a8d3)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 7.4.17 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 8.5.0 release.