Train node import failing with websocket._exceptions.WebSocketBadStatusException: Handshake status 503 Service Unavailable

[Sandeep Yadav]

Description:

Train node import failing with websocket._exceptions.WebSocketBadStatusException: Handshake status 503 Service Unavailable

log snippet:

https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/dd652e3/logs/undercloud/home/zuul/overcloud_import_nodes.log.txt.gz
~~~
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_node.py", line 424, in take_action
    instance_boot_option=parsed_args.instance_boot_option
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/baremetal.py", line 62, in register_or_update
    with tripleoclients.messaging_websocket() as ws:
  File "/usr/lib/python3.6/site-packages/tripleoclient/plugin.py", line 223, in messaging_websocket
    cacert=self._instance.cacert)
  File "/usr/lib/python3.6/site-packages/tripleoclient/plugin.py", line 89, in __init__
    sslopt=OS_CACERT)
  File "/usr/lib/python3.6/site-packages/websocket/_core.py", line 514, in create_connection
    websock.connect(url, **options)
  File "/usr/lib/python3.6/site-packages/websocket/_core.py", line 226, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "/usr/lib/python3.6/site-packages/websocket/_handshake.py", line 79, in handshake
    status, resp = _get_resp_headers(sock)
  File "/usr/lib/python3.6/site-packages/websocket/_handshake.py", line 160, in _get_resp_headers
    raise WebSocketBadStatusException("Handshake status %d %s", status, status_message, resp_headers)
websocket._exceptions.WebSocketBadStatusException: Handshake status 503 Service Unavailable
Handshake status 503 Service Unavailable
~~~

Another failure:

https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/5406364/logs/undercloud/home/zuul/overcloud_import_nodes.log.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/be0cba5/logs/undercloud/home/zuul/overcloud_import_nodes.log.txt.gz

[Harald Jensås (harald-jensas)]

AFICT the isse here is that Zaqar is not operational?

In logfile[1] many messages like this:
2022-05-22 20:06:12.204 7 INFO autobahn.asyncio.websocket.WebSocketServerProtocol [-] failing WebSocket opening handshake ('HTTP method 'OPTIONS' not allowed')
2022-05-22 20:06:12.205 7 WARNING autobahn.asyncio.websocket.WebSocketServerProtocol [-] dropping connection to peer tcp4:192.168.24.1:52714 with abort=False: HTTP method 'OPTIONS' not allowed

SSL issues[2] ?

[1] https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/dd652e3/logs/undercloud/var/log/containers/zaqar/zaqar-server.log.txt.gz
[2] https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/dd652e3/logs/undercloud/var/log/containers/httpd/zaqar/zaqar_wsgi_error.log.txt.gz

[Rabi Mishra (rabi)]

Seems like https://github.com/openstack/puppet-tripleo/commit/6af63e1bb13e79d8e7a0f9c12986b24b3e64ff68 has broken zaqar.

[Takashi Kajinami (kajinamit)]

I guess we need to remove options added by https://review.opendev.org/c/openstack/puppet-tripleo/+/828447/6/manifests/haproxy.pp#1739 , especially httpchk does not work with zaqar ws.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/842989

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/842989
Committed: https://opendev.org/openstack/puppet-tripleo/commit/26535b4912aeeb4c1301f86fdb74e689803a5fc6
Submitter: "Zuul (22348)"
Branch: stable/train

commit 26535b4912aeeb4c1301f86fdb74e689803a5fc6
Author: Takashi Kajinami <email address hidden>
Date: Tue May 24 00:28:33 2022 +0900

    Trainl-only: Fix broken healthcheck of zaqar websocket endpoint

    This partially reverts 6af63e1bb13e79d8e7a0f9c12986b24b3e64ff68 and
    removes the healthcheck options added to the zaqar websocket endpoint.
    The endpoint does not support the OPTIONS verb used by the httpchk.

    Because websocket service behaves differently from usual REST servers,
    this also removes httplog.

    This fix is directly submitted to stable/train because the options
    parameter were updated only in stable/train. (Because Zaqar was removed
    in stable/wallaby and we "missed" updating the endpoint in
    stale/victoria fortunately)

    Closes-Bug: #1975462
    Change-Id: I48330f0a6b79f25dcc25c59d33c90fa62407c635

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo train-eol release.