nova-conductor may crash during deploy due to haproxy-keystone 504

[Radosław Piliszek]

keystone was busy (behind haproxy)

nova-conductor:
2019-10-04 15:39:17.103 6 CRITICAL nova [-] Unhandled error: GatewayTimeout: Gateway Timeout (HTTP 504)
2019-10-04 15:39:17.103 6 ERROR nova Traceback (most recent call last):
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/bin/nova-conductor", line 10, in <module>
2019-10-04 15:39:17.103 6 ERROR nova sys.exit(main())
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/cmd/conductor.py", line 44, in main
2019-10-04 15:39:17.103 6 ERROR nova topic=rpcapi.RPC_TOPIC)
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/service.py", line 257, in create
2019-10-04 15:39:17.103 6 ERROR nova periodic_interval_max=periodic_interval_max)
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/service.py", line 129, in __init__
2019-10-04 15:39:17.103 6 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/conductor/manager.py", line 117, in __init__
2019-10-04 15:39:17.103 6 ERROR nova self.compute_task_mgr = ComputeTaskManager()
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/conductor/manager.py", line 243, in __init__
2019-10-04 15:39:17.103 6 ERROR nova self.report_client = report.SchedulerReportClient()
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 186, in __init__
2019-10-04 15:39:17.103 6 ERROR nova self._client = self._create_client()
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 229, in _create_client
2019-10-04 15:39:17.103 6 ERROR nova client = self._adapter or utils.get_sdk_adapter('placement')
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/utils.py", line 1039, in get_sdk_adapter
2019-10-04 15:39:17.103 6 ERROR nova return getattr(conn, service_type)
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack/service_description.py", line 92, in __get__
2019-10-04 15:39:17.103 6 ERROR nova endpoint = proxy_mod.Proxy.get_endpoint(proxy)
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 282, in get_endpoint
2019-10-04 15:39:17.103 6 ERROR nova return self.session.get_endpoint(auth or self.auth, **kwargs)
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 1200, in get_endpoint
2019-10-04 15:39:17.103 6 ERROR nova return auth.get_endpoint(self, **kwargs)
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2019-10-04 15:39:17.103 6 ERROR nova allow_version_hack=allow_version_hack, **kwargs)
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2019-10-04 15:39:17.103 6 ERROR nova service_catalog = self.get_access(session).service_catalog
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2019-10-04 15:39:17.103 6 ERROR nova self.auth_ref = self.get_auth_ref(session)
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2019-10-04 15:39:17.103 6 ERROR nova return self._plugin.get_auth_ref(session, **kwargs)
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 184, in get_auth_ref
2019-10-04 15:39:17.103 6 ERROR nova authenticated=False, log=False, **rkwargs)
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 1106, in post
2019-10-04 15:39:17.103 6 ERROR nova return self.request(url, 'POST', **kwargs)
2019-10-04 15:39:17.103 6 ERROR nova File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 943, in request
2019-10-04 15:39:17.103 6 ERROR nova raise exceptions.from_response(resp, method, url)
2019-10-04 15:39:17.103 6 ERROR nova GatewayTimeout: Gateway Timeout (HTTP 504)
2019-10-04 15:39:17.103 6 ERROR nova

[Matt Riedemann (mriedem)]

It's blowing up trying to get the service catalog from keystone to get the endpoint URL for the placement service. We could do something like in bug 1807219 and offload that placement SchedulerReportClient instantiation to a singleton lazy-load pattern so it's not hit when conductor starts up but if keystone is having problems we could just fail later when trying to lazy-load the client, so I'm not sure if that's helpful. I'd consider this very low priority since the root problem appears to be in keystone.

[Mark Goddard (mgoddard)]

Are there any retries for this request?

[Mark Goddard (mgoddard)]

Radoslaw suggested looking at timeouts.

[Radosław Piliszek (yoctozepto)]

yup, related: https://bugs.launchpad.net/kolla-ansible/+bug/1847727

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/852900

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/852901

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/852900
Committed: https://opendev.org/openstack/nova/commit/c178d9360665c219cbcc71c9f37b9e6e3055a5e5
Submitter: "Zuul (22348)"
Branch: master

commit c178d9360665c219cbcc71c9f37b9e6e3055a5e5
Author: Dan Smith <email address hidden>
Date: Thu Aug 11 09:50:30 2022 -0700

    Unify placement client singleton implementations

    We have many places where we implement singleton behavior for the
    placement client. This unifies them into a single place and
    implementation. Not only does this DRY things up, but may cause us
    to initialize it fewer times and also allows for emitting a common
    set of error messages about expected failures for better
    troubleshooting.

    Change-Id: Iab8a791f64323f996e1d6e6d5a7e7a7c34eb4fb3
    Related-Bug: #1846820

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/852901
Committed: https://opendev.org/openstack/nova/commit/232684b44022f1bc4d72b07045900780de456e63
Submitter: "Zuul (22348)"
Branch: master

commit 232684b44022f1bc4d72b07045900780de456e63
Author: Dan Smith <email address hidden>
Date: Thu Aug 11 10:18:25 2022 -0700

    Avoid n-cond startup abort for keystone failures

    Conductor creates a placement client for the potential case where
    it needs to make a call for certain operations. A transient network
    or keystone failure will currently cause it to abort startup, which
    means it is not available for other unrelated activities, such as
    DB proxying for compute.

    This makes conductor test the placement client on startup, but only
    abort startup on errors that are highly likely to be permanent
    configuration errors, and only warn about things like being unable
    to contact keystone/placement during initialization. If a non-fatal
    error is encountered at startup, later operations needing the
    placement client will retry initialization.

    Closes-Bug: #1846820
    Change-Id: Idb7fcbce0c9562e7b9bd3e80f2a6d4b9bc286830

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 26.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/858997

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/858998

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/858999

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/859000

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/859001

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/859002

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/858997
Committed: https://opendev.org/openstack/nova/commit/77273f067d96a4ec401c3b36f2922d63c4ad7103
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 77273f067d96a4ec401c3b36f2922d63c4ad7103
Author: Dan Smith <email address hidden>
Date: Thu Aug 11 09:50:30 2022 -0700

    Unify placement client singleton implementations

    We have many places where we implement singleton behavior for the
    placement client. This unifies them into a single place and
    implementation. Not only does this DRY things up, but may cause us
    to initialize it fewer times and also allows for emitting a common
    set of error messages about expected failures for better
    troubleshooting.

    Change-Id: Iab8a791f64323f996e1d6e6d5a7e7a7c34eb4fb3
    Related-Bug: #1846820
    (cherry picked from commit c178d9360665c219cbcc71c9f37b9e6e3055a5e5)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/858998
Committed: https://opendev.org/openstack/nova/commit/19346082058d51c78bb157ca5e1304d15691dd9a
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 19346082058d51c78bb157ca5e1304d15691dd9a
Author: Dan Smith <email address hidden>
Date: Thu Aug 11 10:18:25 2022 -0700

    Avoid n-cond startup abort for keystone failures

    Conductor creates a placement client for the potential case where
    it needs to make a call for certain operations. A transient network
    or keystone failure will currently cause it to abort startup, which
    means it is not available for other unrelated activities, such as
    DB proxying for compute.

    This makes conductor test the placement client on startup, but only
    abort startup on errors that are highly likely to be permanent
    configuration errors, and only warn about things like being unable
    to contact keystone/placement during initialization. If a non-fatal
    error is encountered at startup, later operations needing the
    placement client will retry initialization.

    Conflicts:
        nova/tests/unit/conductor/test_conductor.py

    NOTE(melwitt): The conflict is because change
    Id5b04cf2f6ca24af8e366d23f15cf0e5cac8e1cc
    (Use unittest.mock instead of third party mock) is not in Yoga.

    Closes-Bug: #1846820
    Change-Id: Idb7fcbce0c9562e7b9bd3e80f2a6d4b9bc286830
    (cherry picked from commit 232684b44022f1bc4d72b07045900780de456e63)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 25.1.1 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/858999
Committed: https://opendev.org/openstack/nova/commit/87f67df721a4a46bf145f217525cc001582764ca
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 87f67df721a4a46bf145f217525cc001582764ca
Author: Dan Smith <email address hidden>
Date: Thu Aug 11 09:50:30 2022 -0700

    Unify placement client singleton implementations

    We have many places where we implement singleton behavior for the
    placement client. This unifies them into a single place and
    implementation. Not only does this DRY things up, but may cause us
    to initialize it fewer times and also allows for emitting a common
    set of error messages about expected failures for better
    troubleshooting.

    Conflicts:
        nova/limit/placement.py

    NOTE(melwitt): The conflict is because change
    I272b59b7bc8975bfd602640789f80d2d5f7ee698
    (Tell oslo.limit how to count nova resources) is not in Xena.

    Change-Id: Iab8a791f64323f996e1d6e6d5a7e7a7c34eb4fb3
    Related-Bug: #1846820
    (cherry picked from commit c178d9360665c219cbcc71c9f37b9e6e3055a5e5)
    (cherry picked from commit 77273f067d96a4ec401c3b36f2922d63c4ad7103)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/859000
Committed: https://opendev.org/openstack/nova/commit/6027a097caf51428425dc4616898302460f833a1
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 6027a097caf51428425dc4616898302460f833a1
Author: Dan Smith <email address hidden>
Date: Thu Aug 11 10:18:25 2022 -0700

    Avoid n-cond startup abort for keystone failures

    Conductor creates a placement client for the potential case where
    it needs to make a call for certain operations. A transient network
    or keystone failure will currently cause it to abort startup, which
    means it is not available for other unrelated activities, such as
    DB proxying for compute.

    This makes conductor test the placement client on startup, but only
    abort startup on errors that are highly likely to be permanent
    configuration errors, and only warn about things like being unable
    to contact keystone/placement during initialization. If a non-fatal
    error is encountered at startup, later operations needing the
    placement client will retry initialization.

    Closes-Bug: #1846820
    Change-Id: Idb7fcbce0c9562e7b9bd3e80f2a6d4b9bc286830
    (cherry picked from commit 232684b44022f1bc4d72b07045900780de456e63)
    (cherry picked from commit 19346082058d51c78bb157ca5e1304d15691dd9a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova xena-eom release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/859001
Reason: stable/wallaby branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/859002
Reason: stable/wallaby branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.