Placement endpoints not being updated, or perhaps regressing to n-c-c endpoints, even after deploying placement service for train

[Paul Goins]

I recently upgraded a customer cloud from OpenStack Stein to Train, following the charm deployment guide for OpenStack: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/placement-charm-upgrade-to-train.html

At this point, all O7k apps have been upgraded to Train, and we are planning to upgrade to Ussuri. However, we've encountered an issue. We've noted that the placement endpoitns listed via "openstack endpoint list" are using the nova-cloud-controller FQDN rather than the placement service's FQDN.

When I first encountered this, I noted that I had not yet set up certificates for the placement service, so I added a vault to placement relation, after which the endpoints got updated. That was one or two days ago. However, when I checked again today, the endpoints again are using the nova-cc FQDN.

We're also having errors creating and deleting instances, which I'm suspecting may also be related to this.

Curl tests against the nova-cc FQDN on port 8778 fail with "connection refused" errors, as I would expect, while curl tests against the nova-placement FQDN on port 8778 succeeds. Yet, the endpoint list shows the nova-cc FQDN.

[Paul Goins (vultaire)]

I've manually adjusted the placement endpoints via "openstack endpoint set <id> --url <url>".

And while the instance creation issues may or may not be caused by this bug, instance creation is still not resolved, even with the above workaround.

[Trent Lloyd (lathiat)]

Restarting the nova services was apparently needed to remove cached endpoint information. The charms may need to send or trigger on the endpoint updates to trigger restarts.

[Trent Lloyd (lathiat)]

Including during SSL-related updated

[Alex Kavanagh (ajkavanagh)]

So, in order to quash this completely (as a workaround) the left over endpoint information from nova-cc needs to be removed. Example of doing this (change relation ids, etc, as required):

$ juju run -u nova-cloud-controller/0 -- relation-ids identity-service
identity-service:47

$ juju run -u keystone/0 -- relation-list -r47
nova-cloud-controller/0
nova-cloud-controller/1
nova-cloud-controller/3

$ juju run -u keystone/0 -- relation-get -r47 - nova-cloud-controller/0
egress-subnets: 10.***/32
ingress-address: 10.*
nova_admin_url: http://10.*:8774/v2.1
nova_internal_url: http://10.*:8774/v2.1
nova_public_url: http://10.*:8774/v2.1
nova_region: serverstack
nova_service: nova
placement_admin_url: http://10.*:8778
placement_internal_url: http://10.*:8778
placement_public_url: http://10.*:8778
placement_region: serverstack
placement_service: placement
private-address: *
subscribe_ep_change: neutron placement

The items with the "placement_" prefix need to be removed:

$ juju run -a nova-cloud-controller -- relation-set -r47 placement_admin_url= \
    placement_internal_url= placement_public_url= placement_region= placement_service=

Note that no characters come after the '='.

After the cloud has settled, you can verify the keystone end again:

juju run -u keystone/0 -- relation-get -r47 - nova-cloud-controller/0
egress-subnets: 10.*/32
ingress-address: 10.*
nova_admin_url: http://10.*:8774/v2.1
nova_internal_url: http://10.*:8774/v2.1
nova_public_url: http://10.*:8774/v2.1
nova_region: serverstack
nova_service: nova

Then verify that the endpoints are set to the placement unit.

---

The relevant place to fix the code in nova-cloud-controller is:

https://github.com/openstack/charm-nova-cloud-controller/blob/b56572cf6bde1cbc91af1f0bab0aab25a1648efd/hooks/nova_cc_utils.py#L1570

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-deployment-guide/+/792684

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-deployment-guide/+/792684
Committed: https://opendev.org/openstack/charm-deployment-guide/commit/64536f938a7d07bbc4aed41e72f4aa06ce90d2e0
Submitter: "Zuul (22348)"
Branch: master

commit 64536f938a7d07bbc4aed41e72f4aa06ce90d2e0
Author: Peter Matulis <email address hidden>
Date: Fri May 21 13:54:52 2021 -0400

    Add OpenStack upgrade issue - Placement endpoints

    Partial-Bug: #1928992
    Change-Id: I52e81a3f5b1320a2348b11bea94811af42a6f387

[Billy Olsen (billy-olsen)]

Marked invalid against placement charm, this is a bug in the nova-cloud-controller

[Billy Olsen (billy-olsen)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/796953

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/796953
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/231a0f1459eb2a9273dc9b752f10ff184c0b7e35
Submitter: "Zuul (22348)"
Branch: master

commit 231a0f1459eb2a9273dc9b752f10ff184c0b7e35
Author: Billy Olsen <email address hidden>
Date: Thu Jun 17 13:42:48 2021 -0700

    Drop placement endpoints from relation in train+

    When a cloud is deployed earlier than the Train release, the placement
    service is provided by nova-cloud-controller. As part of an upgrade to
    Train, the new placement service is added and updates the placement
    endpoint in the service catalog. The nova-cloud-controller charm no
    longer advertises the placement service URL, but because the data
    exists on the relation until removed, the service catalog changes the
    placement URL to the placement endpoints advertised from
    nova-cloud-controller.

    Fix this by explicitly removing the placement service URLs when the
    placement service is not provided by nova-cloud-controller.

    Change-Id: Ibb3b1429820a4188fe3d2c1142c295c0de4ee24e
    Closes-Bug: #1928992