Adding a unit to an existing service overwrites the vip in the catalogue

[Leonardo Borda]

When adding and removing a new nova-cloud-controller we notice that juju registers that new controller as the new nova-api endpoint. When we remove that same unit it does not update the VIP.

$ juju-deployer -f keystone
2016-02-10 13:59:20 Service: keystone address: 10.5.0.219
10.5.0.219

$ keystone endpoint-list
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+
| id | region | publicurl | internalurl | adminurl | service_id |
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+
| 14eb9c3acb124c998056952e0a8a46d2 | RegionOne | http://10.5.0.224:8774/v2/$(tenant_id)s | http://10.5.0.224:8774/v2/$(tenant_id)s | http://10.5.0.224:8774/v2/$(tenant_id)s | 3f5fc02efa084a41b8bceb7f59599547 |
| 15e648ddbd1742738c92346a9d2c57f5 | RegionOne | http://10.5.0.218:8000/v1 | http://10.5.0.218:8000/v1 | http://10.5.0.218:8000/v1 | 98a4e475fe114978aad641730199a95c |
| 33e17dad386e4dc2b9acc87057929bc4 | RegionOne | http://10.5.0.219:5000/v2.0 | http://10.5.0.219:5000/v2.0 | http://10.5.0.219:35357/v2.0 | 11e189355f734b5f9d7a755dfc4ebaa0 |
| 556ce143e6564fff8aef61fd6d51fe0b | RegionOne | http://10.5.0.216:8776/v2/$(tenant_id)s | http://10.5.0.216:8776/v2/$(tenant_id)s | http://10.5.0.216:8776/v2/$(tenant_id)s | 6a609c0d87d44f2b96dbb1413014d018 |
| 8c23368a7694465eb00e4c1042aa6144 | RegionOne | http://10.5.0.212:8777 | http://10.5.0.212:8777 | http://10.5.0.212:8777 | a896e1fde4cf4b7b9b08d74688743d39 |
| 9da5dd218ec546ae908b56bb6060c8b6 | RegionOne | http://10.5.0.218:8004/v1/$(tenant_id)s | http://10.5.0.218:8004/v1/$(tenant_id)s | http://10.5.0.218:8004/v1/$(tenant_id)s | 30b8c40ec3954b8f874cadb65a4153a5 |
| a76c5fbb2f384b58b4f45dabe8c3a744 | RegionOne | http://10.5.0.230:8080/v1/AUTH_$(tenant_id)s | http://10.5.0.230:8080/v1/AUTH_$(tenant_id)s | http://10.5.0.230:8080 | 06965a3aa2664622997c289e3890babe |
| b1f50dd93b5043c488cdc5bbba5d2863 | RegionOne | http://10.5.0.217:9292 | http://10.5.0.217:9292 | http://10.5.0.217:9292 | 77030f9ab1834ef284f25a8d3528d782 |
| b2ec91885f204c57ba1359366a7be34f | RegionOne | http://10.5.0.216:8776/v1/$(tenant_id)s | http://10.5.0.216:8776/v1/$(tenant_id)s | http://10.5.0.216:8776/v1/$(tenant_id)s | f6cc6ea9f8fc44acb6083a5541339f0a |
| fda0adfd56074a5a855612fc1c230d3b | RegionOne | http://10.5.0.222:9696 | http://10.5.0.222:9696 | http://10.5.0.222:9696 | 43436fb4474f4e6fb54da09c6c8531b0 |
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+

$ juju add-unit nova-cloud-controller

$ !806
keystone endpoint-list
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+
| id | region | publicurl | internalurl | adminurl | service_id |
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+
| 15e648ddbd1742738c92346a9d2c57f5 | RegionOne | http://10.5.0.218:8000/v1 | http://10.5.0.218:8000/v1 | http://10.5.0.218:8000/v1 | 98a4e475fe114978aad641730199a95c |
| 33e17dad386e4dc2b9acc87057929bc4 | RegionOne | http://10.5.0.219:5000/v2.0 | http://10.5.0.219:5000/v2.0 | http://10.5.0.219:35357/v2.0 | 11e189355f734b5f9d7a755dfc4ebaa0 |
| 556ce143e6564fff8aef61fd6d51fe0b | RegionOne | http://10.5.0.216:8776/v2/$(tenant_id)s | http://10.5.0.216:8776/v2/$(tenant_id)s | http://10.5.0.216:8776/v2/$(tenant_id)s | 6a609c0d87d44f2b96dbb1413014d018 |
| 8c23368a7694465eb00e4c1042aa6144 | RegionOne | http://10.5.0.212:8777 | http://10.5.0.212:8777 | http://10.5.0.212:8777 | a896e1fde4cf4b7b9b08d74688743d39 |
| 9da5dd218ec546ae908b56bb6060c8b6 | RegionOne | http://10.5.0.218:8004/v1/$(tenant_id)s | http://10.5.0.218:8004/v1/$(tenant_id)s | http://10.5.0.218:8004/v1/$(tenant_id)s | 30b8c40ec3954b8f874cadb65a4153a5 |
| a76c5fbb2f384b58b4f45dabe8c3a744 | RegionOne | http://10.5.0.230:8080/v1/AUTH_$(tenant_id)s | http://10.5.0.230:8080/v1/AUTH_$(tenant_id)s | http://10.5.0.230:8080 | 06965a3aa2664622997c289e3890babe |
| b1f50dd93b5043c488cdc5bbba5d2863 | RegionOne | http://10.5.0.217:9292 | http://10.5.0.217:9292 | http://10.5.0.217:9292 | 77030f9ab1834ef284f25a8d3528d782 |
| b2ec91885f204c57ba1359366a7be34f | RegionOne | http://10.5.0.216:8776/v1/$(tenant_id)s | http://10.5.0.216:8776/v1/$(tenant_id)s | http://10.5.0.216:8776/v1/$(tenant_id)s | f6cc6ea9f8fc44acb6083a5541339f0a |
| d9516d98cf8447fea2ff8e3a4c673fdd | RegionOne | http://10.5.0.235:8774/v2/$(tenant_id)s | http://10.5.0.235:8774/v2/$(tenant_id)s | http://10.5.0.235:8774/v2/$(tenant_id)s | 3f5fc02efa084a41b8bceb7f59599547 |
| fda0adfd56074a5a855612fc1c230d3b | RegionOne | http://10.5.0.222:9696 | http://10.5.0.222:9696 | http://10.5.0.222:9696 | 43436fb4474f4e6fb54da09c6c8531b0 |
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+

It's changed from http://10.5.0.224:8774 to http://10.5.0.235:8774

$juju remove-unit nova-cloud-controller/2

then we lose connection to the nova-api

$ nova list
ERROR: HTTPConnectionPool(host='10.5.0.235', port=8774): Max retries exceeded with url: /v2/705c2be789d847689909184d29615784/servers/detail (Caused by <class 'socket.error'>: [Errno 113] No route to host)

[Leonardo Borda (lborda)]

same thing with next-ha.yaml
http://bazaar.launchpad.net/~ost-maintainers/openstack-charm-testing/trunk/view/head:/bundles/dev/next-ha.yaml

[Leonardo Borda (lborda)]

workaround:
find out legacy_endpoint_id x keystone endpoint-list and change the database directly.

use keystone
select * from endpoint where legacy_endpoint_id='<id>'
update endpoint set url='http://<ip_nova-cloud-controller-vip>:8774/v2/$(tenant_id)s' where legacy_endpoint_id='<id>';

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/508845

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/509121

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/509133

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/509164

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/509172

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/508845
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=36e95189217d1cd68de34985e90382cb39020006
Submitter: Jenkins
Branch: master

commit 36e95189217d1cd68de34985e90382cb39020006
Author: Liam Young <email address hidden>
Date: Mon Oct 2 10:43:13 2017 +0000

    Block endpoint reg if cluster partially formed

    When an existing cluster of the service is scaled out the new unit
    will join with keystone before it is fully clustered. In identity
    joined hook the charmhelpers function canonical_url is called which
    in turn uses another charmhelpers function, resolve_address.
    resolve_address will only return the vip if the vip is set in config
    AND the unit is clustered. This means that the units local address
    is returned and that is then registered with keystone.

    This change gates registering an endpoint if the cluster is
    partially formed.

    Change-Id: Ibe7d2286de130f045257ad6552f7a5b1279958b6
    Partial-Bug: #1544959

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/510034

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/510034
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=1fcbc696490e4b130837ea574c7afc4f0350b788
Submitter: Jenkins
Branch: master

commit 1fcbc696490e4b130837ea574c7afc4f0350b788
Author: Liam Young <email address hidden>
Date: Fri Oct 6 09:34:06 2017 +0000

    Return early from identity-service-relation-joined

    After the previous commit landed it was suggested on that it would
    be cleaner to return early from identity-service-relation-joined
    if the cluster was partially formed rather than use a long if/else
    . This change makes that style update.

    Change-Id: I9dab5d930e13dda6cfe2c6e11648587c87bdb955
    Partial-Bug: #1544959

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/509164
Committed: https://git.openstack.org/cgit/openstack/charm-glance/commit/?id=be3fcbcad318ff654538772dcca31722b5b4e5ff
Submitter: Jenkins
Branch: master

commit be3fcbcad318ff654538772dcca31722b5b4e5ff
Author: Liam Young <email address hidden>
Date: Tue Oct 3 10:50:02 2017 +0000

    Block endpoint reg if cluster partially formed

    When an existing cluster of the service is scaled out the new unit
    will join with keystone before it is fully clustered. In identity
    joined hook the charmhelpers function canonical_url is called which
    in turn uses another charmhelpers function, resolve_address.
    resolve_address will only return the vip if the vip is set in config
    AND the unit is clustered. This means that the units local address
    is returned and that is then registered with keystone.

    This change gates registering an endpoint if the cluster is
    partially formed.

    Change-Id: I85e7304b1480a95c784f46490dbcc0c4cc1c3129
    Partial-Bug: #1544959

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/509172
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer/commit/?id=24e359957214893326aa5940bccc8b7399d35350
Submitter: Jenkins
Branch: master

commit 24e359957214893326aa5940bccc8b7399d35350
Author: Liam Young <email address hidden>
Date: Tue Oct 3 13:21:39 2017 +0000

    Block endpoint reg if cluster partially formed

    When an existing cluster of the service is scaled out the new unit
    will join with keystone before it is fully clustered. In identity
    joined hook the charmhelpers function canonical_url is called which
    in turn uses another charmhelpers function, resolve_address.
    resolve_address will only return the vip if the vip is set in config
    AND the unit is clustered. This means that the units local address
    is returned and that is then registered with keystone.

    This change gates registering an endpoint if the cluster is
    partially formed.

    Change-Id: Icf2753b6b21347af8106ebad03d81309177d09f2
    Partial-Bug: #1544959

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/509121
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=6cc620ab270bc0d9f12a2ff3e3b439c7f4e10a90
Submitter: Jenkins
Branch: master

commit 6cc620ab270bc0d9f12a2ff3e3b439c7f4e10a90
Author: Liam Young <email address hidden>
Date: Tue Oct 3 09:38:21 2017 +0000

    Block endpoint reg if cluster partially formed

    When an existing cluster of the service is scaled out the new unit
    will join with keystone before it is fully clustered. In identity
    joined hook the charmhelpers function canonical_url is called which
    in turn uses another charmhelpers function, resolve_address.
    resolve_address will only return the vip if the vip is set in config
    AND the unit is clustered. This means that the units local address
    is returned and that is then registered with keystone.

    This change gates registering an endpoint if the cluster is
    partially formed.

    Change-Id: I233e0cccb8ccd732080fd239df6d1e7db174eba5
    Partial-Bug: #1544959

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/509133
Committed: https://git.openstack.org/cgit/openstack/charm-cinder/commit/?id=1975c4127121bd026bcbc128c21aeb8585e242a2
Submitter: Jenkins
Branch: master

commit 1975c4127121bd026bcbc128c21aeb8585e242a2
Author: Liam Young <email address hidden>
Date: Tue Oct 3 09:54:35 2017 +0000

    Block endpoint reg if cluster partially formed

    When an existing cluster of the service is scaled out the new unit
    will join with keystone before it is fully clustered. In identity
    joined hook the charmhelpers function canonical_url is called which
    in turn uses another charmhelpers function, resolve_address.
    resolve_address will only return the vip if the vip is set in config
    AND the unit is clustered. This means that the units local address
    is returned and that is then registered with keystone.

    This change gates registering an endpoint if the cluster is
    partially formed.

    Change-Id: If483147e17dab8de2883058ee0f2718a3b7f8ca6
    Partial-Bug: #1544959