remove-unit of nova-compute does not actually remove the nova service or the neutron agent

[Nicolas Thomas]

When removing units successfully from Juju points of you it does not appear in Horizon as expected.

How to reproduce : deploy an Icehouse on precise ... with several units. Check in horizon. remove units and check horizon nothing is done.

Expected result compute removed.

[James Page (james-page)]

Believe it or not this is quite tricky to fix as even if you terminate the service unit etc... nova will still have a record of it in its database.

That said, it won't be used for scheduling as it should be marked as down in some way.

[Liang Chen (cbjchen)]

Navigating to http://<horizon-host>/horizon/admin/info/, under the "Compute Service" tab, the removed service would be marked as state down.

[Nicolas Thomas (thomnico)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I assume it will be the same if the server crashed.. whereas here the
operator made a conscious decision to remove a unit. Possible scenario
a rolling hardware upgrade.

On 03/12/2014 21:44, Liang Chen wrote:
> Navigating to http://<horizon-host>/horizon/admin/info/, under the
> "Compute Service" tab, the removed service would be marked as
> state down.
>

- --
Best Regards,
Nicolas Thomas - Solution Architect - Canonical
http://insights.ubuntu.com/?p=889
GPG FPR: D592 4185 F099 9031 6590 6292 492F C740 F03A 7EB9
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJUhXFRAAoJEEkvx0DwOn65Tm4IAJ+uKo7zJDCzEZzVmr84UZf1
Jv/QHqbM45E4V15QUwLU8Ctid8yyGUJk6HR951DH6aWozh77IL0FzOoGd28ychVM
RgOioE+3yJmll7RyjD+cHSu7Ncw0SEeCg93D+XEOzWeDYoUxAQVd66Y1QYYAp1Jr
IAE2YAMWhPws3hGnkdFOGfdfoO9Zv5lgcmSK8hPfF/VXqjb8eiZPxhJWNrQjQALY
95yrf30ofVZrOHCkyJ6na/Kjw7OY9ncXo6+w6s6iaQdYb2xnbWXyWv+trlAJ65I8
4oKq9PQltFQD5byUMFIVfL32tKhg70EGQ9Xg4g9ISAdtD/3VMTjgNVPyxJfCrjA=
=WHfI
-----END PGP SIGNATURE-----

[Liang Chen (cbjchen)]

Sure. Icehouse added a new API - os-extended-services-delete that seems to be useful one for this case.

[Liang Chen (cbjchen)]

But calling admin API from inside charms doesn't seem good. Need to find another way to do this.

[JuanJo Ciarlante (jjo)]

Just a note on the operational side of this issue: current behavior
has actually saved us from having to migrate away hosted VMs for a
hardware upgrade.

Although not needed (we could have just taken down
nova-compute at the node whilst), we did remove-unit/add-unit
against the same MaaS'd host, and was great to see the hosted
VMs still there.

[Liang Chen (cbjchen)]

That's the reason nova only exposed enable/disable services before. os-extended-services-delete is used when retiring a compute node, through it doesn't destroy the hosted VMs either.

[Liang Chen (cbjchen)]

I submitted a patch to remove services from the nova-manage util - https://review.openstack.org/#/c/141105/. Once that gets merged, nova-compute charm can use a stop hook to completely wipe off the corresponding service.

[Nicolas Thomas (thomnico)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

An expected behavior if live migration is enabled will be that the VMs
on that node get migrated before stopping.

My 2cents

On 11/12/2014 21:29, Liang Chen wrote:
> ** Changed in: nova-compute (Juju Charms Collection) Status:
> Triaged => In Progress
>

- --
Best Regards,
Nicolas Thomas - Solution Architect - Canonical
http://insights.ubuntu.com/?p=889
GPG FPR: D592 4185 F099 9031 6590 6292 492F C740 F03A 7EB9
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJUispYAAoJEEkvx0DwOn65kHsH/2XHogDpe4F2Dbbw4JqI6fX5
o2aAZm+uyPDUN/hKdvXThaTBRAtO+LaV/JfhWziIJqx8NPf17Kgs8idODhVMKwwB
1g1L72sOErZPAeLDdIz1lzJ3Ra2Ia6xxEwvGyiXvZAw9uD7joMKvyTgaGggXDj+P
0I1OdQZyo7vtslJIBFUI9OEhEbK4JbprscfWDzjkFJEUjOrnqPXk/cOJIbD6V+xd
37ze3B/JxES/SWnyarANaTSmevAJEgVV3gmMAy7UTmfV5ikxH3lPWwncvqqAzNUr
fBK/GCM1Ysakdc3Od0VTqHG6Avh1WUTAQr9aciFrrDrsaV+4rVjWPJ6JmJfBgbc=
=OzZk
-----END PGP SIGNATURE-----

[Liang Chen (cbjchen)]

That seems good. And even if live migration is not enabled, the VMs can still be evacuated to other compute nodes. But let's now focus on the removal of the orphaned service records - doing similar thing as the os-extended-services-delete admin API does for this bug. We can file another bug/bp for the migration/evacuation feature, and handle it there.

[Edward Hope-Morley (hopem)]

Note that nova-compute nodes do not have direct db access (they go though nova-conductor) so a nova-manage operation would have to be performed by a service that does have direct access e.g. nova-cloud-controller.

[Liang Chen (cbjchen)]

Thank you very much for the reminder. You are absolutely right. The nova-manage operation will have to be performed at where the nova-conductor is deployed (nova-cloud-controller). That way, the database access can be ensured. So I am thinking of putting the nova-manage invocation to somewhere like the compute relation departure hook in nova-cloud-controller charm.

[Junien F (axino)]

This is still the case with pike and 17.11 charms. Neither the "nova-compute" nova service nor the Open vSwitch neutron agent are removed from the DB. My experience is that this causes problems when reinstalling a node.

[Wouter van Bommel (woutervb)]

Just hit this bug today on a cloud running 17.04 charms, quite annoying esp as the host was in multiple aggregates, and removed the host first, before realizing this. So that became quit a challenge.

[Ramon Grullon (rgrullon)]

Hit this issue on Bionic running Queens.

Had to go into the mysql db and search nova-api database until found node in host_mappings table and removed it.

[Peter Matulis (petermatulis)]

The first part of this bug (nova service) was fixed via bug 1691998.

[macchese (max-liccardo)]

this bug affects also openstack 2023.2 Bobcat

[Alex Kavanagh (ajkavanagh)]

Hi @macchese; when you say it still affects 2023.2 does that include both the nova-service as well as the neutron agent? (i.e. did you do the actions as indicated via bug 1691998 and in the charm guide? https://docs.openstack.org/charm-guide/latest/admin/ops-scale-back-nova-compute.html) Thanks.

[macchese (max-liccardo)]

sorry, I didn't.
I used remove-unit only and then some sql command to remove the hypervisor.