"metadata_proxy_shared_secret" missmatch after migration ML2+OVS to ML2+OVN

Bug #1916609 reported by TWENTY |20
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Nova Compute Charm
Confirmed
High
Liam Young

Bug Description

After migration from ML2+OVS to ML2+OVN there are different values for the metadata_proxy_shared_secret option in /etc/nova/nova.conf and /etc/neutron/neutron_ovn_metadata_agent.ini
As a result, I cannot reach the metadata server in a newly created instance. This means, for example, that the ssh key is not created correctly in the instance.

Test:
juju run --application nova-compute 'grep metadata_proxy_shared_secret /etc/nova/nova.conf /etc/neutron/neutron_ovn_metadata_agent.ini'

- Stdout: |
    /etc/nova/nova.conf:metadata_proxy_shared_secret = e4578821-b0d0-4ad9-b80e-1309c875908c
    /etc/neutron/neutron_ovn_metadata_agent.ini:metadata_proxy_shared_secret=8a883e56-1f27-4e1d-a496-238856d6dd14
  UnitId: nova-compute/0
- Stdout: |
    /etc/nova/nova.conf:metadata_proxy_shared_secret = 5b42bd2a-1ad2-4c59-8819-995c55361dd7
    /etc/neutron/neutron_ovn_metadata_agent.ini:metadata_proxy_shared_secret=36b16951-ef0e-416f-9959-d3117e889788
  UnitId: nova-compute/1
- Stdout: |
    /etc/nova/nova.conf:metadata_proxy_shared_secret = a23a4a6e-00a8-4afd-bc86-cd1654b83ccd
    /etc/neutron/neutron_ovn_metadata_agent.ini:metadata_proxy_shared_secret=fc687165-67b1-4ab5-bf24-8c3db0cca868
  UnitId: nova-compute/2

Woraround:

I trigger a change in nova-compute charm options, for example debug on/off

juju config nova-compute debug=true
juju config nova-compute debug=false

After this the secrets are correct now:

juju run --application nova-compute 'grep metadata_proxy_shared_secret /etc/nova/nova.conf /etc/neutron/neutron_ovn_metadata_agent.ini'
- Stdout: |
    /etc/nova/nova.conf:metadata_proxy_shared_secret = 8a883e56-1f27-4e1d-a496-238856d6dd14
    /etc/neutron/neutron_ovn_metadata_agent.ini:metadata_proxy_shared_secret=8a883e56-1f27-4e1d-a496-238856d6dd14
  UnitId: nova-compute/0
- Stdout: |
    /etc/nova/nova.conf:metadata_proxy_shared_secret = 36b16951-ef0e-416f-9959-d3117e889788
    /etc/neutron/neutron_ovn_metadata_agent.ini:metadata_proxy_shared_secret=36b16951-ef0e-416f-9959-d3117e889788
  UnitId: nova-compute/1
- Stdout: |
    /etc/nova/nova.conf:metadata_proxy_shared_secret = fc687165-67b1-4ab5-bf24-8c3db0cca868
    /etc/neutron/neutron_ovn_metadata_agent.ini:metadata_proxy_shared_secret=fc687165-67b1-4ab5-bf24-8c3db0cca868

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Hi Matthias

Please could you add the following to the bug report:

1. Ubuntu version (as applicable)
2. OpenStack version
3. Charm versions or the pre- and post- bundles.
4. What steps you did migrate from ML2+OVS to ML2+OVN.
5. Log files (if possible) from a nova-compute unit.

Many thanks.

Changed in charm-nova-compute:
status: New → Incomplete
Revision history for this message
TWENTY |20 (tw20) wrote :

Hi Alex, thanks for the fast respond.

1. Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-65-generic x86_64)

2. Openstack Ussuri

3. Charm-Version #325 before and after migration

4. I followed this official documentation for migration: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-ovn.html#migration-from-neutron-ml2-ovs-to-ml2-ovn

I did all steps in this documentation, but without the deployment of ovn-dedicated-chassis units, because we don't want the extra nodes.

5. I have currently not kept any log files. I will try to add it here in a new test.

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for OpenStack nova-compute charm because there has been no activity for 60 days.]

Changed in charm-nova-compute:
status: Incomplete → Expired
Changed in charm-nova-compute:
status: Expired → New
Revision history for this message
Liam Young (gnuoy) wrote :

I think I have finally got to the bottom of this. When the nova-compute charm looks for the metadata-shared-secret it inspects all the units of all the applications related to it on the neutron-plugin relation *1 . There is usually only one application but after the ovn migration there are two (neutron-openvswitch and ovn-chassis).

Due to the way the relations are inspected its mostly luck whether the charm picks the secret from the old neutron-openvswitch relation or the new ovn-chassis relation. The relation-ids seem to be sorted as strings *2 so although its luck which one the secret comes from it should be consistent from then on. But, from the bug description, that is not whats happening. It appears that the old secret is picked up initially and then after a config-changed hook it switches to the new one. The only way I can account for this is if the neutron-openvswitch unit was removed after the upgrade but before the debug=true trigger. This is exactly what the documentation suggests that you do.

I have manually simulated this and sure enough if the neutron-openvswtich relation id comes last in a string ordered list of neutron-plugin relation ids then the wrong metadata secret will be written. Even removing the neutron-openvswitch application will not trigger the secret to be corrected. But manually triggering a config changed hook will cause the compute charm to reinspect its relation and update nova.conf with the correct secret.

*1 https://opendev.org/openstack/charm-nova-compute/src/branch/master/hooks/nova_compute_context.py#L127
*2 https://github.com/juju/juju/blame/develop/worker/uniter/runner/jujuc/relation-ids.go#L83

Changed in charm-nova-compute:
status: New → Confirmed
importance: Undecided → High
Liam Young (gnuoy)
Changed in charm-nova-compute:
assignee: nobody → Liam Young (gnuoy)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.