StarlingX R2 duplex: VM not getting IP assigned to the vlan network when re-spawned on controller-0 after shutting down controller-1

[Akshay]

Brief Description
-----------------
Setup: I have deployed Bare Metal StarlingX R2 duplex mode. While testing HA, I tested a case in which I have spawned 2 VMs from horizon together on IPv6 flat and IPv6 vlan networks. One gets spawned on controller-0 and other on controller-1. Both VM gets IP assigned on each network and they are able to ping each other.

Test Case: Now I switched off the controller-1 and the VM on controller-1 gets rebuilt on controller-0 showing same IPs as of earlier on horizon.

Issue: But when I tried to check inside the VM, it gets IP on flat network but it did NOT get any IP assigned on the vlan network.

I tried this case many times with same result.
Please guide me to solve this issue.

Severity
--------

Critical

Steps to Reproduce
------------------
1. Deploy Bare Metal StarlingX R2 duplex mode.
2. Spawn 2 VMs together on IPv6 flat and IPv6 vlan from horizon.
3. Switch off the controller-1.
4. Check the rebuilt VM whether it got IPs assigned or not.

Expected Behavior
------------------
Rebuild VM should get the IP assigned inside the VM similar to as visible on horizon.

Actual Behavior
----------------
Rebuilt VM not getting IP assigned on vlan network.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Two node system

Last Pass
---------
NO

[Ghada Khalil (gkhalil)]

Assigning to the networking team to triage/investigate. This issue is not reported for any stx.2.0 testing done by the verification teams. It may be a procedural issue.

[marvin Yu (marvin-yu)]

Hi Akshay,

Could you please provide the log when VM gets rebulid on controller-0? This bug is similar to the previous one. I suspect that the vlan dhcp-agent hasn`t been reconstructed on controller-0.

[Akshay (yadavakshay58)]

Hi Marvin,

Which logs ?
1. Are you asking about the "nova-conductor.log" or nova components related logs ?
   -If yes, I can't find any logs in any nova containers. I can't even enable it in configuration files as it
    states "Changing a read-only file" and "Can't open file for writing".
2. or you are asking for >collect all ?
   -If yes, then for which controller node?

[marvin Yu (marvin-yu)]

Hi Akshay,

It`s ok, I just want to see instance console log. I have tried to reproduce this bug, but I've run into some problems now. could you please provide more detailed steps? e.g. steps to create ipv6 network, steps to create vm.

I also have several questions as below:
1. When the VM is created, does the VM get IPv6 address inside ?
2. Which controller node does the flat VM run on?

[Akshay (yadavakshay58)]

Sure Marvin,

Answers to your queries:
1. Yes, it gets IPv6 address assigned inside the VM initially before HA.
2. Flat IPv6 and vlan IPv6 both networks are attached on VMs, Initially both VMs have IPs assigned on both the networks ON EACH CONTROLLER NODE.

Steps tp create IPv6 networks:
1. I have created both the networks from horizon with abcd:abcd:abcd::0/64 as subnet and IPv6 as network type.
2. For flat_IPv6, physical network: physnet0
3. For vlan_IPv6, physical network: physnet1

Steps to create VM:
1. Image:centos
2. Networks: flat_IPv6, vlan_IPv6
3. No Security group added.
4. Cloud Config used only to change login credentials as username:centos and passwd:XXXXXX.

[Akshay (yadavakshay58)]

HI Marvin,

One new finding for your information !

When I tried the above same test case WITH VOLUMES ATTACHED to VMs, then I didn't face this VLAN IP issue. Respawned VM gets IPs assigned as expected.

[marvin Yu (marvin-yu)]

Hi Akshay,
Thanks for the info. This seems to be an occasional bug, I'll keep an eye on it. If you have any new discoveries, please let me know. thanks a lot~

[Akshay (yadavakshay58)]

Hi Marvin,

I have tested both the cases (with volume and without volume) many times and it is reproducible every time for me. So I don't think it is an occasional bug.

[marvin Yu (marvin-yu)]

Hi Akshay,

which ipv6-address-mode did you specify?

[marvin Yu (marvin-yu)]

and could you please provide the openstack image file?
Thanks a lot.

[ANIRUDH GUPTA (anyrude10)]

Hi Team,

I am creating 2 VM simultaneously on VLAN IPv4 network using cirros image.
Out of the 2 VM, one VM gets spawned on Controller-0 and other on Controller-1.

The VM which gets spawned on Controller-1 is failing the lease of getting IP as a result of which there is no IP on the VM interface, which gets spawned on Controller-1.

The Controller-0 VM successfully comes up.

I am attaching "collect" logs of both the controllers for your reference.

[Matt Peters (mpeters-wrs)]

The network configuration can be configured via cloud-init, but for IPv6 the metadata service is not available. Therefore you have to use the config-drive=true when booting a nova instance.

Manual network configuration, you can specify additional configuration via cloud-init. This will require a #cloud-config network entry. For the latest ubuntu images, this would use netplan to configure the static addressing.

[Yi Wang (wangyi4)]

@Matt I just take over this bug. Could you share more details of the network configuration? I am not familiar with cloud-init.

[Yi Wang (wangyi4)]

@Akshay, which centos image you used? Could you point out me where to download it?

[Akshay (akshay346)]

Hi Marvin,

I did not specify any IPv6 mode. I used "Not specified (default)" option.

[Ghada Khalil (gkhalil)]

Based on review in the starlingx networking team on 2020-02-20, we believe this is a configuration/setup issue. There is nothing in the starlingx software that would result in this issue. See comment above from Matt Peters:
https://bugs.launchpad.net/starlingx/+bug/1849655/comments/12

Regarding IPv6, the initial issue was reported for IPv6.
Steps to Reproduce:
2. Spawn 2 VMs together on IPv6 flat and IPv6 vlan from horizon.

Will leave open for reporter to provide more information, but lowering the priority.