Instance created in MN-External and Duplex configs cannot be pinged

[Paulina Flores]

Brief Description
-----------------
Instance created in MN-External config cannot be pinged

Severity
--------
Minor

Steps to Reproduce
------------------
1. Create an instance
2. Ping the instance from within its host

Expected Behavior
------------------
The instance should be pingable from within its host

Actual Behavior
----------------
PING 192.168.0.156 (192.168.0.156) 56(84) bytes of data.
From 192.168.0.2 icmp_seq=1 Destination Host Unreachable
From 192.168.0.2 icmp_seq=2 Destination Host Unreachable
From 192.168.0.2 icmp_seq=3 Destination Host Unreachable
From 192.168.0.2 icmp_seq=4 Destination Host Unreachable
From 192.168.0.2 icmp_seq=5 Destination Host Unreachable

--- 192.168.0.156 ping statistics ---
5 packets transmitted, 0 received, +5 errors, 100% packet loss, time 4004ms

Reproducibility
---------------
The issue is 100% reproducible

System Configuration
--------------------
Multi-node system, Dedicated storage. Duplex system.

Branch/Pull Time/Commit
-----------------------
bootimage.iso 2019-May-21 12:38:33
ISO: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190521T132734Z/outputs/iso/
Helm chart: stx-openstack-1.0-13-centos-stable-latest.tgz 2019-May-21 12:38:57

Last Pass
---------
I do not have Last pass information.

Timestamp/Logs
--------------
Attached controller log.

[Erich Cordoba (ericho)]

Hi, just to discard booting problems, can you verify if the instance actually booted? Sometimes if there is a problem with your image (I've seen corrupt cirros images) then the guest OS won't boot.

Can you see and use the instance console?

Then, if the instance is usable, can you please list the network interfaces with ifconfig or ip addr commands ?

[Paulina Flores (paulina-flores)]

Hi. The instances are booting up as normally expected:

+--------------------------------------+---------+--------+-------------------------------+---------------+----------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+---------+--------+-------------------------------+---------------+----------------+
| 8a0e66f3-a0f7-42c8-994d-cf84f43f8de9 | test_vm | ACTIVE | test_vm-network=192.168.0.156 | test_vm-image | test_vm-flavor |
+--------------------------------------+---------+--------+-------------------------------+---------------+----------------+

And here's the network interfaces:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth1000: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 52:54:00:d8:09:07 brd ff:ff:ff:ff:ff:ff
3: eth1001: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 52:54:00:4d:cb:b8 brd ff:ff:ff:ff:ff:ff
4: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:8a:75:bc brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.3/24 brd 10.10.10.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet 10.10.10.2/24 scope global secondary ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe8a:75bc/64 scope link
       valid_lft forever preferred_lft forever
5: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:c3:11:6e brd ff:ff:ff:ff:ff:ff
    inet 192.168.204.3/24 brd 192.168.204.255 scope global ens4:1
       valid_lft forever preferred_lft forever
    inet 192.168.206.3/24 brd 192.168.206.255 scope global ens4:5
       valid_lft forever preferred_lft forever
    inet 169.254.202.2/24 scope global ens4
       valid_lft forever preferred_lft forever
    inet 192.168.206.2/24 scope global secondary ens4
       valid_lft forever preferred_lft forever
    inet 192.168.204.2/24 scope global secondary ens4
       valid_lft forever preferred_lft forever
    inet 192.168.204.6/24 brd 192.168.204.255 scope global secondary ens4
       valid_lft forever preferred_lft forever
    inet 192.168.204.5/24 brd 192.168.204.255 scope global secondary ens4
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fec3:116e/64 scope link
       valid_lft forever preferred_lft forever
6: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:80:b2:0b:46 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:80ff:feb2:b46/64 scope link
       valid_lft forever preferred_lft forever
7: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqu...

[Ghada Khalil (gkhalil)]

Likely a connectivity/setup issue. Suggest getting help from someone from the Mexico test team who's familiar with stx networking to help triage this issue. Please talk to Elio.

The controller logs will not have enough information to investigate.
Some things that you can check:
1) Look at the VM console logs
2) Verify connectivity to the router
3) Verifiy connectivity to default gateway
4) List router subnets
5) Verify ping of VM over VM management network
6) Check the OVS/OVS-DPDK configuration, openflow rules and datapath flows

[Elio Martinez (elio1979)]

I just found that sometimes , after the VM creationg, no matter that the "openstack server list" instruction shows active state. If you get into the console for that instances, the IP that should be attached to the eth0 NIC is not present, no matter that you can see it from the controller. My suggestion should be to force the assigned IP to the VM. Other funny fact is that on matter this. and no matter that all your instances are created using the same network. If they live in different compute nodes. They are not going to listen each other. Just the ones that lives in the same compute. Is this normal?

[Ghada Khalil (gkhalil)]

Elio, As per current email threads, connectivity issues across compute nodes are likely an L2 switch configuration issue.

For both of these reports, someone with networking experience will need to debug this in the specific setup reporting the issue.

[Elio Martinez (elio1979)]

Will work with Paulina about this, thanks

[Ghada Khalil (gkhalil)]

Any update on this issue?

[Ghada Khalil (gkhalil)]

As per Elio, this is still being investigated. The issue is specific to a particular configuration/lab. Matt suggested to ensure that LLDP is enabled on the switches and to look at the LLDP information to see if they can point to the switch misconfiguration.

[Gopinath (gprabakx)]

Hi, I have created two instance in External with Below steps:

1.openstack flavor create --ram 2048 --vcpus 1 --disk 1 --property hw:mem_page_size=large vm_flavor
2.openstack image create --file cirros-0.4.0-x86_64-disk.img --disk-format qcow2 vm_image
3.openstack network create vm_network
4.openstack subnet create --network vm_network --subnet-range 192.168.0.0/24 --ip-version 4 --dhcp vm_subnet
5.openstack server create --nic net-id=75025019-1a3d-4c3f-aed2-a65a36bcf2d9 --flavor vm_flavor --image vm_image vm
6.openstack server create --nic net-id=75025019-1a3d-4c3f-aed2-a65a36bcf2d9 --flavor vm_flavor --image vm_image vm_2
7.Find the host and from host sudo virsh console

Snippet:
compute-0:~$ sudo virsh list
Password:
 Id Name State
-----------------------------------
 1 instance-00000009 running
 2 instance-00000006 running

compute-0:~$ sudo virsh console 1
Connected to domain instance-00000009
Escape character is ^]

It hung and could not perform any command in instance.

Switch Configuration:
switch configuration on vlan:Ports:
Port Configuration
Port Virtual Port Link Auto Speed Duplex Flow Load Media
        router State State Neg Cfg Actual Cfg Actual Cntrl Master Pri Red
================================================================================
7 None E A OFF 10000 10000 FULL FULL NONE Q+CR4F3m
8 None E A OFF 10000 10000 FULL FULL NONE Q+CR4F3m
9 None E A OFF 10000 10000 FULL FULL NONE Q+CR4F3m
10 None E A OFF 10000 10000 FULL FULL NONE Q+CR4F3m
================================================================================
indicates Port Display Name truncated past 8 characters
Link State: A-Active, R-Ready, NP-Port Not Present, L-Loopback
Port State: D-Disabled, E-Enabled, L-License Disabled
Media: !-Unsupported, $-Unlicensed
Media Red: * - use "show port info detail" for redundant media type
Flow Cntrl: Shows link partner's abilities. NONE if Auto Neg is OFF
VLAN:
external1_external 100 192.168.51.1 /24 ---------------------------- ANY 4 /4 data_ext_vr1
external1_internal 101 10.1.1.1 /8 ---------------------------- ANY 4 /4 data_ext_vr1
external1_private 102 192.168.201.1 /24 ---------------------------- ANY 4 /4 data_ext_vr1
external1_public 103 192.168.101.1 /24 ---------------------------- ANY 4 /4 data_ext_vr1
Virtual router:
Virtual Router : data_ext_vr1 Type : User
Operational State : Up
IPv4 Admin State : Enabled IPv6 Admin State : Enabled
IPv4 Route Sharing : Disabled IPv6 Route Sharing : Disabled
L3VPN SNMP Traps : Disabled
VLANs : data_external_1, external1_external, external1_internal,
                       external1_private, external1_public
Virtual Router Totals :
Total Protocols : 0 Max Protocols : 8
Total Ports : 0
Total VLANs : 5
Total IPv4 VLANs : 4 Total IPv6 VLANs : 0
Active IPv4 VLANs : 4 Active IPv6 VLANs : 0
Inactive IPv4 VLANs : ...

[Gopinath (gprabakx)]

I have tried with alpine ISO and still the VM console hangs. ISO : http://dl-cdn.alpinelinux.org/alpine/v3.10/releases/x86_64/alpine-standard-3.10.1-x86_64.iso

[Elio Martinez (elio1979)]

This bug was caused because a L2 configuration and it is fixed on testing environments. We can close this bug

[Cristopher Lemus (cjlemusc)]

Hola Vic, buen día!

Platicamos hace unos minutos en el pasillo del laboratorio respecto a un rack sin energía.

El siguiente es el ticket: INTGM-Z005006, no tenemos energía eléctrica en nuestro rack H0066 ubicado en la fila S11.

  * Si se va a cambiar el PDU es posible:
     * Orientar los enchufes hacia afuera para no doblar los cables?
     * Instalar un PDU adicional para balancear la carga? Comprendo que este punto depende de si hay o no circuito para conectarlo, ojala sea posible.

Ya deje la regleta actual completamente desconectada, sin ningún cable, para que sea fácil reemplazarla.

De antemano, muchas gracias por la ayuda!

Cristopher Lemus
<javascript:void(0)>