vm did not acquire an ipv4 address

[Elio Martinez]

Brief Description
----------------
Instance schedules and shows as "ACTIVE" but getting into the instance there is no IP assigned

Severity
--------
Critical

Steps to Reproduce
------------------

1. Create an instance throw horizon or CLI with or without security group or keypar
     openstack server create --image centos --flavor m1.medium --network public-net0 --
     security-group security1 --key-name key1 <instance name>

2. Verify that your instance is up and active

 openstack server list
+--------------------------------------+---------+--------+----------------------------+-------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+---------+--------+----------------------------+-------+-----------+
| d650a5de-7336-43e0-8d82-1abaa46510f4 | centos2 | ACTIVE | public-net0=192.168.101.52 | | m1.medium |
| e349334b-3a5e-4e0c-ad77-c05497ce2336 | centos | ACTIVE | public-net0=192.168.101.69 | | m1.medium |
+--------------------------------------+---------+--------+----------------------------+-------+-----------+

3. Verify that you instance has the same ip showed
   sudo virsh console <instance>

Expected Behavior
------------------

The instance should have the same IP in order to perform simple communication through TCP or ICMP according with the default security group or custom made security group rules.

Actual Behavior
----------------

There is no assigned IP for any instance. Not communication even if you force the ip assigning the same showed before.

--Inside the Instance:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet6 fe80::f816:3eff:fe88:9dd9 prefixlen 64 scopeid 0x20<link>
        ether fa:16:3e:88:9d:d9 txqueuelen 1000 (Ethernet)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 32 bytes 8856 (8.6 KiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10<host>
        loop txqueuelen 1000 (Local Loopback)
        RX packets 6 bytes 416 (416.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 6 bytes 416 (416.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

The instance is showing the following error during booting up:

2s)[K[[1;31mFAILED[0m] Failed to start LSB: Bring up/down networking.
See 'systemctl status network.service' for details.
         Starting Initial cloud-init job (metadata service crawler)...
[[32m OK [0m] Reached target Network.
         Starting Dynamic System Tuning Daemon...
         Starting Postfix Mail Transport Agent...
[ 304.828028] cloud-init[765]: Cloud-init v. 0.7.9 running 'init' at Tue, 05 Nov 2019 10:41:26 +0000. Up 304.80 seconds.
[ 304.859081] cloud-init[765]: ci-info: +++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++
[ 304.861210] cloud-init[765]: ci-info: +--------+------+-----------+-----------+-------+-------------------+
[ 304.863153] cloud-init[765]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 304.865078] cloud-init[765]: ci-info: +--------+------+-----------+-----------+-------+-------------------+
[ 304.867040] cloud-init[765]: ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . |
[ 304.869118] cloud-init[765]: ci-info: | lo: | True | . | . | d | . |
[ 304.871125] cloud-init[765]: ci-info: | eth0: | True | . | . | . | fa:16:3e:88:9d:d9 |
[ 304.873062] cloud-init[765]: ci-info: | eth0: | True | . | . | d | fa:16:3e:88:9d:d9 |
[ 304.875019] cloud-init[765]: ci-info: +--------+------+-----------+-----------+-------+-------------------+
[[32m OK [0m] Started Dynamic System Tuning Daemon.
[ 305.000066] cloud-init[765]: 2019-11-05 10:41:26,461 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [('Connection aborted.', error(101, 'Network is unreachable'))]

No matter the image, centos or cirros, the problem is the same

Reproducibility
---------------

100 % with all configurations

System Configuration
--------------------

All configurations with IPV4

Branch/Pull Time/Commit
-----------------------
###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.09"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20191101T013000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="303"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-11-01 01:30:00 +0000"

Last Pass
---------

BUILD_ID="20191018T013000Z"

Timestamp/Logs
--------------

Attached

Test Activity
-------------

Regression 3.0

[Ghada Khalil (gkhalil)]

Assigning to networking PL for next steps

[Ricardo Perez (richomx)]

Neither creating a rule for traffic in port 22 for the security group used (security1) the WA works, this is still a blocker for QA test execution.

openstack security group rule create --proto tcp --dst-port 22 security1

[Ghada Khalil (gkhalil)]

Marking as stx.3.0 / critical as this is blocking networking regression for stx.3.0

[Le, Huifeng (hle2)]

@Elio, As this is a critical issue, could your team help to do the ping test on daily build (from 2019.10.18 to 2019.11.1) to find out which build first introduce this issue? At the meantime, we will start to reproduce the issue in latest build and do the investigation. Thanks!

[ChenjieXu (midone)]

Hi Elio,

I checked your environment and find you are using OVS-DPDK. But you haven't configured flavor with property “hw:mem_page_size=large”. You need to add this property as below:
openstack flavor set $i --property hw:mem_page_size=large

Your flavor is listed below:
openstack flavor show m1.medium
+----------------------------+--------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| access_project_ids | None |
| disk | 40 |
| id | fdcf63e8-4ee5-4993-855d-52968354d6e8 |
| name | m1.medium |
| os-flavor-access:is_public | True |
| properties | |
| ram | 4096 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 2 |
+----------------------------+--------------------------------------+

I have added this property to m1.medium on your environment and create vm1. vm1 gets IP successfully and can ping vm1 in dhcp namespace.

[Elio Martinez (elio1979)]

Yes Chenjie, you are right

I didn't check the property after setting the flavor. Ping between instances and ssh connection from node to instances is working.

[Ghada Khalil (gkhalil)]

Marking as Invalid since this is a procedural issue.
Thanks Chenjie for getting to the bottom of this.

[Ghada Khalil (gkhalil)]

As a side note, this was previously mentioned in https://bugs.launchpad.net/starlingx/+bug/1820378 and was noted in the installation guide wiki at the time

[Ricardo Perez (richomx)]

@all, just a quick comment here, beside you use this parameter in the flavor:

hw:mem_page_size=large

using OVS-DPDK in an external configuration, you won't get an IP address inside the VM.

Horizon will show you an assigned IP by the system, but you must go inside the VM (Using Horizon console or virsh), assign manually such IP address. Which I believe isn't the expected behavior.