Displayed IP address is different than in the VM

[Hanif Kukkalli]

Hi All,

I have an unusual issue with OpenStack Pike release when I create a VM.
The IP address assigned to the VM is different than the IP address in the VM.
For example, the assigned IP in the Dashboard shows 10.20.5.14, but the IP address in the VM will be 10.20.5.17.

I am using Layer 2, Flat Provider network in my configuration.
Has anyone encountered such an issue?

I have attached the console log of the VM machine.

Regards,
Hanif Kukkalli

[Slawek Kaplonski (slaweq)]

In attached console log I can see that You have configured IP 10.30.5.14 inside VM (via DHCP) so it looks fine IMHO.
Can You attach more logs, e.g. result of commands like "nova show" and "neutron port show" and also result of e.g. "ip a" command done inside VM?
Also please check how Your DHCP agent is configured. Is there something strange in logs there?

[Hanif Kukkalli (kukkalli)]

This is the only running instance.

openstack server show ae03d805-a440-48ef-9674-134dc27f49f6
+-------------------------------------+----------------------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | Compute03 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | Compute03 |
| OS-EXT-SRV-ATTR:instance_name | instance-0000002d |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2017-12-13T17:56:50.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | highstreet=10.20.5.17 |
| config_drive | |
| created | 2017-12-13T17:56:34Z |
| flavor | odl_flavor (5) |
| hostId | 4a5e7952347e408740b35db577cf98fb3c20c74d3e2655a1f7b0fd38 |
| id | ae03d805-a440-48ef-9674-134dc27f49f6 |
| image | ODL_Boron_SR3 (65704783-160f-4bb2-89bd-9ad812c26ad7) |
| key_name | highstreet |
| name | vm1-sdn-controller |
| progress | 0 |
| project_id | 918f27ec138847428537296faf5ba65c |
| properties | |
| security_groups | name='default' |
| status | ACTIVE |
| updated | 2017-12-13T17:56:50Z |
| user_id | 80bfe53a1c2d45a78ddbf6b6463bdb7c |
| volumes_attached ...

[Brian Haley (brian-haley)]

Since this is a provider network is the VM actually getting DHCP from the neutron dhcp agent? Or is there something in the provider network responding?

[Hanif Kukkalli (kukkalli)]

How do I verify that if this is from neutron or some other network responding?
I tried enabling the DHCP in openstack provider subnet as well as disabling the openstack DHCP.
The issue exists.

I can ping the VM from the Compute Host, but not from any other machine in the same subnet.
In VM:
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:95:0c:34 brd ff:ff:ff:ff:ff:ff
    inet 10.20.5.25/22 brd 10.20.7.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe95:c34/64 scope link
       valid_lft forever preferred_lft forever

In Compute host:
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master brq4e8bbde3-1d state UP group default qlen 1000
    link/ether 34:40:b5:9d:65:8c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::3640:b5ff:fe9d:658c/64 scope link
       valid_lft forever preferred_lft forever
14: brq4e8bbde3-1d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 34:40:b5:9d:65:8c brd ff:ff:ff:ff:ff:ff
    inet 10.20.4.32/22 brd 10.20.7.255 scope global brq4e8bbde3-1d
       valid_lft forever preferred_lft forever
15: tap9769fce6-fb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master brq4e8bbde3-1d state UNKNOWN group default qlen 1000
    link/ether fe:16:3e:95:0c:34 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe95:c34/64 scope link
       valid_lft forever preferred_lft forever

These are the results of ip a in the VM and compute host respectively.

[Brian Haley (brian-haley)]

The DHCP client will store the lease file locally, which will have the IP of the server that gave it the IP. I would try using ps to find the client, it usually has a lease file as an argument when it is started, or possibly the directory where it will write files.

It's a line like this:

option dhcp-server-identifier 10.0.0.2;

If that IP isn't the neutron DHCP server then you could have this problem.

[Hanif Kukkalli (kukkalli)]

Hi Brian,

Would you kindly elaborate the neutron DHCP server?
OpenStack created a DHCP port at 10.20.5.1.
There is an external DHCP server at 10.20.4.1.

In the servers, the dhcp-server-identifier is:
option dhcp-server-identifier 10.20.4.1;

In the VM, I see the same file as empty.

Shall I disable the DHCP in OpenStack network?

[Brian Haley (brian-haley)]

The VM might be running a different dhcp client, so might be storing the information in a different location.

It looks like the external DHCP server is allocating addresses from a different subnet, so that looks Ok, and you need to have the neutron DHCP server running to service the VMs, I think you just need to find out why the VMs are getting a different IP from what neutron has allocated for them. Either finding the lease file or watching with tcpdump would help, once we have that we can figure out what needs to be fixed.

[Hanif Kukkalli (kukkalli)]

I checked the dhclient.ens3.lease file in the VM, it shows the dhcp-server-identifier as: 10.20.4.1, which is the same external DHCP.
Shouldn't this be the IP of the OpenStack DHCP?

I can perform ICMP ping and SSH to the VM only from the Compute Host it is hosted on and not from the controller.
I can ping from the VM to the IP of the compute-host it is hosted on and not to the other compute or controller machines.

The IP in the VM is 10.20.5.17, the IP displayed in the dashboard is 10.20.5.6.

This is the lease file from the VM:
lease {
  interface "ens3";
  fixed-address 10.20.5.17;
  option subnet-mask 255.255.252.0;
  option routers 10.20.4.1;
  option dhcp-lease-time 3600;
  option dhcp-message-type 5;
  option domain-name-servers 10.20.4.1;
  option dhcp-server-identifier 10.20.4.1;
  option dhcp-renewal-time 1800;
  option broadcast-address 10.20.7.255;
  option dhcp-rebinding-time 3150;
  option host-name "test";
  renew 4 2017/12/21 15:43:31;
  rebind 4 2017/12/21 15:43:31;
  expire 4 2017/12/21 15:43:31;
}

The machine name is also wrong in the VM. Instead of displaying the given name during creation of VM, it shows as test.

I checked the parameter "option host-name" in an instance of other OpenStack Pike setup. It looked like:
  option host-name "host-10-20-5-17";
  option domain-name "openstacklocal";

What could be wrong with the OpenStack?

One more thing I would like to add, I initially did not set DHCP in the neutron during creation of the subnet.
After creation of subnet and adding VM's, I enabled the DHCP checkbox. The existing VM's were working as usual.
The newly created VM's started showing this issue.

I deleted all the VM's, deleted the subnet and network, deleted the network agent list, deleted the compute service list and added them one by one.
I still find the same issue.

[Hanif Kukkalli (kukkalli)]

There is two more interesting thing in the VM.
1. As I am using Ubuntu 16.04 Cloud image, the interface are renamed from eth0 to ens3.
2. in the DHCP lease files, I see 2 lease files for interfaces ens3 and ens4, whereas there is only one interface.

The lease files are as below

lease file for ens4:
lease {
  interface "ens3";
  fixed-address 10.20.5.17;
  option subnet-mask 255.255.252.0;
  option dhcp-lease-time 3600;
  option routers 10.20.4.1;
  option dhcp-message-type 5;
  option dhcp-server-identifier 10.20.4.1;
  option domain-name-servers 10.20.4.1;
  option dhcp-renewal-time 1800;
  option dhcp-rebinding-time 3150;
  option broadcast-address 10.20.7.255;
  option host-name "test";
  renew 4 2017/10/26 15:50:43;
  rebind 4 2017/10/26 16:14:29;
  expire 4 2017/10/26 16:21:59;
}
lease {
  interface "ens3";
  fixed-address 10.20.6.2;
  option subnet-mask 255.255.252.0;
  option routers 10.20.4.1;
  option dhcp-lease-time 3600;
  option dhcp-message-type 5;
  option domain-name-servers 10.20.4.1;
  option dhcp-server-identifier 10.20.4.1;
  option dhcp-renewal-time 1800;
  option broadcast-address 10.20.7.255;
  option dhcp-rebinding-time 3150;
  option host-name "test";
  renew 4 2017/12/21 18:55:48;
  rebind 4 2017/12/21 19:23:26;
  expire 4 2017/12/21 19:30:56;
}

lease file for ens4:
lease {
  interface "ens4";
  fixed-address 10.20.5.20;
  option subnet-mask 255.255.252.0;
  option dhcp-lease-time 86400;
  option routers 10.20.4.1;
  option dhcp-message-type 5;
  option dhcp-server-identifier 10.20.5.10;
  option domain-name-servers 8.8.8.8,8.8.4.4;
  option interface-mtu 1500;
  option dhcp-renewal-time 43200;
  option dhcp-rebinding-time 75600;
  option broadcast-address 10.20.7.255;
  option rfc3442-classless-static-routes 32,169,254,169,254,10,20,5,10,0,10,20,4,1;
  option host-name "host-10-20-5-20";
  option domain-name "openstacklocal";
  renew 5 2017/10/27 01:55:41;
  rebind 5 2017/10/27 12:21:59;
  expire 5 2017/10/27 15:21:59;
}

[Brian Haley (brian-haley)]

You mention in comment #9 that you looked on another Pike installation and things seem Ok? If so I would recommend you look at the differences between the two, which could point at the issue.

[Hanif Kukkalli (kukkalli)]

Dear Brian,
I wish you a very happy and prosperous new year 2018!

Thank you for your kind reply.
I have been looking into the differences.
The OpenStack setup which is working fine has DHCP only from the neutron and no external DHCP.

The OpenStack allows not to enable DHCP from neutron as a check box in dashboard.
Initially this worked fine, until I enabled DHCP in neutron.
I have been trying to resolve this issue, and it hasn't worked.

I have deleted the network, subnet and all the compute and neutron services from OpenStack.
Restarted them adding one by one, but the issue still persists.

Could this be that something might have been stored in the MariaDB database which might be creating this descrepency during VM creation?

The names of the launched VM's are not the intended onces.
For example: if I create the VM with the name 'vm-controller', instead of this name, it gets a name 'test'.

I will be testing further by switching off the external DHCP server.

Kindly let me know how can I debug this issue in the neutron python code files.
I can place some debug statements in the code and check for the log files for expected output.
Any hints would be appreciated.

[Hanif Kukkalli (kukkalli)]

Hi Brian,

Turning off external DHCP resolved the issue.
It works perfectly fine.

Isn't it still a bug when we use an external DHCP server for OpenStack and play around with it and it fails?

Kind Regards,
Hanif Kukkalli

[Brian Haley (brian-haley)]

Having two DHCP servers on the same network can be problematic, especially in this case where they both are assigning IPs from the same overlapping range. Since the neutron-started DHCP agent will only assign IPs to instances it knows about, if you are going to use a second external DHCP it will need to be configured to only assign IPs to things specifically, say by MAC address, and it should be configured in such a way that it is not in an over-lapping IP range with neutron. I guess I don't see that as a bug but just a configuration issue.

[Hanif Kukkalli (kukkalli)]

Thanks Brian!

So, in this case, I should always use Neutron-DHCP and cannot use an external DHCP to do this in OpenStack.

One observation as I have already mentioned before, the external DHCP was working before I enabled the Neutron DHCP configuration.
I feel that the check-box to enable or disable the Neutron-DHCP is availbale for a purpose to either choose Neutron-DHCP or External-DHCP.
After enabling the Neutron-DHCP, it started to fail.
Even when I disabled the Neutron-DHCP from the check box later, it still failed.

If OpenStack is supposed to use DHCP by Neutron, what is the purpose of the DHCP checkbox during configration of subnet?

If disabling checkbox is to use external DHCP, then it is a bug.

I maybe wrong in this. Kindly clarify my thoughts as stated above.

[Hanif Kukkalli (kukkalli)]

Hello Team,

I would like to know the significance of DHCP checkbox while creating subnet.
Could anyone please provide me information on this?

Kind regards,
Hanif

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]