IP not getting assigned to vSRX(ffp) interface via DHCP in a scaling scenario

[Vijay Anand]

IP is not getting assigned to vSRX (firefly) interface via dhcp in a scaling scenario.

Test:
====
1. 5 project or tenants, 3 networks with /24 subnet per tenant.
2. Launch 50 parallel instances (10 per tenant)

Observation:
==========
1. vSRX interface (Eg:ge-0/0/0) dhcp gets IP upto 38 instances.
2. IP is not getting assigned to last 12 of 50 instances.

This problem is consistently reproducible i.e., for 50 instances launch, 38 gets IP and last 12 don't.

Logs are stored in the local server, please reach out to me for log details.

System status:
============
root@nfv-server04:~# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr initializing (NTP state unsynchronized.)
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen active
contrail-analytics-api active
contrail-analytics-nodemgr initializing (NTP state unsynchronized.)
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr initializing (NTP state unsynchronized.)
contrail-device-manager active
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor active
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
supervisor-database: active
contrail-database active
contrail-database-nodemgr initializing (NTP state unsynchronized.)
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

root@nfv-server04:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 2.20-64 64
contrail-config 2.20-64 64
contrail-config-openstack 2.20-64 64
contrail-control 2.20-64 64
contrail-dns 2.20-64 64
contrail-f5 2.20-64 64
contrail-fabric-utils 2.20-64 64
contrail-heat 2.20-64 64
contrail-install-packages 2.20-64~icehouse 64
contrail-lib 2.20-64 64
contrail-nodemgr 2.20-64 64
contrail-nova-networkapi 2.20-64 64
contrail-openstack 2.20-64 64
contrail-openstack-analytics 2.20-64 64
contrail-openstack-config 2.20-64 64
contrail-openstack-control 2.20-64 64
contrail-openstack-dashboard 2.20-64 64
contrail-openstack-database 2.20-64 64
contrail-openstack-webui 2.20-64 64
contrail-setup 2.20-64 64
contrail-utils 2.20-64 64
contrail-web-controller 2.20-64 64
contrail-web-core 2.20-64 64
ifmap-python-client 0.1-2 64
ifmap-server 0.3.2-1contrail1 64
neutron-plugin-contrail 2.20-64 64
nova-api 1:2014.1.3-0ubuntu1~cloud0.3contrail64
nova-common 1:2014.1.3-0ubuntu1~cloud0.3contrail64
nova-conductor 1:2014.1.3-0ubuntu1~cloud0.3contrail64
nova-console 1:2014.1.3-0ubuntu1~cloud0.3contrail64
nova-consoleauth 1:2014.1.3-0ubuntu1~cloud0.3contrail64
nova-novncproxy 1:2014.1.3-0ubuntu1~cloud0.3contrail64
nova-objectstore 1:2014.1.3-0ubuntu1~cloud0.3contrail64
nova-scheduler 1:2014.1.3-0ubuntu1~cloud0.3contrail64
python-contrail 2.20-64 64
python-neutronclient 2:2.3.4-0ubuntu1.2contrail 64
python-nova 1:2014.1.3-0ubuntu1~cloud0.3contrail64
root@nfv-server04:~# openstack-statu
openstack-statu: command not found
root@nfv-server04:~# openstack-status
== Nova services ==
openstack-nova-api: active
openstack-nova-compute: inactive (disabled on boot)
openstack-nova-network: inactive (disabled on boot)
openstack-nova-scheduler: active
openstack-nova-volume: inactive (disabled on boot)
openstack-nova-conductor: active
== Glance services ==
openstack-glance-api: active
openstack-glance-registry: active
== Keystone service ==
openstack-keystone: active
== Cinder services ==
openstack-cinder-api: active
openstack-cinder-scheduler: active
openstack-cinder-volume: inactive (disabled on boot)
== Support services ==
mysql: inactive (disabled on boot)
rabbitmq-server: active
memcached: inactive (disabled on boot)
== Keystone users ==
Warning keystonerc not sourced
root@nfv-server04:~#

[Deepak Jeyaraman (jdeepak)]

Hari,

Please find attached the pkt capture of pkt0 interface.

In this test. of the 75 instances, only 20 (first 20) have the dhcp ips

snippet:
root@nfv-server04:~# nova list | grep ACTIVE | wc -l
75
root@nfv-server04:~# nova list | grep ACTIVE | more
| 19a758f9-61ba-4796-9e84-ad98997ce41c | default-domain__admin__0b27baf3-5c90-47cf-a90a-6d1613a7503d-lxciptable-0-vnf-instance__1 | ACTIVE | - | Run
ning | 0b27baf3-5c90-47cf-a90a-6d1613a7503d-left-net=11.0.23.3; mgmt-net=10.102.56.154; right-net=192.168.4.28 |
| 0c59e45e-ba3f-4b77-a628-8a3e38f2eff0 | default-domain__admin__0dcb291e-dd1a-4b0f-9055-adb9cd3d9c0e-lxciptable-0-vnf-instance__1 | ACTIVE | - | Run
ning | right-net=192.168.4.20; mgmt-net=10.102.56.147; 0dcb291e-dd1a-4b0f-9055-adb9cd3d9c0e-left-net=11.0.16.3 |
| e0607514-2679-445a-bb25-c5f07796294a | default-domain__admin__0ea0ca94-1427-4519-9fee-0c83d66534a3-lxciptable-0-vnf-instance__1 | ACTIVE | - | Run
ning | right-net=192.168.4.79; 0ea0ca94-1427-4519-9fee-0c83d66534a3-left-net=11.0.74.3; mgmt-net=10.102.56.205 |
| 8a23314c-c06d-4ebf-a067-2b67e2fbd75b | default-domain__admin__106d1918-db62-48b6-8f53-c9b466e4f87d-lxciptable-0-vnf-instance__1 | ACTIVE | - | Run
ning | right-net=192.168.4.77; mgmt-net=10.102.56.203; 106d1918-db62-48b6-8f53-c9b466e4f87d-left-net=11.0.72.3 |
| db8378b3-1ecf-432f-b16d-53166dde8880 | default-domain__admin__1209fce2-36c3-434c-8e60-a9864f3e934a-lxciptable-0-vnf-instance__1 | ACTIVE | - | Run
ning | right-net=192.168.4.44; mgmt-net=10.102.56.170; 1209fce2-36c3-434c-8e60-a9864f3e934a-left-net=11.0.39.3 |

<snip>

Thanks
Deepak

[Hari Prasad Killi (haripk)]

The packet capture shows DHCP packets from 20 instances along with the responses for those. There are no requests from the other instances. Need to check if the Vms are sending DHCP requests or if they are getting dropped in the vrouter module.

[Hari Prasad Killi (haripk)]

In R2.20, the DHCP flag on interface was moved to the route - vrouter traps DHCP packets to agent based on this route only. When configuration download to agent is delayed, this route is not created and the DHCP requests from the VM are ignored. The VM retries 5 times and gives up, due to which it doesnt get a DHCP lease.

This is fixed in R2.21 - vrouter traps DHCP packets based on a flag on the interface as well. When configuration is not downloaded to agent, a short lease is given to the VM so that it retries upon the short lease expiry, till it gets a final lease.

Verified that issue is not present with R2.21.