neutron

  1. Bug #1828423
  2. Comment #4

Comment 4 for bug 1828423

Revision history for this message
Fred S (fsbiz) wrote :

This fix helps and removes the semaphore that was present in the prior code but it is not fixing the root cause of the problem unfortunately.

Multiple port update messages are serialized to a Priority Queue. The messages are then drained serially without the code yielding. The end result still looks similar (somewhat better than without the fix).

The below is an output from my /var/log/dnsmasq.log when 8 baremetal nodes are provisioned simultaneously.
As can be seen dnsmasq could not process DHCP packets from 23:36:00 to 23:37:33 (over 93 seconds). This leads to the PXE boots on some of the baremetals timing out.

Oct 7 23:36:00 stg-sc-control01 dnsmasq-dhcp[2328509]: DHCPACK(ns-7f4ce6ff-2a) 10.33.27.101 6c:b3:11:4f:87:40 host-10-33-27-101
Oct 7 23:36:26 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:26 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:29 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:29 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:32 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:32 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:34 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:34 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:36 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:36 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:38 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:38 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:40 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:40 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:42 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:42 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:44 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:44 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:46 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:46 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:48 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:48 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:50 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:50 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:52 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:52 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:54 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:54 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:56 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:56 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:36:58 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:36:58 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:00 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:00 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:01 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:01 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:03 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:03 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:05 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:05 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:07 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:07 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:09 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:09 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:11 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:11 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:13 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:13 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:15 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:15 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:17 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:17 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:18 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:18 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:20 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:20 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:22 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:22 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:24 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:24 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:26 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:28 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:28 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:30 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:30 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:32 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/host
Oct 7 23:37:32 stg-sc-control01 dnsmasq-dhcp[2328509]: read /var/lib/neutron/dhcp/077aa2d1-605c-48ec-842d-7dd6767bfd01/opts
Oct 7 23:37:33 stg-sc-control01 dnsmasq-dhcp[2328509]: DHCPRELEASE(ns-7f4ce6ff-2a) 10.33.27.101 6c:b3:11:4f:87:40
Oct 7 23:37:33 stg-sc-control01 dnsmasq-dhcp[2328509]: DHCPREQUEST(ns-7f4ce6ff-2a) 10.33.27.97 9c:71:3a:f4:a1:3b