Comment 6 for bug 2006467

I am not aware about any issues with udhcpc, the change(to switch to dhcpcd) in neutron/tempest were required for move to cirros-0.6.1(but i don't think that move was to fix any udhcpc issue in cirros-0.5.2).

It can be updated in nova too but once we have confirm it's an issue with cirros/udhcpc.

As per [1] it's seen 7 times in last 14 day in master, stable/zed for jobs:-
nova-next 28.6%
neutron-ovs-tempest-dvr 28.6%
devstack-platform-ubuntu-jammy-ovs 28.6%
neutron-ovs-tempest-fips 14.3%

For the same example from comment #1,
network_id=9680b161-ad56-444b-aed2-053bed433701, port_id=77cc62f3-8d3f-4a4f-bbef-ac68fdcd9c9e, in dhcp agent log:-

Feb 14 18:18:50.991856 np0033093378 neutron-dhcp-agent[76814]: INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[], binding:host_id=, binding:profile=, binding:vif_details=, binding:vif_type=unbound, binding:vnic_type=normal, created_at=2023-02-14T18:18:50Z, description=, device_id=6a265379-ebfd-4aea-a081-8b271f32c0ea, device_owner=, extra_dhcp_opts=[], fixed_ips=[<neutron.agent.linux.dhcp.DictModel object at 0x7f5718c1f970>], id=77cc62f3-8d3f-4a4f-bbef-ac68fdcd9c9e, mac_address=fa:16:3e:57:88:0b, name=, network=admin_state_up=True, availability_zone_hints=[], availability_zones=['nova'], created_at=2023-02-14T18:18:26Z, description=, id=9680b161-ad56-444b-aed2-053bed433701, ipv4_address_scope=None, ipv6_address_scope=None, mtu=1380, name=tempest-AttachVolumeNegativeTest-1639666997-network, port_security_enabled=True, project_id=30309aa994164cd1a33985bb09feee0b, provider:network_type=vxlan, provider:physical_network=None, provider:segmentation_id=428, qos_policy_id=None, revision_number=2, router:external=False, shared=False, standard_attr_id=1540, status=ACTIVE, subnets=['2a932882-9ae4-4727-8396-40db023a31d1'], tags=[], tenant_id=30309aa994164cd1a33985bb09feee0b, updated_at=2023-02-14T18:18:26Z, vlan_transparent=None, network_id=9680b161-ad56-444b-aed2-053bed433701, port_security_enabled=True, project_id=30309aa994164cd1a33985bb09feee0b, qos_network_policy_id=None, qos_policy_id=None, resource_request=None, revision_number=1, security_groups=['0571006c-8dcf-45d6-8856-87ee19949ca3'], standard_attr_id=1555, status=DOWN, tags=[], tenant_id=30309aa994164cd1a33985bb09feee0b, updated_at=2023-02-14T18:18:50Z on network _ns_name=qdhcp-9680b161-ad56-444b-aed2-053bed433701, admin_state_up=True, id=9680b161-ad56-444b-aed2-053bed433701, mtu=1380, non_local_subnets=[], ports=[<neutron.agent.linux.dhcp.DictModel object at 0x7f5718be4310>, <neutron.agent.linux.dhcp.DictModel object at 0x7f5718be6650>], project_id=30309aa994164cd1a33985bb09feee0b, subnets=[<neutron.agent.linux.dhcp.DictModel object at 0x7f5718d42080>], tenant_id=30309aa994164cd1a33985bb09feee0b
Feb 14 18:18:50.993190 np0033093378 neutron-dhcp-agent[76814]: DEBUG neutron.agent.dhcp.agent [-] Calling driver for network: 9680b161-ad56-444b-aed2-053bed433701/seg=None action: reload_allocations {{(pid=76814) _call_driver /opt/stack/neutron/neutron/agent/dhcp/agent.py:242}}
Feb 14 18:18:50.996591 np0033093378 neutron-dhcp-agent[76814]: DEBUG neutron.agent.linux.dhcp [-] Building host file: /opt/stack/data/neutron/dhcp/9680b161-ad56-444b-aed2-053bed433701/host {{(pid=76814) _output_hosts_file /opt/stack/neutron/neutron/agent/linux/dhcp.py:949}}
Feb 14 18:18:50.998687 np0033093378 neutron-dhcp-agent[76814]: DEBUG neutron.agent.linux.dhcp [-] Done building host file /opt/stack/data/neutron/dhcp/9680b161-ad56-444b-aed2-053bed433701/host {{(pid=76814) _output_hosts_file /opt/stack/neutron/neutron/agent/linux/dhcp.py:990}}
Feb 14 18:18:51.013538 np0033093378 dnsmasq[129395]: read /opt/stack/data/neutron/dhcp/9680b161-ad56-444b-aed2-053bed433701/addn_hosts - 2 addresses
Feb 14 18:18:51.016844 np0033093378 neutron-dhcp-agent[76814]: DEBUG oslo.privsep.daemon [-] privsep: reply[0c40309a-a2df-43e7-a1db-76f0f9a895d8]: (4, ('', '', 0)) {{(pid=78395) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}}
Feb 14 18:18:51.016844 np0033093378 neutron-dhcp-agent[76814]: DEBUG neutron.agent.linux.dhcp [-] Reloading allocations for network: 9680b161-ad56-444b-aed2-053bed433701 {{(pid=76814) reload_allocations /opt/stack/neutron/neutron/agent/linux/dhcp.py:664}}
Feb 14 18:18:51.013684 np0033093378 dnsmasq-dhcp[129395]: read /opt/stack/data/neutron/dhcp/9680b161-ad56-444b-aed2-053bed433701/host
Feb 14 18:18:51.013748 np0033093378 dnsmasq-dhcp[129395]: read /opt/stack/data/neutron/dhcp/9680b161-ad56-444b-aed2-053bed433701/opts
Feb 14 18:18:51.421237 np0033093378 neutron-dhcp-agent[76814]: INFO neutron.agent.dhcp.agent [None req-e1c755fe-019d-4228-a63f-760a03a66e53 None None] DHCP configuration for ports {'77cc62f3-8d3f-4a4f-bbef-ac68fdcd9c9e'} is completed

DHCPDISCOVER/OFFER request received thrice matching cirros attempts and it's offering the IP:-
Feb 14 18:19:31.315663 np0033093378 dnsmasq-dhcp[129395]: DHCPDISCOVER(tap0532a4d4-89) fa:16:3e:57:88:0b
Feb 14 18:19:31.315704 np0033093378 dnsmasq-dhcp[129395]: DHCPOFFER(tap0532a4d4-89) 10.1.0.9 fa:16:3e:57:88:0b

Feb 14 18:20:31.384469 np0033093378 dnsmasq-dhcp[129395]: DHCPDISCOVER(tap0532a4d4-89) fa:16:3e:57:88:0b
Feb 14 18:20:31.384510 np0033093378 dnsmasq-dhcp[129395]: DHCPOFFER(tap0532a4d4-89) 10.1.0.9 fa:16:3e:57:88:0b

Feb 14 18:21:31.388987 np0033093378 dnsmasq-dhcp[129395]: DHCPDISCOVER(tap0532a4d4-89) fa:16:3e:57:88:0b
Feb 14 18:21:31.389081 np0033093378 dnsmasq-dhcp[129395]: DHCPOFFER(tap0532a4d4-89) 10.1.0.9 fa:16:3e:57:88:0b

But no DHCPREQUEST and DHCPACK messages, don't know why. May be @ralonsoh has some idea on this.

[1] https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_status,negate:!f,params:(query:FAILURE),type:phrase),query:(match_phrase:(build_status:FAILURE))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:filename,negate:!f,params:(query:job-output.txt),type:phrase),query:(match_phrase:(filename:job-output.txt)))),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22udhcpc:%20no%20lease,%20failing%22'),sort:!())