On master branch Dell Ironic CI jobs started failing on 9/2 around 8am CDT (-0500 UTC). We've noticed a failure to PXE boot into iPXE and a haproxy error in q-dhcp log:
DEBUG neutron.agent.linux.utils [None req-f5a2841b-950e-4b2c-bef1-d4113ab0e12d None None] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qdhcp-966d8976-a674-4c33-991b-5618d0881af4', 'haproxy', '-f', '/opt/stack/data/neutron/ns-metadata-proxy/966d8976-a674-4c33-991b-5618d0881af4.conf'] {{(pid=21074) execute_rootwrap_daemon /opt/stack/new/neutron/neutron/agent/linux/utils.py:103}}
Sep 08 20:04:30.193354 d-p-c-local-01-108424 haproxy-metadata-proxy-966d8976-a674-4c33-991b-5618d0881af4[22603]: Proxy listener started.
Sep 08 20:04:30.213883 d-p-c-local-01-108424 neutron-dhcp-agent[21074]: ERROR neutron.agent.linux.utils [None req-f5a2841b-950e-4b2c-bef1-d4113ab0e12d None None] Exit code: 1; Cmd: ['ip', 'netns', 'exec', 'qdhcp-966d8976-a674-4c33-991b-5618d0881af4', 'haproxy', '-f', '/opt/stack/data/neutron/ns-metadata-proxy/966d8976-a674-4c33-991b-5618d0881af4.conf']; Stdin: ; Stdout: ; Stderr: [ALERT] 251/200430 (22603) : [haproxy.main()] Some configuration options require full privileges, so global.uid cannot be changed.
After commenting out user and group in neutron/agent/metadata/driver.py and neutron/agent/ovn/metadata/driver.py, the error goes away and the jobs are successful.
Hi Eric,
Do You know if something else changed recently in Your job and what could trigger such issue? Like e.g. change of the os version on which it is run, or change of haproxy version used there?