Trusty AIO fails to start DHCP on test network

[Logan V]

On my local gating for stable/newton, I am seeing tempest failures on Trusty due to a dhcp agent startup failure.

2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for bdf786b1-2282-47d1-bc2c-e92171e217d8.
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-14.0.0/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 114, in call_driver
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs)
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-14.0.0/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 214, in enable
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent self.spawn_process()
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-14.0.0/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 428, in spawn_process
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent self._spawn_or_reload_process(reload_with_HUP=False)
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-14.0.0/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 442, in _spawn_or_reload_process
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent pm.enable(reload_cfg=reload_with_HUP)
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-14.0.0/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 94, in enable
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent run_as_root=self.run_as_root)
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-14.0.0/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 885, in execute
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent log_fail_as_error=log_fail_as_error, **kwargs)
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-14.0.0/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 138, in execute
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent raise RuntimeError(msg)
2016-10-09 03:28:36.315 7292 ERROR neutron.agent.dhcp.agent RuntimeError: Exit code: 127; Stdin: ; Stdout: ; Stderr: dnsmasq: error while loading shared libraries: libdbus-1.so.3: cannot stat shared object: Permission denied

I am seeing this consistently on all of my stable/newton trusty builds. Xenial builds against the same branch pass consistently.

[Logan V (loganv)]

Also I locked one of the trusty CI builds so we can dig into this broken env further to gather more debugging. Ping me if needed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/385687

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kevin Carter (cloudnull) (<email address hidden>) on branch: master
Review: https://review.openstack.org/385687
Reason: this is no longer needed because we've determined that overlayfs use in trusty is simply not stable.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351776
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=0a5a11704fb6f85fb1bc151a54c6b4e9ea40a2be
Submitter: Jenkins
Branch: master

commit 0a5a11704fb6f85fb1bc151a54c6b4e9ea40a2be
Author: Jesse Pretorius <email address hidden>
Date: Fri Aug 5 15:42:58 2016 +0100

    Prevent overlayfs use in test when kernel < 3.18 or release == trusty

    The overlayfs version in kernel version < 3.18 was not production-ready
    and should be avoided on Trusty due general instability. This patch
    prevents overlayfs from being used when implementing an AIO with
    kernel version and release versions that are not not suitable.

    Related-Bug: #1612412
    Related-Bug: #1631690
    Change-Id: I224c27ed645c3f3817721baccd5d9e5ce19f3a03

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/386128

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/386128
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=6bbb3d5e1ab18f0cd375847a7087f99896651377
Submitter: Jenkins
Branch: stable/newton

commit 6bbb3d5e1ab18f0cd375847a7087f99896651377
Author: Jesse Pretorius <email address hidden>
Date: Fri Aug 5 15:42:58 2016 +0100

    Prevent overlayfs use in test when kernel < 3.18 or release == trusty

    The overlayfs version in kernel version < 3.18 was not production-ready
    and should be avoided on Trusty due general instability. This patch
    prevents overlayfs from being used when implementing an AIO with
    kernel version and release versions that are not not suitable.

    Related-Bug: #1612412
    Related-Bug: #1631690
    Change-Id: I224c27ed645c3f3817721baccd5d9e5ce19f3a03
    (cherry picked from commit 0a5a11704fb6f85fb1bc151a54c6b4e9ea40a2be)

[Kevin Carter (kevin-carter)]

This issue was closed because we no longer support Trusty due to Newton becoming EOL.