gate: apt install for containers fail on hpcloud-b4

[Jesse Pretorius]

In hpcloud-b4 gate check builds fail when doing apt installations in the first container built. This happens every single time.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/172096
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=954f8ff8cb4a57fbf68781cf51bce37a9a3a10e3
Submitter: Jenkins
Branch: master

commit 954f8ff8cb4a57fbf68781cf51bce37a9a3a10e3
Author: Jesse Pretorius <email address hidden>
Date: Thu Apr 9 15:46:30 2015 +0100

    Add AIO diagnostics and better gate scripts consistency

    Implements extra diagnostic information for gate check failure analysis and
    also appends DNS settings instead of replacing them for the AIO bootstrap.

    Some extra settings are added to the DNS to try to catch edge cases for DNS
    resolution. The DNS `resolv.conf` was being overwritten on every run using
    google name servers however we need to add google name servers to the existing
    resolver not overwrite the whole conf so a conditional was added.

    The gate scripts were attempting to create a symlink to logs on every run.
    While this is fine for the gate which is creates a new node on every run
    the link will cause failures on recheck when doing more local testing.

    The ansible log entry was being appended to the `ansible.cfg` file on every
    run which should also be limited to one entry. The conditional added is
    helpful when doing local testing with the gate scripts.

    Closes-Bug: #1442630
    Change-Id: I25aac98d7408ba32f94befea22da4471bd1697b6

[Jesse Pretorius (jesse-pretorius)]

The apt-get installs always fail, but only for containers:

ansible neutron_agent -m shell -a 'DEBIAN_FRONTEND=noninteractive apt-get install iptables-persistent' -t /openstack/log/ansible_cmd_logs//add_neutron_agent_checksum_rule

There is evidence of DNS resolution failure in the logs:

Err http://us.archive.ubuntu.com trusty-updates Release.gpg
  Could not resolve 'us.archive.ubuntu.com'

The host is able to apt-get update/install just fine.

[Jesse Pretorius (jesse-pretorius)]

I'm wondering if this has something to do with our iptables flush in bootstrap-aio.

[David Wilde (dave-wilde)]

It very well could, I've never liked that rule.

[Jesse Pretorius (jesse-pretorius)]

master review: https://review.openstack.org/215040
kilo review: https://review.openstack.org/227765