After deployments on scenario003, we attempt to install dstat on the overcloud. Unfortunately it appears to be failing fairly regularly because it can't download the rpm.
http://logs.openstack.org/20/603220/1/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/43093ac/job-output.txt#_2018-09-28_03_02_02_287193
2018-09-28 03:02:02.287193 | primary | PLAY [setup dstat performance monitoring] **************************************
2018-09-28 03:02:02.326067 | primary |
2018-09-28 03:02:02.326291 | primary | TASK [validate-perf : Install the latest version of dstat on overcloud] ********
2018-09-28 03:02:02.377924 | primary | Friday 28 September 2018 03:02:02 +0000 (0:00:00.331) 0:00:00.331 ******
2018-09-28 03:02:08.786581 | primary | fatal: [subnode-2]: FAILED! => {"changed": false, "msg": "http://mirror.gra1.ovh.openstack.org/centos/7/os/x86_64/Packages/dstat-0.7.2-12.el7.noarch.rpm: [Errno 14] curl#6 - \"Could not resolve host: mirror.gra1.ovh.openstack.org; Unknown error\"\nTrying other mirror.\n\n\nError downloading packages:\n dstat-0.7.2-12.el7.noarch: [Errno 256] No more mirrors to try.\n\n", "rc": 1, "results": ["Loaded plugins: fastestmirror, priorities\nLoading mirror speeds from cached hostfile\n130 packages excluded due to repository priority protections\nResolving Dependencies\n--> Running transaction check\n---> Package dstat.noarch 0:0.7.2-12.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n dstat noarch 0.7.2-12.el7 quickstart-centos-base 163 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 163 k\nInstalled size: 752 k\nDownloading packages:\n"]}
2018-09-28 03:02:08.786773 | primary | ...ignoring
I believe this is because named which is for designate may be causing issues.
http://logs.openstack.org/20/603220/1/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/43093ac/logs/subnode-2/var/log/extra/netstat.txt.gz
udp6 0 0 :::53 :::* 82170/named
udp6 0 0 ::1:53 :::* 1755/unbound
udp6 0 0 ::1:53 :::* 1755/unbound
udp6 0 0 ::1:53 :::* 1755/unbound
udp6 0 0 ::1:53 :::* 1755/unbound
I'm a little surprised this is even allowed, but I think we can shut off BIND listening on localhost. The issue is I think it will break Designate functionality in ci because I used localhost as the predictable ip. If scenario003 isn't already using predictable ips we'll need to enable that so Designate knows where to look for BIND. This is good anyway since that's how Designate would be configured in real deployments, but it does complicate the fix for this problem.