Quickstart installed undercloud async tasks broken

[Steven Hardy]

I installed an undercloud via quickstart, and I'm seeing this in /var/log/messages:

Feb 14 14:54:09 undercloud registry: 192.168.24.1 - - [14/Feb/2018:14:54:09 +0000] "OPTIONS / HTTP/1.0" 200 0 "" ""
Feb 14 14:54:10 undercloud ansible-async_wrapper.py: 2616 still running (5445)
Feb 14 14:54:11 undercloud registry: 192.168.24.1 - - [14/Feb/2018:14:54:11 +0000] "OPTIONS / HTTP/1.0" 200 0 "" ""
Feb 14 14:54:11 undercloud sshd[26240]: Failed password for root from 192.168.23.1 port 48924 ssh2
Feb 14 14:54:11 undercloud sshd[26240]: Failed password for root from 192.168.23.1 port 48924 ssh2
Feb 14 14:54:11 undercloud sshd[26240]: Connection closed by 192.168.23.1 port 48924 [preauth]

It appears that we're running some async task (dstat?) too frequently so the previous task is still running and/or it's trying and failing to use an ssh connection vs locally running the task?

I see this in ps ax output:

[root@undercloud ~]# ps ax | grep ansible
 2615 ? S 0:00 /usr/bin/python /home/stack/.ansible/tmp/ansible-tmp-1518618286.83-217113905514223/async_wrapper.py 737612437336 7200 /home/stack/.ansible/tmp/ansible-tmp-1518618286.83-217113905514223/command.py _
 2616 ? S 0:00 /usr/bin/python /home/stack/.ansible/tmp/ansible-tmp-1518618286.83-217113905514223/async_wrapper.py 737612437336 7200 /home/stack/.ansible/tmp/ansible-tmp-1518618286.83-217113905514223/command.py _
 2617 ? S 0:00 /usr/bin/python /home/stack/.ansible/tmp/ansible-tmp-1518618286.83-217113905514223/command.py
 2618 ? S 0:00 /usr/bin/python /tmp/ansible_d3zb4q/ansible_module_command.py
26526 pts/0 S+ 0:00 grep --color=auto ansible

/var/log/extra/dstat-csv.log is getting populated, so perhaps there's some other async magic happening - personally I'd like an easy way to disable all this CI specific stuff for local testing.

[Steven Hardy (shardy)]

[shardy@tripleodev2 openshift_openstack]$ ssh -F /home/shardy/.quickstart-shiftstack/ssh.config.ansible undercloud
Warning: Permanently added 'tripleodev2.localdomain,192.168.1.91' (ECDSA) to the list of known hosts.
Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
Last login: Wed Feb 14 14:57:38 2018 from gateway
[stack@undercloud ~]$ sudo su -
Last login: Wed Feb 14 14:42:32 UTC 2018 on pts/0
Last failed login: Wed Feb 14 14:57:51 UTC 2018 from gateway on ssh:notty
There were 352 failed login attempts since the last successful login.
[root@undercloud ~]#

So we see constantly increasing failed logins which align with the warnings in the log