periodic HA master job pingtest times out

[Gabriele Cerami]

periodic HA job fails during ping test with the error

2016-10-31 07:57:58.084956 | Timing out after 300 seconds:

as seen in

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/0ae3179/console.html#_2016-10-31_07_57_58_084956

Logs at

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/0ae3179/logs/overcloud-controller-0/var/log/heat/heat-engine.txt.gz

show that tenant stack is created successfully 1 minute after the timeout.

Investigating delay, eventually increasing timeout.

[Gabriele Cerami (gcerami)]

Looks like gnocchi-metricd is eating all the CPU again.

logs are full of this error

2016-10-31 07:45:29.205 28623 ERROR cotyledon ToozConnectionError: Error while reading from socket: ('Connection closed by server.',)

As seen in
http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/0ae3179/logs/overcloud-controller-0/var/log/gnocchi/metricd.txt.gz

[Gabriele Cerami (gcerami)]

metricd is continuosly trying to contact redis server, but redis is down. Fails at start with this error

22419:M 31 Oct 07:16:30.624 # Opening Unix socket: bind: Permission denied

as seen in

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/0ae3179/logs/overcloud-controller-0/var/log/redis/redis.txt.gz

[Gabriele Cerami (gcerami)]

Redis starts correctly with the configuration when launched manually. With systemd, it fails.

[Gabriele Cerami (gcerami)]

Selinux policy problem.

I see this in audit/audit.log

type=AVC msg=audit(1477956868.065:23265): avc: denied { write } for pid=11970 comm="redis-server" name="redis" dev="tmpfs" ino=130656 scontext=system_u:system_r:redis_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=dir

When setting enforce to permissive let redis start

[Ryan Hallisey (rthall14)]

`restorecon -Rv /var/run/redis`

This will fix the problem. I think this command will need to be run as part of deployment maybe by puppet?

[Ryan Hallisey (rthall14)]

The restorecon needs to be done on the fly because after openstack-selinux is installed /var/run/redis may not exist. The restorecon needs to be run after that directory is created. Puppet should be able to handle this.

[Alfredo Moralejo (amoralej)]

/var/run is tmpfs, content is lost after every reboot so fixing it with puppet will not help. When running redis from pacemaker, the directory is created by redis resource-agent:

https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/redis#L325

so i think restorecon should be done in the resource agent.

[Gabriele Cerami (gcerami)]

created pull request for resource agents in https://github.com/ClusterLabs/resource-agents/pull/872

[Gabriele Cerami (gcerami)]

Pull request has been merged. We only have to wait for it to be packaged now.

[Gabriele Cerami (gcerami)]

Package is in base RHEL, this is the downstream bugzilla that requires packaging https://bugzilla.redhat.com/show_bug.cgi?id=1390974. This will take some days.
We are working around the issue with this https://review.openstack.org/392521

[Gabriele Cerami (gcerami)]

Added a workaround to make selinux permissive until the resource-agents package is out https://review.openstack.org/392703

[Gabriele Cerami (gcerami)]

new root cause:

https://bugzilla.redhat.com/show_bug.cgi?id=1374728

/var/run/redis automatic creation was removed from redis package 3.2.4.

New package with revert is now in testing at

http://cbs.centos.org/koji/buildinfo?buildID=13831