Possible DNS issue in the CI

[Cédric Jeanneret deactivated]

Hello,

I suspect a small DNS issue in the CI for gate-tripleo-ci-centos-7-nonha-multinode-oooq:
http://logs.openstack.org/97/509397/3/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/d67f8e5/console.html#_2017-10-06_06_52_33_436723

fatal: [undercloud]: FAILED! => {"changed": false, "cmd": "/usr/bin/git clone --origin origin https://github.com/redhat-openstack/rdoinfo /home/jenkins/DLRN/rdoinfo", "failed": true, "msg": "fatal: unable to access 'https://github.com/redhat-openstack/rdoinfo/': Could not resolve host: github.com; Name or service not known", "rc": 128, "stderr": "fatal: unable to access 'https://github.com/redhat-openstack/rdoinfo/': Could not resolve host: github.com; Name or service not known\n", "stdout": "Cloning into '/home/jenkins/DLRN/rdoinfo'...\n", "stdout_lines": ["Cloning into '/home/jenkins/DLRN/rdoinfo'..."]}

[Alex Schultz (alex-schultz)]

This happens from time to time and we need to fix the unbound.log so we can see why

[David Moreau Simard (dmsimard)]

We are not seeing any logs in unbound because the log file is configured with a selinux context that does not allow unbound to write to it:

https://github.com/openstack-infra/project-config/blob/544fe001f477be137725705d92086e3bb0445b29/nodepool/elements/nodepool-base/finalise.d/89-unbound#L64-L65

Oct 06 16:57:44 upstream-centos-7-rdo-cloud-tripleo-14922 unbound[1191]: Oct 06 16:57:44 unbound[1191:0] error: Could not open logfile /var/log/unbound.log: Permission denied

type=AVC msg=audit(1507309060.078:44): avc: denied { open } for pid=1058 comm="unbound" path="/var/log/unbound.log" dev="vda1" ino=2682530 scontext=system_u:system_r:named_t:s0 tcontext=system_u:object_r:var_log_t:s0 tclass=file

bash-4.2# ls -lZ /var/log/unbound.log
-rw-r--r--. unbound root system_u:object_r:var_log_t:s0 /var/log/unbound.log

bash-4.2# chcon -t named_log_t /var/log/unbound.log

bash-4.2# systemctl restart unbound

bash-4.2# cat /var/log/unbound.log
Oct 06 17:54:52 unbound[2483:0] info: start of service (unbound 1.4.20).

[David Moreau Simard (dmsimard)]

It turns out that /var/log/unbound.log is not supposed to be used in the upstream nodepool images, rather /var/lib/unbound/unbound.log: https://review.openstack.org/#/c/510202/

/var/log/unbound.log is likely used on review.rdoproject.org images due to the delta between upstream and downstream images since we have stopped synchronizing project-config.

[Paul Belanger (pabelanger)]

Some more information, github.com only has TTL of 30secods, which doesn't make it good for caching results in unbound. we've discussed maybe setting cache-min-ttl: 300 in unbound.conf, but that could be an issue if github.com changes the DNS (maybe the CDN network) during that cache period.

val-bogus-ttl: 60 might be able to help with this, but needs to be tested. Which dmsimard signed up for :)

A parallel effort should be to move rdoinfo off github.com.

[Alan Pevec (apevec)]

rdoinfo is actually at https://review.rdoproject.org/r/rdoinfo
github.com is only a mirror but it was argued that it is more reliable than rdoproject.org hence it is used.
Alternative would be to mirror it into upstream infra but I've heard that proxy-cache for git over http will not work, any other suggestions?

[David Moreau Simard (dmsimard)]

cache-bogus-ttl is not what we expected it to be.
It sounds like what we'd like to do doesn't exist (yet), it would be the opposite of "cache-max-negative-ttl" see: https://bugzilla.redhat.com/show_bug.cgi?id=1360222

In the meantime, we're improving the configure-unbound role to allow for configuration of a minimum TTL. Jobs impacted by frequent DNS issues will be encouraged to tweak the minimum TTL value to see if this helps.

See:
- https://review.openstack.org/#/c/523228/
- https://review.openstack.org/#/c/523178/

[Alan Pevec (apevec)]

Min. TTL in tripleo-ci is enforced after https://review.openstack.org/524018
merged Dec 1.
Are there anymore DNS issues since then?

[Sagi (Sergey) Shnaidman (sshnaidm)]

Reopening this, because it happens again:

http://logs.openstack.org/31/518331/16/experimental-tripleo/tripleo-ci-centos-7-ovb-fakeha-caserver/96557c2/job-output.txt.gz#_2018-01-04_17_53_15_331819

[Juan Antonio Osorio Robles (juan-osorio-robles)]

Is this still an issue?

[Javier Peña (jpena-c)]

I haven't seen this for a long time, so I think we could close it.