[8.0-pkg-systest-ubuntu] Timeout failure during setting up master node

[Timur Sufiev]

Job https://packaging-ci.infra.mirantis.net/job/8.0-pkg-systest-ubuntu/1818/ is affected.

Log snippet with error:

======================================================================
ERROR: Create environment and set up master node
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 83, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/tests/base_test_case.py", line 141, in setup_master
    self.env.setup_environment()
  File "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/environment.py", line 453, in setup_environment
    self.wait_bootstrap()
  File "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/environment.py", line 586, in wait_bootstrap
    timeout=(float(settings.PUPPET_TIMEOUT))
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 94, in wait
    raise TimeoutError(timeout_msg)
TimeoutError: Waiting timed out

[Artem Panchenko (apanchenko-8)]

Here is an error from puppet logs on master:

Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: try number 80
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: return code is 1
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: httpd does not appear to be running and proxying cobbler, or SELinux is in the way. Original traceback:
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: Traceback (most recent call last):
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: File "/usr/lib/python2.7/site-packages/cobbler/cli.py", line 252, in check_setup
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: s.ping()
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: return self.__send(self.__name, args)
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: verbose=self.__verbose
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: return self.single_request(host, handler, request_body, verbose)
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: File "/usr/lib64/python2.7/xmlrpclib.py", line 1292, in single_request
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: self.send_content(h, request_body)
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: File "/usr/lib64/python2.7/xmlrpclib.py", line 1439, in send_content
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: connection.endheaders(request_body)
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: File "/usr/lib64/python2.7/httplib.py", line 969, in endheaders
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: self._send_output(message_body)
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: File "/usr/lib64/python2.7/httplib.py", line 829, in _send_output
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: self.send(msg)
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: File "/usr/lib64/python2.7/httplib.py", line 791, in send
Debug: /Stage[main]/Docker/Docker::Build[cobbler]/Exec[container9check]/returns: self.connect()
Debug: /Stage[main]/Docker/...

[Matthew Mosesohn (raytrac3r)]

Cobbler container failed to build, but somehow /var/log/docker-logs/puppet/ is empty. On my local env, it has a cobbler.log file. If you reproduce this, please give me a live env (or logs are ok if we can get /var/log/docker-logs/pupet/cobbler.log )

[Ksenia Svechnikova (kdemina)]

This issue was reproduced on each test for swarms of ISO# fuel-8.0-483.

There is no /var/log/docker-logs/pupet/cobbler.log on master after this issue, but I can share the env.

On env from testgroup "services_reconfiguration_thread_1" i've seen no Cobbler error, but it was RabbitMQ:

ESC[0;36mDebug: /Stage[main]/Docker/Docker::Build[rabbitmq]/Exec[container1check]/returns: try number 81ESC[0m
ESC[0;36mDebug: /Stage[main]/Docker/Docker::Build[rabbitmq]/Exec[container1check]/returns: return code is 1ESC[0m
ESC[0;36mDebug: /Stage[main]/Docker/Docker::Build[rabbitmq]/Exec[container1check]/returns: checking with command "! shell_container rabbitmq pgrep puppet"ESC[0m
ESC[0;36mDebug: /Stage[main]/Docker/Docker::Build[rabbitmq]/Exec[container1check]/returns: ERROR: rabbitmq failed to start.ESC[0m
ESC[1;31mError: dockerctl --debug check rabbitmq returned 1 instead of one of [0]ESC[0m
ESC[1;31mError: /Stage[main]/Docker/Docker::Build[rabbitmq]/Exec[container1check]/returns: change from notrun to 0 failed: dockerctl --debug check rabbitmq returned 1 instead of one of [0]ESC[0m
ESC[mNotice: /Stage[main]/Docker/Docker::Build[keystone]/Exec[container2build]: Dependency Exec[container1check] has failures: trueESC[0m
ESC[1;31mWarning: /Stage[main]/Docker/Docker::Build[keystone]/Exec[container2build]: Skipping because of failed dependenciesESC[0m

[Ksenia Svechnikova (kdemina)]

Raise the issue to Critical as it affect both bvt and more then 50% of swarms

[Matthew Mosesohn (raytrac3r)]

I believe this was caused by refactoring fuelmenu --save-only to use the same functions as the full-fledged modules. The issue in rabbitmq is:
ERROR: epmd error for host nailgun: timeout (timed out)

This means that rabbit can't look up its own hostname in /etc/hosts. We should fix /etc/hosts in any of the following places:
kickstart (with default IP)
fuelmenu --save-only mode
during puppet fuel master run (but before starting containers)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/272958

[Artem Panchenko (apanchenko-8)]

@Ksenia, @Matthew,

guys, you are talking about another issue, please check bug #1538509

[Matthew Mosesohn (raytrac3r)]

This is related to a strange cobbler error, not rabbitmq. Leaving incomplete.

[Artem Panchenko (apanchenko-8)]

Moved this bug back to 'Incomplete', because according to diagnostic snapshot there was an issue with Cobbler start and it's not related to DNS problem (bug #1538509)

[Matthew Mosesohn (raytrac3r)]

We can reopen this bug if QA (or anybody) can reproduce Fuel Master deployment failure related to Cobbler container failing to deploy

[Arthur Svechnikov (asvechnikov)]

Tag swarm-blocker removed because this tag related to DNS problem (another bug) not Cobbler error.

[Matthew Mosesohn (raytrac3r)]

This won't be fixed by 8.0 HCF tomorrow. Moving to 8.0-updates.