periodic centos-8-ovb featureset039-master fails overcloud ipa-client install No such file or directory

[Marios Andreou]

At [1][2] the periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master fails during overcloud deploy running the root/setup-ipa-client.sh on the controllers:

          2020-04-26 19:07:29 | TASK [run enrollment script]
          2020-04-26 19:07:30 | fatal: [overcloud-controller-0]: FAILED! => changed=true
          2020-04-26 19:07:30 | cmd: /root/setup-ipa-client.sh >> /var/log/setup-ipa-client-ansible.log 2>&1
          2020-04-26 19:07:30 | fatal: [overcloud-controller-2]: FAILED! => changed=true

Looking more closely the actual error is in the ipaclient-install.log [3]

          2020-04-26T19:07:30Z DEBUG Backing up system configuration file '/
        etc/hostname'
          2020-04-26T19:07:30Z ERROR [Errno 2] No such file or directory: '/var/lib/ipa-client/sysrestore/d7bbe230ea6082f548e8f8faf1965356127b9ecf1466ca658cc72552dc408687-hostname'

This is likely some race/timing issue because it is not happening every run looking at zuul builds [4] at time of filing (attaching screenshot) there are green runs too so not consistent.

[1] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/a7ce16f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[2] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/e5a6722/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[3] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/a7ce16f/logs/overcloud-controller-0/var/log/ipaclient-install.log.txt.gz
[4] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master

[Marios Andreou (marios-b)]

16:51 < akahat|ruck> marios, hey.. you reported this issue: https://bugs.launchpad.net/tripleo/+bug/1875353
16:51 <@openstack> Launchpad bug 1875353 in tripleo "(inconsistent/race) periodic centos-8-ovb featureset039-master fails overcloud
                   ipa-client install No such file or directory: /var/lib/ipa-client/sysrestore" [High,Triaged]
16:51 < akahat|ruck> i think there is problem with the novajoin.
16:52 < akahat|ruck> it still uses python instead of python3. I think those small failures are leading to failure of ipa-install.
16:52 < akahat|ruck> marios, https://opendev.org/x/novajoin/src/branch/master/files/cloud-config-novajoin.yaml#L52
16:53 < akahat|ruck> I found one more job with the same failures.
16:53 < akahat|ruck> https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/4558612/logs/overcloud-controller-1/var/log/setup-ipa-client-ansible.log.txt.gz

[Amol Kahat (amolkahat)]

WIP: https://review.opendev.org/#/c/725465/

[Grzegorz Grasza (xek)]

I checked, that the directory should be provided by ipa-client-common package.

$ rpm -ql ipa-client-common | grep sysrestore
/var/lib/ipa-client/sysrestore

[Pooja Jadhav (poojajadhav22)]

This FS39 master job also alternatively failing with below error logs :

2020-06-03T02:53:29Z INFO Configured sudoers in /etc/authselect/user-nsswitch.conf
2020-06-03T02:53:29Z DEBUG File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 179, in execute
    return_value = self.run()
  File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 340, in run
    return cfgr.run()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 360, in run
    return self.execute()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 386, in execute
    for rval in self._executor():
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 655, in _configure
    next(executor)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 515, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "...

[Rafael Folco (rafaelfolco)]

from today's run:

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/5758322/logs/overcloud-controller-0/var/log/ipaclient-install.log.txt.gz

2020-06-05T11:10:06Z DEBUG The ipa-client-install command failed, exception: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/ipa-client/sysrestore/...'

consistently failing on ipa-client install...

[Marios Andreou (marios-b)]

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo