CentOS: Restore silently failing on task "Update host config data"

[Thiago Paiva Brito]

Brief Description
-----------------
The restore process is silently failing on task "Update host config data". The playbook finishes but after the reboot, rabbimq doesn't come up.

Severity
--------
Critical: Feature is not usable due to the defect

Steps to Reproduce
------------------
Take a backup and restore it.

Expected Behavior
------------------
Restore finishes, controller-0 reboots and the system is usable.

Actual Behavior
----------------
Restore finishes, controller-0 reboots but the sysinv never comes up due to rabbitmq never coming up to receive the messages.

Reproducibility
---------------
1/1

System Configuration
--------------------
AIO-SX

Branch/Pull Time/Commit
-----------------------
2022-07-21

Last Pass
---------
not sure, seems intermittent

Timestamp/Logs
--------------
Running ansible-playbook with -vvv:

2022-07-22 10:39:41,317 p=10780 u=sysadmin | TASK [restore-platform/restore-more-data : Update host config data] ************
2022-07-22 10:39:41,317 p=10780 u=sysadmin | task path: /usr/share/ansible/stx-ansible/playbooks/roles/restore-platform/restore-more-data/tasks/main.yml:663
2022-07-22 10:39:41,318 p=10780 u=sysadmin | Friday 22 July 2022 10:39:41 +0000 (0:00:00.032) 0:23:28.237 ***********
2022-07-22 10:39:41,359 p=10780 u=sysadmin | Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
2022-07-22 10:39:45,687 p=10780 u=sysadmin | changed: [localhost] => changed=true
  cmd:
  - /usr/bin/sysinv-puppet
  - create-host-config
  delta: '0:00:04.184723'
  end: '2022-07-22 10:39:45.653752'
  invocation:
    module_args:
      _raw_params: /usr/bin/sysinv-puppet create-host-config
      _uses_shell: false
      argv: null
      chdir: null
      creates: null
      executable: null
      removes: null
      stdin: null
      warn: true
  rc: 0
  start: '2022-07-22 10:39:41.469029'
  stderr: |-
    /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      InsecureRequestWarning)
    /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      InsecureRequestWarning)
    /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      InsecureRequestWarning)
    /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      InsecureRequestWarning)
    /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      InsecureRequestWarning)
    /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      InsecureRequestWarning)
    Exception AttributeError: "'Queue' object has no attribute 'not_empty'" in <Finalize object, dead> ignored
    Traceback (most recent call last):
      File "/usr/lib64/python2.7/multiprocessing/util.py", line 268, in _run_finalizers
        finalizer()
      File "/usr/lib64/python2.7/multiprocessing/util.py", line 201, in __call__
        res = self._callback(*self._args, **self._kwargs)
      File "/usr/lib64/python2.7/multiprocessing/pool.py", line 482, in _terminate_pool
        cls._help_stuff_finish(inqueue, task_handler, len(pool))
      File "/usr/lib64/python2.7/multiprocessing/pool.py", line 725, in _help_stuff_finish
        inqueue.not_empty.acquire()
    AttributeError: 'Queue' object has no attribute 'not_empty'
  stderr_lines:
  - '/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings'
  - ' InsecureRequestWarning)'
  - '/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings'
  - ' InsecureRequestWarning)'
  - '/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings'
  - ' InsecureRequestWarning)'
  - '/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings'
  - ' InsecureRequestWarning)'
  - '/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings'
  - ' InsecureRequestWarning)'
  - '/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings'
  - ' InsecureRequestWarning)'
  - 'Exception AttributeError: "''Queue'' object has no attribute ''not_empty''" in <Finalize object, dead> ignored'
  - 'Traceback (most recent call last):'
  - ' File "/usr/lib64/python2.7/multiprocessing/util.py", line 268, in _run_finalizers'
  - ' finalizer()'
  - ' File "/usr/lib64/python2.7/multiprocessing/util.py", line 201, in __call__'
  - ' res = self._callback(*self._args, **self._kwargs)'
  - ' File "/usr/lib64/python2.7/multiprocessing/pool.py", line 482, in _terminate_pool'
  - ' cls._help_stuff_finish(inqueue, task_handler, len(pool))'
  - ' File "/usr/lib64/python2.7/multiprocessing/pool.py", line 725, in _help_stuff_finish'
  - ' inqueue.not_empty.acquire()'
  - 'AttributeError: ''Queue'' object has no attribute ''not_empty'''
  stdout: |-
    sysinv 2022-07-22 10:39:43.703 155272 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: None[00m
    sysinv 2022-07-22 10:39:44.925 155272 INFO sysinv.puppet.interface [-] Interface data0 has no primary address[00m
    sysinv 2022-07-22 10:39:44.926 155272 INFO sysinv.puppet.interface [-] Interface data1 has no primary address[00m
  stdout_lines: <omitted>

If I try to rerun sysinv-puppet manually I got the same error:

[sysadmin@controller-0 ~(keystone_admin)]$ sudo /usr/bin/sysinv-puppet create-host-config
Password:
sysinv 2022-07-22 12:17:48.570 277792 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: None
sysinv 2022-07-22 12:17:49.664 277792 INFO sysinv.puppet.interface [-] Interface data0 has no primary address
sysinv 2022-07-22 12:17:49.664 277792 INFO sysinv.puppet.interface [-] Interface data1 has no primary address
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
88983ac0b001f4fd6c274883f668f18e29732cd03abbdafaa21f93050906d355
sysinv 2022-07-22 12:17:53.666 277792 INFO sysinv.puppet.kubernetes [-] get_kubernetes_join_cmd join_cmd=kubeadm join 192.168.206.1:6443 --token 7m08h0.xt349hbkr07uysmq --discovery-token-ca-cert-hash sha256:ba88b22ce5ddeb480ae6f3bcabd1bb89caa1d4ec9c728587418fbddaf3ce0b95 --control-plane --certificate-key 88983ac0b001f4fd6c274883f668f18e29732cd03abbdafaa21f93050906d355 --apiserver-advertise-address 192.168.206.2 --cri-socket /var/run/containerd/containerd.sock
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
Exception AttributeError: "'Queue' object has no attribute 'not_empty'" in <Finalize object, dead> ignored
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/util.py", line 268, in _run_finalizers
    finalizer()
  File "/usr/lib64/python2.7/multiprocessing/util.py", line 201, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 482, in _terminate_pool
    cls._help_stuff_finish(inqueue, task_handler, len(pool))
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 725, in _help_stuff_finish
    inqueue.not_empty.acquire()
AttributeError: 'Queue' object has no attribute 'not_empty'

Test Activity
-------------
Feature Testing

Workaround
----------
n/a

[Thiago Paiva Brito (outbrito)]

Al Bailey commented that this might be related to the python3 changes introduced on the debian transition.

[John Kung (john-kung)]

No progress on this bug for more than a year and CentOS specific. Candidate for closure.

If there is no update, this issue is targeted to be closed as 'Won't Fix' in 2 weeks.