check grafana containers task fails when adding new dashboard

[Dawud]

When adding a new dashboard to etc/kolla/config/grafana/dashboards and running Kolla Ansible the task `TASK [grafana : Check grafana containers]` fails with a file not found error.

Steps to reproduce:
 - Add a new dashboard to etc/kolla/config/grafana/dashboards
 - Reconfigure Grafana
 - You will then hit the issue below.

Workaround:
 The current work around at the moment is to manually restart the Grafana container

System Details:
 - OS: Stable/Yoga

Error Log:

TASK [grafana : Remove templated Grafana dashboards] ***********************************************************************************************************
changed: [host1]
changed: [host2]
changed: [host3]

TASK [grafana : Copying over custom dashboards] ****************************************************************************************************************
changed: [host1]
changed: [host2]
changed: [host3]

TASK [grafana : Check grafana containers] **********************************************************************************************************************
failed: [host2] (item={'key': 'grafana', 'value': {'container_name': 'grafana', 'group': 'grafana', 'enabled': True, 'image': '10.10.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200', 'volumes': ['/etc/kolla/grafana/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '', 'grafana:/var/lib/grafana/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'haproxy': {'grafana_server': {'enabled': True, 'mode': 'http', 'external': False, 'port': '3000'}, 'grafana_server_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '3000'}}}}) => changed=true
  ansible_loop_var: item
  item:
    key: grafana
    value:
      container_name: grafana
      dimensions: {}
      enabled: true
      group: grafana
      haproxy:
        grafana_server:
          enabled: true
          external: false
          mode: http
          port: '3000'
        grafana_server_external:
          enabled: true
          external: true
          mode: http
          port: '3000'
      image: 10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200
      volumes:
      - /etc/kolla/grafana/:/var/lib/kolla/config_files/:ro
      - /etc/localtime:/etc/localtime:ro
      - ''
      - grafana:/var/lib/grafana/
      - kolla_logs:/var/log/kolla/
  msg: '''Traceback (most recent call last):\n File "/tmp/ansible_kolla_docker_payload_ya9jvcwp/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py", line 381, in main\n File "/tmp/ansible_kolla_docker_payload_ya9jvcwp/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 115, in compare_container\n self.compare_config()):\n File "/tmp/ansible_kolla_docker_payload_ya9jvcwp/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 416, in compare_config\n (exec_inspect[\''ExitCode\''], output))\nException: Failed to compare container configuration: ExitCode: 2 Message: b\''INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\\nINFO:__main__:Validating config file\\nERROR:__main__:Unexpected error:\\nTraceback (most recent call last):\\n File "/usr/local/bin/kolla_set_configs", line 420, in main\\n execute_config_check(config)\\n File "/usr/local/bin/kolla_set_configs", line 406, in execute_config_check\\n config_file.check()\\n File
    "/usr/local/bin/kolla_set_configs", line 231, in check\\n if not self._cmp_dir(source, dest):\\n File "/usr/local/bin/kolla_set_configs", line 211, in _cmp_dir\\n if not self._cmp_file(full_path, dest_full_path):\\n File "/usr/local/bin/kolla_set_configs", line 169, in _cmp_file\\n with open(source, \\\''rb\\\'') as f1, open(dest, \\\''rb\\\'') as f2:\\nFileNotFoundError: [Errno 2] No such file or directory: \\\''/var/lib/grafana/dashboards/openstack/hardware_overview.json\\\''\\n\''\n'''
failed: [host3] (item={'key': 'grafana', 'value': {'container_name': 'grafana', 'group': 'grafana', 'enabled': True, 'image': '10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200', 'volumes': ['/etc/kolla/grafana/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '', 'grafana:/var/lib/grafana/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'haproxy': {'grafana_server': {'enabled': True, 'mode': 'http', 'external': False, 'port': '3000'}, 'grafana_server_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '3000'}}}}) => changed=true
  ansible_loop_var: item
  item:
    key: grafana
    value:
      container_name: grafana
      dimensions: {}
      enabled: true
      group: grafana
      haproxy:
        grafana_server:
          enabled: true
          external: false
          mode: http
          port: '3000'
        grafana_server_external:
          enabled: true
          external: true
          mode: http
          port: '3000'
      image: 10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200
      volumes:
      - /etc/kolla/grafana/:/var/lib/kolla/config_files/:ro
      - /etc/localtime:/etc/localtime:ro
      - ''
      - grafana:/var/lib/grafana/
      - kolla_logs:/var/log/kolla/
  msg: '''Traceback (most recent call last):\n File "/tmp/ansible_kolla_docker_payload__46t34ml/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py", line 381, in main\n File "/tmp/ansible_kolla_docker_payload__46t34ml/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 115, in compare_container\n self.compare_config()):\n File "/tmp/ansible_kolla_docker_payload__46t34ml/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 416, in compare_config\n (exec_inspect[\''ExitCode\''], output))\nException: Failed to compare container configuration: ExitCode: 2 Message: b\''INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\\nINFO:__main__:Validating config file\\nERROR:__main__:Unexpected error:\\nTraceback (most recent call last):\\n File "/usr/local/bin/kolla_set_configs", line 420, in main\\n execute_config_check(config)\\n File "/usr/local/bin/kolla_set_configs", line 406, in execute_config_check\\n config_file.check()\\n File
    "/usr/local/bin/kolla_set_configs", line 231, in check\\n if not self._cmp_dir(source, dest):\\n File "/usr/local/bin/kolla_set_configs", line 211, in _cmp_dir\\n if not self._cmp_file(full_path, dest_full_path):\\n File "/usr/local/bin/kolla_set_configs", line 169, in _cmp_file\\n with open(source, \\\''rb\\\'') as f1, open(dest, \\\''rb\\\'') as f2:\\nFileNotFoundError: [Errno 2] No such file or directory: \\\''/var/lib/grafana/dashboards/openstack/hardware_overview.json\\\''\\n\''\n'''
failed: [host1] (item={'key': 'grafana', 'value': {'container_name': 'grafana', 'group': 'grafana', 'enabled': True, 'image': '10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200', 'volumes': ['/etc/kolla/grafana/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '', 'grafana:/var/lib/grafana/', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'haproxy': {'grafana_server': {'enabled': True, 'mode': 'http', 'external': False, 'port': '3000'}, 'grafana_server_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '3000'}}}}) => changed=true
  ansible_loop_var: item
  item:
    key: grafana
    value:
      container_name: grafana
      dimensions: {}
      enabled: true
      group: grafana
      haproxy:
        grafana_server:
          enabled: true
          external: false
          mode: http
          port: '3000'
        grafana_server_external:
          enabled: true
          external: true
          mode: http
          port: '3000'
      image: 10.103.1.0/stackhpc/centos-source-grafana:yoga-20221007T094200
      volumes:
      - /etc/kolla/grafana/:/var/lib/kolla/config_files/:ro
      - /etc/localtime:/etc/localtime:ro
      - ''
      - grafana:/var/lib/grafana/
      - kolla_logs:/var/log/kolla/
  msg: '''Traceback (most recent call last):\n File "/tmp/ansible_kolla_docker_payload_0uk3g3pb/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py", line 381, in main\n File "/tmp/ansible_kolla_docker_payload_0uk3g3pb/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 115, in compare_container\n self.compare_config()):\n File "/tmp/ansible_kolla_docker_payload_0uk3g3pb/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py", line 416, in compare_config\n (exec_inspect[\''ExitCode\''], output))\nException: Failed to compare container configuration: ExitCode: 2 Message: b\''INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\\nINFO:__main__:Validating config file\\nERROR:__main__:Unexpected error:\\nTraceback (most recent call last):\\n File "/usr/local/bin/kolla_set_configs", line 420, in main\\n execute_config_check(config)\\n File "/usr/local/bin/kolla_set_configs", line 406, in execute_config_check\\n config_file.check()\\n File
    "/usr/local/bin/kolla_set_configs", line 231, in check\\n if not self._cmp_dir(source, dest):\\n File "/usr/local/bin/kolla_set_configs", line 211, in _cmp_dir\\n if not self._cmp_file(full_path, dest_full_path):\\n File "/usr/local/bin/kolla_set_configs", line 169, in _cmp_file\\n with open(source, \\\''rb\\\'') as f1, open(dest, \\\''rb\\\'') as f2:\\nFileNotFoundError: [Errno 2] No such file or directory: \\\''/var/lib/grafana/dashboards/openstack/hardware_overview.json\\\''\\n\''\n'''

RUNNING HANDLER [grafana : Restart first grafana container] ****************************************************************************************************

RUNNING HANDLER [grafana : Waiting for grafana to start on first node] *****************************************************************************************

RUNNING HANDLER [grafana : Restart remaining grafana containers] ***********************************************************************************************

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla/+/873972

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla/+/893766

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/892876
Committed: https://opendev.org/openstack/kolla/commit/288a400dbf30797b70e64af64f6deadcca582851
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 288a400dbf30797b70e64af64f6deadcca582851
Author: Dawud M <email address hidden>
Date: Wed Feb 15 18:02:05 2023 +0000

    Add comprehensive checks for container restarts

    When adding a dashboard to grafana the containers aren't restarted when
    they should be. This is due to a bug in Kolla where the logic to
    determine whether or the container needs to be restarted fails in the
    case where the file does not exist in the container. This patch adds
    more comprehensive checks for container restarts in the set_configs.py
    file. This patch also adds a test to ensure that the functions work as
    expected.

    Closes-Bug: #1997984
    Co-Authored-By: Will Szumski <email address hidden>
    Change-Id: I67f5f12700d7b55f26bff81e9b54559303da6d83
    (cherry picked from commit d9a6c5f3901e078a30d341397ac57d867d8191cc)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Michal Nasiadka <email address hidden>" on branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla/+/893766
Reason: xena going EOL

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/892878
Committed: https://opendev.org/openstack/kolla/commit/0353c51c68c30632ecb29fd0ae4372a375086865
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 0353c51c68c30632ecb29fd0ae4372a375086865
Author: Dawud M <email address hidden>
Date: Wed Feb 15 18:02:05 2023 +0000

    Add comprehensive checks for container restarts

    When adding a dashboard to grafana the containers aren't restarted when
    they should be. This is due to a bug in Kolla where the logic to
    determine whether or the container needs to be restarted fails in the
    case where the file does not exist in the container. This patch adds
    more comprehensive checks for container restarts in the set_configs.py
    file. This patch also adds a test to ensure that the functions work as
    expected.

    Closes-Bug: #1997984
    Co-Authored-By: Will Szumski <email address hidden>
    Change-Id: I67f5f12700d7b55f26bff81e9b54559303da6d83
    (cherry picked from commit d9a6c5f3901e078a30d341397ac57d867d8191cc)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 16.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 14.11.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 15.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 17.0.0.0rc1 release candidate.