ceilometer_agent_notification container cannot start with disable-panko.yaml

[Nagasai Vinaykumar Kapalavai]

Description of problem:

We reported ceilometer_agent_notification cannot start without panko on bz#1633900.
But the report was closed by duplicate of bz#1613601.

We deploy overcloud with disable-panko.yaml which is included in openstack-tripleo-heat-templates-8.2.0-6.2.el7ost.noarch and released as fix of bz#1613601.

BUT, ceilometer_agent_notification could not start AGAIN.

  # docker ps | grep ceilometer_agent_notification
  c0aec96c1d4f 192.168.14.1:5000/rhosp13/openstack-ceilometer-notification:13.0-67 "kolla_start" About an hour ago Restarting (2) 9 minutes ago ceilometer_agent_notification

  # docker inspect ceilometer_agent_notification | grep Status
            "Status": "restarting",
                "Status": "starting",

  # docker logs ceilometer_agent_notification
  + sudo -E kolla_set_configs
  INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
  INFO:__main__:Validating config file
  ERROR:__main__:Unexpected error:
  Traceback (most recent call last):
    File "/usr/local/bin/kolla_set_configs", line 406, in main
      config = load_config()
    File "/usr/local/bin/kolla_set_configs", line 297, in load_config
      validate_config(config)
    File "/usr/local/bin/kolla_set_configs", line 238, in validate_config
      if not data.viewkeys() >= required_keys:
  AttributeError: 'unicode' object has no attribute 'viewkeys'

It seems that kolla_config for ceilometer_agent_notification is something wrong.
I think fix of bz#1613601 is obviously insufficient.
And I wonder why QA testing on bz#1613601 was passed.

Version-Release number of selected component (if applicable):

openstack-tripleo-heat-templates-8.2.0-6.2.el7ost.noarch

How reproducible:

Steps to Reproduce:
1. Deploy overcloud with ceilometer and disable-panko.yaml

Actual results:

ceilometer_agent_notification cannot start.

# docker ps | grep ceilometer
c0aec96c1d4f 192.168.14.1:5000/rhosp13/openstack-ceilometer-notification:13.0-67 "kolla_start" 2 hours ago Restarting (2) 4 minutes ago ceilometer_agent_notification

Expected results:

ceilometer_agent_notification can start

Additional info:

Following is latest config (after bz#1613601).

-----------------------------------------------
# jq . /var/lib/kolla/config_files/ceilometer_agent_notification.json
{
  "permissions": [
    {
      "recurse": true,
      "path": "/etc/panko",
      "owner": "root:ceilometer"
    }
  ],
  "command": "/usr/bin/ceilometer-agent-notification --logfile /var/log/ceilometer/agent-notification.log",
  "config_files": [
    {
      "merge": true,
      "preserve_properties": true,
      "source": "/var/lib/kolla/config_files/src/*",
      "dest": "/"
    },
    ""
  ]
}
-----------------------------------------------

Following is previous config (before bz#1613601).

-----------------------------------------------
# jq . ceilometer_agent_notification.json
{
  "permissions": [
    {
      "recurse": true,
      "path": "/etc/panko",
      "owner": "root:ceilometer"
    }
  ],
  "command": "/usr/bin/ceilometer-agent-notification --logfile /var/log/ceilometer/agent-notification.log",
  "config_files": [
    {
      "preserve_properties": true,
      "merge": true,
      "source": "/var/lib/kolla/config_files/src/*",
      "dest": "/"
    },
    {
      "preserve_properties": true,
      "merge": true,
      "source": "/var/lib/kolla/config_files/src-panko/*",
      "dest": "/"
    }
  ]
}
-----------------------------------------------

And once I modified config as follows, ceilometer_agent_notification could start

-----------------------------------------------
$ jq . ceilometer_agent_notification.json
{
  "permissions": [
    {
      "recurse": true,
      "path": "/etc/panko",
      "owner": "root:ceilometer"
    }
  ],
  "command": "/usr/bin/ceilometer-agent-notification --logfile /var/log/ceilometer/agent-notification.log",
  "config_files": [
    {
      "merge": true,
      "preserve_properties": true,
      "source": "/var/lib/kolla/config_files/src/*",
      "dest": "/"
    }
  ]
}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/652699

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/652702

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/652704

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/652704
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5203a4a8326b462226a3ac76a64b64164c8b27ca
Submitter: Zuul
Branch: stable/rocky

commit 5203a4a8326b462226a3ac76a64b64164c8b27ca
Author: Nagasai Vinaykumar Kapalavai <email address hidden>
Date: Mon Apr 15 11:24:15 2019 -0400

    ceilometer_agent_notification: disable-panko.yaml

    ceilometer-agent-notification.yaml unable to start
    as it gets '' instead of null when disable-panko.yaml
    file is passed.

    Closes-Bug: #1824840

    Change-Id: I947f6a4c65a9ef6a9220a07098f4c91507711cf6
    (cherry picked from commit 34b7ba0fa44bc46e11a603fac662909976bdb0c1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/652702
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=847ec663cc3af2c878cae5cd7acc575a9702f239
Submitter: Zuul
Branch: stable/queens

commit 847ec663cc3af2c878cae5cd7acc575a9702f239
Author: Nagasai Vinaykumar Kapalavai <email address hidden>
Date: Mon Apr 15 11:24:15 2019 -0400

    ceilometer_agent_notification: disable-panko.yaml

    ceilometer-agent-notification.yaml unable to start
    as it gets '' instead of null when disable-panko.yaml
    file is passed.

    Closes-Bug: #1824840

    Change-Id: I947f6a4c65a9ef6a9220a07098f4c91507711cf6
    (cherry picked from commit 34b7ba0fa44bc46e11a603fac662909976bdb0c1)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/656383

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/656383
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1762e9b636566e4a78c31def630fffaea44d8f84
Submitter: Zuul
Branch: stable/stein

commit 1762e9b636566e4a78c31def630fffaea44d8f84
Author: Nagasai Vinaykumar Kapalavai <email address hidden>
Date: Mon Apr 15 11:24:15 2019 -0400

    ceilometer_agent_notification: disable-panko.yaml

    ceilometer-agent-notification.yaml unable to start
    as it gets '' instead of null when disable-panko.yaml
    file is passed.

    Closes-Bug: #1824840

    Change-Id: I947f6a4c65a9ef6a9220a07098f4c91507711cf6
    (cherry picked from commit 34b7ba0fa44bc46e11a603fac662909976bdb0c1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.4.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/652699
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8fac4c4334ad2b2ef67040d053a58a9ba8857425
Submitter: Zuul
Branch: master

commit 8fac4c4334ad2b2ef67040d053a58a9ba8857425
Author: Nagasai Vinaykumar Kapalavai <email address hidden>
Date: Mon Apr 15 11:24:15 2019 -0400

    ceilometer_agent_notification: disable-panko.yaml

    ceilometer-agent-notification.yaml updated the
    kolla_config section in template file to include
    nothing related to panko when disable-panko env file is passed.

    Closes-Bug: #1824840

    Change-Id: I947f6a4c65a9ef6a9220a07098f4c91507711cf6

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/668495

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/668497

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/668498

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668495
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9a1491a42fd6679b9222eb22af5844ce3edf5067
Submitter: Zuul
Branch: stable/stein

commit 9a1491a42fd6679b9222eb22af5844ce3edf5067
Author: Nagasai Vinaykumar Kapalavai <email address hidden>
Date: Mon Apr 15 11:24:15 2019 -0400

    ceilometer_agent_notification: disable-panko.yaml

    ceilometer-agent-notification.yaml updated the
    kolla_config section in template file to include
    nothing related to panko when disable-panko env file is passed.

    Closes-Bug: #1824840

    Change-Id: Ia71e80c84449209653609c911fef4cb3cdce5b5c
    (cherry picked from commit 8fac4c4334ad2b2ef67040d053a58a9ba8857425)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668497
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=fd62ee68503df96091ef529041e394a550b3a5d4
Submitter: Zuul
Branch: stable/rocky

commit fd62ee68503df96091ef529041e394a550b3a5d4
Author: Nagasai Vinaykumar Kapalavai <email address hidden>
Date: Mon Apr 15 11:24:15 2019 -0400

    ceilometer_agent_notification: disable-panko.yaml

    ceilometer-agent-notification.yaml updated the
    kolla_config section in template file to include
    nothing related to panko when disable-panko env file is passed.

    Closes-Bug: #1824840

    Change-Id: Ia71e80c84449209653609c911fef4cb3cdce5b5c
    (cherry picked from commit 8fac4c4334ad2b2ef67040d053a58a9ba8857425)
    (cherry picked from commit 9a1491a42fd6679b9222eb22af5844ce3edf5067)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668498
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=de9453ba2a4dabbf5d5e0838e7c12745237876aa
Submitter: Zuul
Branch: stable/queens

commit de9453ba2a4dabbf5d5e0838e7c12745237876aa
Author: Nagasai Vinaykumar Kapalavai <email address hidden>
Date: Mon Apr 15 11:24:15 2019 -0400

    ceilometer_agent_notification: disable-panko.yaml

    ceilometer-agent-notification.yaml updated the
    kolla_config section in template file to include
    nothing related to panko when disable-panko env file is passed.

    Closes-Bug: #1824840

    Change-Id: Ia71e80c84449209653609c911fef4cb3cdce5b5c
    (cherry picked from commit 8fac4c4334ad2b2ef67040d053a58a9ba8857425)
    (cherry picked from commit 9a1491a42fd6679b9222eb22af5844ce3edf5067)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.4.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.4.1 release.