undercloud upgrade from Ocata to Pike failed due to /usr/libexec/os-refresh-config/configure.d/50-heat-config-docker-cmd (heat-config) [WARNING] Could not load config json: No JSON object could be decoded

[Artem Hrechanychenko]

Description
===========
I tried to upgrade Ocata cluster to Pike with containerized services.
undercloud upgrade failed:
2017-05-23 13:15:37,700 INFO: dib-run-parts Tue May 23 13:15:37 UTC 2017 Running /usr/libexec/os-refresh-config/configure.d/50-heat-config-docker-cmd
2017-05-23 13:15:37,732 INFO: [2017-05-23 13:15:37,730] (heat-config) [WARNING] Could not load config json: No JSON object could be decoded
2017-05-23 13:15:37,738 INFO: [2017-05-23 13:15:37,732] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
2017-05-23 13:15:37,738 INFO:
2017-05-23 13:15:37,738 INFO: [2017-05-23 13:15:37,732] (os-refresh-config) [ERROR] Aborting...
2017-05-23 13:15:37,739 ERROR:
#############################################################################
Undercloud upgrade failed.

Reason: os-refresh-config failed. See log for details.

See the previous output for details about what went wrong. The full install
log can be found at /home/stack/.instack/install-undercloud.log.

#############################################################################

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1567, in install
    _run_orc(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1254, in _run_orc
    _run_live_command(args, instack_env, 'os-refresh-config')
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 585, in _run_live_command
    raise RuntimeError('%s failed. See log for details.' % name)
RuntimeError: os-refresh-config failed. See log for details.
Command 'instack-upgrade-undercloud' returned non-zero exit status 1
[stack@undercloud ~]$

Steps to reproduce
==================
1)Install undercloud v.Ocata using tripleo-quickstart:
bash quickstart.sh --release ocata $HOST

2)deploy overcloud:
openstack overcloud deploy \
    --libvirt-type qemu \
    --ntp-server clock.redhat.com \
    --control-scale 1 \
    --templates $THT \
    -e $THT/environments/low-memory-usage.yaml \

3) perform upgrade procedure for undercloud
### UPGRADE UNDERCLOUD ###

# master repos
sudo curl -L -o /etc/yum.repos.d/delorean.repo https://trunk.rdoproject.org/centos7-master/current-passed-ci/delorean.repo

sudo curl -L -o /etc/yum.repos.d/delorean-current.repo https://trunk.rdoproject.org/centos7/current/delorean.repo
sudo sed -i 's/\[delorean\]/\[delorean-current\]/' /etc/yum.repos.d/delorean-current.repo
sudo /bin/bash -c "cat <<EOF>>/etc/yum.repos.d/delorean-current.repo

includepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common*,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,openstack-puppet-modules,openstack-tripleo-ui,puppet-*
EOF"
sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo https://trunk.rdoproject.org/centos7/delorean-deps.repo

sudo systemctl stop openstack-*
sudo systemctl stop neutron-*
sudo systemctl stop httpd
sudo yum -y update instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient
openstack undercloud upgrade

Expected result
===============
Undercloud was upgraded

Actual result
=============
2017-05-23 13:15:37,267 INFO: + systemctl reload keepalived
2017-05-23 13:15:37,323 INFO: dib-run-parts Tue May 23 13:15:37 UTC 2017 30-reload-keepalived completed
2017-05-23 13:15:37,324 INFO: dib-run-parts Tue May 23 13:15:37 UTC 2017 Running /usr/libexec/os-refresh-config/configure.d/40-hiera-datafiles
2017-05-23 13:15:37,699 INFO: dib-run-parts Tue May 23 13:15:37 UTC 2017 40-hiera-datafiles completed
2017-05-23 13:15:37,700 INFO: dib-run-parts Tue May 23 13:15:37 UTC 2017 Running /usr/libexec/os-refresh-config/configure.d/50-heat-config-docker-cmd
2017-05-23 13:15:37,732 INFO: [2017-05-23 13:15:37,730] (heat-config) [WARNING] Could not load config json: No JSON object could be decoded
2017-05-23 13:15:37,738 INFO: [2017-05-23 13:15:37,732] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
2017-05-23 13:15:37,738 INFO:
2017-05-23 13:15:37,738 INFO: [2017-05-23 13:15:37,732] (os-refresh-config) [ERROR] Aborting...
2017-05-23 13:15:37,739 ERROR:
#############################################################################
Undercloud upgrade failed.

Environment
===========
1. Ocata

2. default

3. default

Logs & Configs
==============
cat /var/run/heat-config/heat-config
[stack@undercloud ~]$ cd /var/run/heat-config/
[stack@undercloud heat-config]$ ls
heat-config
[stack@undercloud heat-config]$ ls -la heat-config
-rw-r--r--. 1 root root 0 May 23 13:15 heat-config

undercloud install log in attachment

[Jiří Stránský (jistr)]

Fix https://review.openstack.org/#/c/467248/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/467248
Committed: https://git.openstack.org/cgit/openstack/heat-agents/commit/?id=3eb25e7df3ba032b17ba147967728e62729eb5fd
Submitter: Jenkins
Branch: master

commit 3eb25e7df3ba032b17ba147967728e62729eb5fd
Author: Jiri Stransky <email address hidden>
Date: Tue May 23 16:05:20 2017 +0200

    Don't fail on warnings in docker-cmd

    It seems that during the undercloud upgrade heat-config can be empty,
    and if it is, the heat-config-docker-cmd hook returns 1 and makes
    os-refresh-config fail, consequently failing the undercloud upgrade.

    Change-Id: Ia62e40e05f46e7e4450c6b230ddeec9b861e99bd
    Related-Bug: #1648249
    Closes-Bug: #1692899

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat-agents 1.2.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/487656