reconfigure_storage abort with KeyNotFoundError seen

[wenqing liang]

In r3.1.0.0-build 13 mitaka, fab reconfigure_storage after storage parameter change in testbed.py aborted with:

2016-08-02 23:53:54:461828: [root@10.87.141.120] out: [13.0.0.1] out: [root@13.0.0.1] out: [ceph_deploy.gatherkeys][WARNIN] Unable to find /etc/ceph/ceph.client.admin.keyring on a5d01e03-storage
2016-08-02 23:53:54:461969: [root@10.87.141.120] out: [13.0.0.1] out: [root@13.0.0.1] out: [ceph_deploy][ERROR ] KeyNotFoundError: Could not find keyring file: /etc/ceph/ceph.client.admin.keyring on host a5d01e03-storage
2016-08-02 23:53:54:462069: [root@10.87.141.120] out: [13.0.0.1] out: [root@13.0.0.1] out:
2016-08-02 23:53:54:462164: [root@10.87.141.120] out: [13.0.0.1] out: [root@13.0.0.1] out:
2016-08-02 23:53:54:494352: [root@10.87.141.120] out: [13.0.0.1] out:
2016-08-02 23:53:54:526163: [root@10.87.141.120] out: [13.0.0.1] out:
2016-08-02 23:53:54:526298: [root@10.87.141.120] out: [13.0.0.1] out: Fatal error: run() received nonzero return code 1 while executing!
2016-08-02 23:53:54:526393: [root@10.87.141.120] out: [13.0.0.1] out:
2016-08-02 23:53:54:526485: [root@10.87.141.120] out: [13.0.0.1] out: Requested: cd /etc/ceph && sudo ceph-deploy gatherkeys a5d01e03-storage
2016-08-02 23:53:54:526576: [root@10.87.141.120] out: [13.0.0.1] out: Executed: /bin/bash -l -c "cd /etc/ceph && sudo ceph-deploy gatherkeys a5d01e03-storage"
2016-08-02 23:53:54:526669: [root@10.87.141.120] out: [13.0.0.1] out:
2016-08-02 23:53:54:526760: [root@10.87.141.120] out: [13.0.0.1] out: Aborting.
2016-08-02 23:53:54:526864: [root@10.87.141.120] out: [13.0.0.1] out:
2016-08-02 23:53:54:791820: [root@10.87.141.120] out:
2016-08-02 23:53:54:792012: [root@10.87.141.120] out:
2016-08-02 23:53:54:792105: [root@10.87.141.120] out: Fatal error: run() received nonzero return code 1 while executing!
2016-08-02 23:53:54:792195: [root@10.87.141.120] out:
2016-08-02 23:53:54:792286: [root@10.87.141.120] out: Requested: sudo storage-fs-setup --storage-master 13.0.0.1 --storage-setup-mode reconfigure --storage-hostnames a5d01e03-storage a5d01e04-storage a5d01e05-storage a5d01e06-storage a5d01e07-storage --storage-compute-hostnames a5d01e06-storage a5d01e07-storage --storage-hosts 13.0.0.1 13.0.0.2 13.0.0.3 13.0.0.4 13.0.0.5 --storage-host-tokens n1keenA n1keenA n1keenA n1keenA n1keenA --storage-disk-config a5d01e07-storage:/dev/sdc a5d01e07-storage:/dev/sdb a5d01e06-storage:/dev/sdb a5d01e06-storage:/dev/sdc a5d01e06-storage:/dev/sdd a5d01e06-storage:/dev/sde a5d01e06-storage:/dev/sdg a5d01e06-storage:/dev/sdh --storage-ssd-disk-config none --storage-journal-config a5d01e07-storage:/dev/sdd a5d01e06-storage:/dev/sdf --storage-local-disk-config none --storage-local-ssd-disk-config none --storage-nfs-disk-config none --storage-directory-config none --storage-chassis-config none --storage-mon-hosts none --collector-hosts 5.0.0.1 5.0.0.2 5.0.0.3 --collector-host-tokens n1keenA n1keenA n1keenA --cfg-host 5.0.0.1 --cinder-vip 5.0.0.99 --config-hosts 5.0.0.2 5.0.0.3 --storage-os-hosts 13.0.0.2 13.0.0.3 --storage-os-host-tokens n1keenA n1keenA --cfg-vip none --storage-replica-size None --openstack-ip 5.0.0.1 --orig-hostnames a5d01e03 a5d01e04 a5d01e05 a5d01e06 a5d01e07 --service-dbpass c0ntrail123 --region-name RegionOne --ssd-cache-tier False --object-storage True --object-storage-pool volumes

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22809
Submitter: Jeya ganesh babu (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22809
Committed: http://github.org/Juniper/contrail-provisioning/commit/ab2522530403ae02a5494e744bcad1c1b14fe367
Submitter: Zuul
Branch: R3.1

commit ab2522530403ae02a5494e744bcad1c1b14fe367
Author: Jeya ganesh babu J <email address hidden>
Date: Wed Aug 3 11:27:46 2016 -0700

Mitaka ceph fix

Closes-Bug: #1609317
Jewel version of ceph requires /var/lib/ceph and /var/run/ceph
directories to be with ownership of ceph.
The issue is seen during a reconfigure.

Change-Id: Ia057411bf1e4304104e5f695435a7b0d0b52a08b

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23340
Submitter: Jeya ganesh babu (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23340
Committed: http://github.org/Juniper/contrail-provisioning/commit/86d4fb08b7ead57c1e687bc2d5feac1523d91877
Submitter: Zuul
Branch: master

commit 86d4fb08b7ead57c1e687bc2d5feac1523d91877
Author: Jeya ganesh babu J <email address hidden>
Date: Tue Aug 16 17:56:53 2016 -0700

Storage provision fix merge

Closes-Bug: #1609317
Closes-Bug: #1609973
Closes-Bug: #1609543
Closes-Bug: #1610330
Jewel version of ceph requires /var/lib/ceph and /var/run/ceph
directories to be with ownership of ceph.
The issue is seen during a reconfigure.
Fixed issue when restarting cinder services. Need to kill zombie
process.
Fixed livemigration permission issue.
Issue in the recent libvirt configuration. The listen
argument should be provided as env variable in init
script.
Object storage uses same port as horizon causing
horizon to error. Changed the default port for object
storage delivery to 9001.

Change-Id: I21c25e3fd253cadd6ea907ba47f6577ff34a5cef