primary-ceph-mon failed if non default hostname used

[Sergey Yezhkov]

I try to deploy Newton release with Ubuntu 16.04+UCA to 3 Controllers and 3 comp nodes; with Ceph for Nova, Glance, Cinder.

I use non default hostname for nodes. I change node-2 to tctl03, node-5 to tcomp03 ...

Expected results: Deployed OpenStack

Actual result: Deployment failed with message "Deployment has failed. All nodes are finished. Failed tasks: Task[primary-ceph-mon/2] Stopping the deployment process!"

In puppet logs I have this errors:
2017-01-17 14:18:12 ERR (/Stage[main]/Osnailyfacter::Ceph::Mon/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/unless) Check "/bin/true # comment to satisfy puppet syntax requirements\nset -x\nOLD_KEYRING=$(mktemp)\nceph --name 'mon.' --keyring '/var/lib/ceph/mon/ceph-tctl03/keyring' auth get client.bootstrap-osd -o $OLD_KEYRING || true\ndiff -N $OLD_KEYRING /etc/ceph/ceph.client.bootstrap-osd.keyring\nrv=$?\nrm $OLD_KEYRING\nexit $rv" exceeded timeout
2017-01-17 14:13:12 ERR (/Stage[main]/Osnailyfacter::Ceph::Mon/Ceph::Key[client.admin]/Exec[ceph-injectkey-client.admin]/unless) Check "/bin/true # comment to satisfy puppet syntax requirements\nset -x\nOLD_KEYRING=$(mktemp)\nceph --name 'mon.' --keyring '/var/lib/ceph/mon/ceph-tctl03/keyring' auth get client.admin -o $OLD_KEYRING || true\ndiff -N $OLD_KEYRING /etc/ceph/ceph.client.admin.keyring\nrv=$?\nrm $OLD_KEYRING\nexit $rv" exceeded timeout

Reproducibility:
Every time when I try to deploy

Description of the environment:
Fuel for OpenStack 10.0
In deployment task details I found:
"puppet_manifest": "/etc/puppet/modules/osnailyfacter/modular/ceph/mon.pp"
version 2.1.0
groups primary-controller

Additional information:
My suggestion that problem in mon.pp in lines
  if $primary_mon_hostname == $::hostname {
    $mon_initial_members = $primary_mon_hostname
    $mon_host = $primary_mon_ip
  } else {
    $mon_initial_members = $mon_hosts
    $mon_host = $mon_ips
  }

I add additional output to mon.pp and see that in my case $primary_mon_hostname="node-2" and $::hostname="tctl03".

[Luca Cervigni (cervigni)]

Hello,

Same problem here, custom hostname creates the same error.
Any workaround to fix this that is not using the default hostnames? Don't really want to call my controllers "node-X"

Deployment has failed. All nodes are finished. Failed tasks: Task[primary-ceph-mon/21] Stopping the deployment process!

2017-01-30 17:06:42 ERR (/Stage[main]/Osnailyfacter::Ceph::Mon/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/onlyif) Check "/bin/true # comment to satisfy puppet syntax requirements\nset -ex\nOLD_KEYRING=$(mktemp)\nceph --name 'mon.' --keyring '/var/lib/ceph/mon/ceph-controller-03/keyring' auth get client.bootstrap-osd -o $OLD_KEYRING || true\ndiff -N $OLD_KEYRING /etc/ceph/ceph.client.bootstrap-osd.keyring | grep '>'\nrm $OLD_KEYRING" exceeded timeout
2017-01-30 17:01:42 ERR (/Stage[main]/Osnailyfacter::Ceph::Mon/Ceph::Key[client.admin]/Exec[ceph-injectkey-client.admin]/onlyif) Check "/bin/true # comment to satisfy puppet syntax requirements\nset -ex\nOLD_KEYRING=$(mktemp)\nceph --name 'mon.' --keyring '/var/lib/ceph/mon/ceph-controller-03/keyring' auth get client.admin -o $OLD_KEYRING || true\ndiff -N $OLD_KEYRING /etc/ceph/ceph.client.admin.keyring | grep '>'\nrm $OLD_KEYRING" exceeded timeout

[Luca Cervigni (cervigni)]

Do you think is possible to fix this before the 11?
It is kind of a problem having this issue if CEPH in use

[Dmitry Belyaninov (dbelyaninov)]

https://product-ci.infra.mirantis.net/job/10.0.acceptance.ubuntu.ha_vlan_group_4/2/testReport/(root)/ceph_rados_gw_no_storage_volumes/ceph_rados_gw_no_storage_volumes/

[Luca Cervigni (cervigni)]

Dmitry, is that link suppose to be public?
I get page not found

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/444880

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/444935

[Luca Cervigni (cervigni)]

Hello,
Is the fix commited for stable/ocata as well?

If not, can you please post the workaround to solve this issue?

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/454168
Reason: This repo is retired now, no further work will get merged.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/444880
Reason: This repo is retired now, no further work will get merged.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/newton
Review: https://review.opendev.org/444935
Reason: This repo is retired now, no further work will get merged.