Error in generating ceph key prevents deployment of new environment
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Invalid
|
Critical
|
Stanislaw Bogatkin |
Bug Description
Detailed bug description:
Hello,
I am using fuel-community-9.0 and have been unable to create a working environment. Each time I deploy a new environment, I get an error when generating ceph master keys.
I looked at the /var/log/
Here are the logs:
2016-11-19 01:32:00 DEBUG [1146] Node master(
2016-11-19 01:32:01 DEBUG [1146] {"nodes"
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Process node: Node[91]
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Process node: Node[90]
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Process node: Node[93]
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Process node: Node[92]
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Process node: Node[86]
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Process node: Node[94]
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Process node: Node[85]
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Process node: Node[virtual_
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Start processing all nodes
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Process node: Node[87]
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Process node: Node[88]
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Process node: Node[89]
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Process node: Node[master]
2016-11-19 01:32:01 DEBUG [1146] Node[master]: Node master: task generate_keys_ceph, task status running
2016-11-19 01:32:01 WARNING [1146] Puppet agent master didn't respond within the allotted time
2016-11-19 01:32:01 DEBUG [1146] Task time summary: generate_keys_ceph with status failed on node master took 00:03:00
2016-11-19 01:32:01 DEBUG [1146] Node[master]: Decreasing node concurrency to: 0
2016-11-19 01:32:01 INFO [1146] Task[generate_
2016-11-19 01:32:01 DEBUG [1146] Node[master]: Increasing node concurrency to: 1
2016-11-19 01:32:01 INFO [1146] Casting message to Nailgun:
{"method"
"args"=>
{"task_
"nodes"=>
[{"
"
"
"
"
"custom"=>
{:time=>
:idling=>0,
2016-11-19 01:32:01 DEBUG [1146] Cluster[]: Process node: Node[91]
Steps to reproduce:
Create a new environment from web UI, configure network, settings, add nodes, verify networking connectivity, deploy.
We have:
1 controller,cinder node
3 ceph osd nodes
1 compute node
1 mongo ceilometer node
4 base-os nodes.
Expected results:
a working openstack environment.
Actual result:
All nodes end up in error state with problem pointing to error in generating_
Reproducibility:
Everytime.
Workaround:
None.
Impact:
Can not get a openstack system up and running.
Description of the environment:
Operation system: fuel master node: CentOS7
Versions of components: fuel-community-9.0
Reference architecture: amd64
Network model: neutron with tunneling segmentation
Related projects installed: n/a
Additional information:
N/A.
description: | updated |
Changed in fuel: | |
importance: | Undecided → Critical |
milestone: | none → 9.2 |
assignee: | nobody → Stanislaw Bogatkin (sbogatkin) |
Changed in fuel: | |
milestone: | 9.2 → 9.3 |
Changed in fuel: | |
status: | Incomplete → Invalid |
Looking at the logs a bit more. It seems like a race condition. The astute logs indicate that the deployment errors out 4 minutes before puppet finally create the ceph key:
astute.log: sync_node] : Found failed tasks: post_deployment _start, deploy_start, pre_deployment_end, post_deployment _end, deploy_end keys_ceph/ master] , Task[generate_ keys/master] Stopping the deployment process!
2016-11-20 08:27:53 DEBUG [1169] Graph[virtual_
2016-11-20 08:27:53 INFO [1169] Cluster[]: All nodes are finished. Failed tasks: Task[generate_
puppet.log: main]/Main/ Exec[generate_ keys_ceph_ shell]/ returns (notice): Generating public/private rsa key pair. main]/Main/ Exec[generate_ keys_ceph_ shell]/ returns (notice): Your identification has been saved in /var/lib/ fuel/keys/ /21/ceph/ ceph. main]/Main/ Exec[generate_ keys_ceph_ shell]/ returns (notice): Your public key has been saved in /var/lib/ fuel/keys/ /21/ceph/ ceph.pub.
2016-11-20 08:31:41 +0000 Scope(Class[main]) (notice): MODULAR: generate_keys_ceph
2016-11-20 08:31:41 +0000 Puppet (notice): Compiled catalog for fuel.domain.tld in environment production in 0.05 seconds
2016-11-20 08:31:42 +0000 /Stage[
2016-11-20 08:31:42 +0000 /Stage[
2016-11-20 08:31:42 +0000 /Stage[