Fuel for OpenStack

Keystone issue when scaling controllers

Bug #1443892 reported by Anastasia Palkina on 2015-04-14

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Invalid	Critical	Bogdan Dobrelya	Fuel for OpenStack 6.1
	6.0.x	Invalid	Critical	Bogdan Dobrelya	Fuel for OpenStack 6.0-updates

Bug Description

"build_id": "2015-04-10_22-54-31",
"ostf_sha": "c2a76a60ec4ebbd78e508216c2e12787bf25e423",
"build_number": "304",
"release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-04-10_22-54-31", "ostf_sha": "c2a76a60ec4ebbd78e508216c2e12787bf25e423", "build_number": "304", "api": "1.0", "nailgun_sha": "69547a71abb4696df7e6f44b1f7864b0535f2df7", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c", "astute_sha": "d96a80b63198a578b2c159edbd76048819039eb0", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "8daac234aea6ac0a98f27871deec039f74f6fdab", "fuellib_sha": "867028fe78837dc2e4635a2cbb976782856964d0"}}}, "auth_required": true,
"api": "1.0",
"nailgun_sha": "69547a71abb4696df7e6f44b1f7864b0535f2df7",
"openstack_version": "2014.2-6.1",
"production": "docker",
"python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c",
"astute_sha": "d96a80b63198a578b2c159edbd76048819039eb0",
"feature_groups": ["mirantis"],
"release": "6.1",
"fuelmain_sha": "8daac234aea6ac0a98f27871deec039f74f6fdab",
"fuellib_sha": "867028fe78837dc2e4635a2cbb976782856964d0"

1. Create new environment (CentOS)
2. Choose nova-network, vlan
3. Add 1 controller, 1 compute
4. Start deployment. It was successful
5. Add 2 controllers
6. Redeploy environment. It was successful
7. Start OSTF tests. It was succesful
8. But there is error on primary controller (node-7):

2015-04-14 09:55:23 ERR

(/Stage[main]/Main/Nova_floating_range[172.16.0.128-172.16.0.254]) Could not evaluate: Oops - not sure what happened: 757: unexpected token at '<html><body><h1>503 Service Unavailable</h1>

Also there is error for the same test case on Ubuntu. Error on node-5

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaMTZlV2ZyNEZCYmc/view?usp=sharing

Vladimir Kuklin (vkuklin) on 2015-04-14

Changed in fuel:
status:	New → Confirmed

Oleksiy Molchanov (omolchanov) on 2015-04-21

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Oleksiy Molchanov (omolchanov)

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2015-04-21:

This Means that either keystone or nova-api were not available. This could have happened to connectivity issues. We need a reproducer here.

Changed in fuel:
status:	Confirmed → Incomplete

Revision history for this message

Anastasia Palkina (apalkina) wrote on 2015-04-21:

Reproduce this test case on ISO #329

"build_id": "2015-04-20_22-54-31", "ostf_sha": "df8db1f48f03b18126ce5ec65317a1eb83a5a95f", "build_number": "329", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-04-20_22-54-31", "ostf_sha": "df8db1f48f03b18126ce5ec65317a1eb83a5a95f", "build_number": "329", "api": "1.0", "nailgun_sha": "69fca3bada96a48ab19b969621e5128390b7052e", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "b4f1ddffd5263489090b65e662173e9e11aafd94", "astute_sha": "bf1751a4fe0d912325e3b4af629126a59c1b2b51", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0c0e299dcc0276bb8262256148c6ee975857ee86", "fuellib_sha": "b878f1ae8e1d4c6f2f463e810b1f8014df5b22a5"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "69fca3bada96a48ab19b969621e5128390b7052e", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "b4f1ddffd5263489090b65e662173e9e11aafd94", "astute_sha": "bf1751a4fe0d912325e3b4af629126a59c1b2b51", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0c0e299dcc0276bb8262256148c6ee975857ee86", "fuellib_sha": "b878f1ae8e1d4c6f2f463e810b1f8014df5b22a5"

The same error on primary controller (node-6):

2015-04-21 14:55:54 ERR

(/Stage[main]/Main/Nova_floating_range[172.16.0.128-172.16.0.254]) Could not evaluate: Oops - not sure what happened: 757: unexpected token at '<html><body><h1>503 Service Unavailable</h1>

Changed in fuel:
status:	Incomplete → Confirmed

Revision history for this message

Anastasia Palkina (apalkina) wrote on 2015-04-21:

Logs (for ISO #329) are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaWDRwVHJ6QUhVMk0/view?usp=sharing

Revision history for this message

Oleksiy Molchanov (omolchanov) wrote on 2015-04-23:

Tried to deploy in my env using different OS, but didn't manage to reproduce.

Anastasia, please provide me with the env during deployment on your side.

Changed in fuel:
status:	Confirmed → In Progress

Vladimir Kuklin (vkuklin) on 2015-04-24

Changed in fuel:
assignee:	Oleksiy Molchanov (omolchanov) → Bogdan Dobrelya (bogdando)

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2015-04-27:

This issue reproduced on our CI in ha scalability test here - http://172.18.66.221:8080/view/6.1_swarm/job/6.1.system_test.centos.thread_4/106/testReport/junit/%28root%29/ha_flat_scalability/ha_flat_scalability/

Logs are attached

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2015-04-27:

fail_error_ha_flat_scalability-2015_04_27__08_09_28.tar.xz Edit (76.4 MiB, application/octet-stream)

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-04-27:

Note, astute.yaml shows there are 5 controllers, is it correct?
- user_node_name: slave-01_controller
  fqdn: node-1.test.domain.local
  role: primary-controller
  name: node-1
- user_node_name: slave-02_controller
  fqdn: node-2.test.domain.local
  role: controller
  name: node-2
- user_node_name: slave-04_controller
  fqdn: node-3.test.domain.local
  role: controller
  name: node-3
- user_node_name: slave-05_controller
  fqdn: node-4.test.domain.local
  role: controller
  name: node-4
- user_node_name: slave-03_controller
  fqdn: node-5.test.domain.local
  role: controller
  name: node-5

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-04-27:

but commands in snapshot showing node-3, node-4 aren't controllers! Looks like we have some inconsistency

[node-1.test.domain.local] out: Online: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
[node-3.test.domain.local] out: /bin/bash: pcs: command not found

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-04-27:

The keystone config expects there are 5 controllers accroding to the deploy info
rabbit_hosts=10.109.17.4:5673, 10.109.17.6:5673, 10.109.17.7:5673, 10.109.17.8:5673, 10.109.17.5:5673

but in fact, we have only 3. So this should be the RC

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-04-27:

#10

http://paste.openstack.org/show/ltt7Lh3ewHsW30Wtk6Ca/

summary:	- (/Stage[main]/Main/Nova_floating_range[172.16.0.128-172.16.0.254]) Could - not evaluate: Oops - not sure what happened: 757: unexpected token at - '<html><body><h1>503 Service Unavailable</h1> + Orchestration assigns wrong tasks for nodes when adding new controllers
Changed in fuel:
importance:	High → Critical
assignee:	Bogdan Dobrelya (bogdando) → Fuel Library Team (fuel-library)

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-04-27: Re: Orchestration assigns wrong tasks for nodes when adding new controllers

#11

My bad, the pupet logs from node-3, node-4 went no further when hiera step.

Bogdan Dobrelya (bogdando) on 2015-04-27

summary:

- Orchestration assigns wrong tasks for nodes when adding new controllers
+ Keystone issue when scaling controllers

Bogdan Dobrelya (bogdando) on 2015-04-27

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Bogdan Dobrelya (bogdando)

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2015-04-27:

#12

The same test also failed on Ubuntu thread but error is different - http://paste.openstack.org/show/208540/

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2015-04-27:

#13

fail_error_ha_flat_scalability-2015_04_27__06_55_57.tar.xz Edit (63.9 MiB, application/octet-stream)

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2015-04-27:

#14

According to the logs, the issue is with heat-domain id setter:

Mon Apr 27 07:59:45 +0000 2015 /Stage[main]/Heat::Keystone::Domain/Heat_domain_id_setter[heat_domain_id]/ensure (err): change from absent to present failed: Received error response from Keystone server at http://10.109.17.2:35357/v3/domains: Unauthorized
Mon Apr 27 07:59:45 +0000 2015 /Stage[main]/Heat::Keystone::Domain/Heat_domain_id_setter[heat_domain_id] (debug): Nothing to manage: no ensure and the resource doesn't exist
Mon Apr 27 07:59:45 +0000 2015 /Stage[main]/Heat::Keystone::Domain/Heat_domain_id_setter[heat_domain_id] (info): Evaluated in 1.14 seconds

There is another issue with nova floating range installation, but it is related to another bug and is in progress now

Changed in fuel:
status:	In Progress → Invalid

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2015-04-27:

#15

So this bug is a mess. I will create another one or two with issue description.

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2015-04-27:

#16

Here are new ones

https://bugs.launchpad.net/fuel/+bug/1449209
https://bugs.launchpad.net/fuel/+bug/1449219

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-04-28:

#17

@Vladimir, I can confirm the comment https://bugs.launchpad.net/fuel/+bug/1443892/comments/12 as I reproduced this case with "(/Stage[main]/Main/L23network::L3::Ifconfig[br-storage]/L3_ifconfig[br-storage]) Could not evaluate: Execution of '/bin/ip addr add 10.109.4.4/24 dev br-storage' returned 2: RTNETLINK answers: File exists" as well on Ubuntu lab built from ISO #354

It looks like another issue

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.