Fuel for OpenStack

keystone of master node is unreachable during applying puppet manifests during fuel installation

Bug #1430787 reported by Leontii Istomin on 2015-03-11

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Released	High	Matthew Mosesohn	Fuel for OpenStack 6.1

Bug Description

[root@fuel ~]# cat /etc/fuel/release_versions/2014.2-6.1.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "182"
  build_id: "2015-03-11_02-09-40"
  nailgun_sha: "058d1adef486c116ab8c79379ea6f925db039177"
  python-fuelclient_sha: "8a292dbdfc3afc1994fd8a81a28903f9a5cca351"
  astute_sha: "93de472789d9fc351d915e401892c9f792c14ca2"
  fuellib_sha: "acd7dfb5f93ee0719464d07faf5883ee804a7205"
  ostf_sha: "8df5f2fcdae3bc9ea7d700ffd64db820baf51914"
  fuelmain_sha: "0f588ec9125cc1f4dd24a07d3bc6903c97b84d27"

deployment was finished with error:
...
ESC[0;36mDebug: Exec[sync_deployment_tasks](provider=posix): Executing 'fuel rel --sync-deployment-tasks --dir /etc/puppet/'ESC[0m
ESC[0;36mDebug: Executing 'fuel rel --sync-deployment-tasks --dir /etc/puppet/'ESC[0m
ESC[0;36mDebug: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: Sleeping for 10.0 seconds between triesESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the o
ngoing version of python-fuelclient.ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: Traceback (most recent call last):ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/bin/fuel", line 10, in <module>ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: sys.exit(main())ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 98, in wrapperESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: return func(*args, **kwargs)ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 226, in mainESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: parser.parse()ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 102, in parseESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: actions[parsed_params.action].action_func(parsed_params)ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/fuelclient/cli/actions/base.py", line 61, in action_funcESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: method(params)ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/fuelclient/cli/actions/base.py", line 106, in wrapped_fESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: return f(self, params)ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/fuelclient/cli/actions/release.py", line 145, in sync_deployment_tasksESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: all_rels = Release.get_all_data()ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/fuelclient/objects/base.py", line 64, in get_all_dataESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: return cls.connection.get_request(cls.class_api_path)ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 98, in wrapperESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: return func(*args, **kwargs)ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 146, in get_requestESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: headers = {'x-auth-token': self.auth_token}ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 57, in auth_tokenESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: if not self.keystone_client.auth_token:ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 76, in keystone_clientESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: self.initialize_keystone_client()ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 90, in initialize_keystone_clientESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: tenant_name=self.tenant)ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 147, in __init__ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: self.authenticate()ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/keystoneclient/utils.py", line 318, in innerESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: return func(*args, **kwargs)ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 425, in authenticateESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: resp = self.get_raw_token_from_identity_service(**kwargs)ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 190, in get_raw_token_from_identity_serviceESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: "%s" % e)ESC[0m
ESC[mNotice: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: keystoneclient.openstack.common.apiclient.exceptions.AuthorizationFailure: Authorization Failed: Bad Gateway (HTTP 502)ESC[0m
ESC[1;31mError: fuel rel --sync-deployment-tasks --dir /etc/puppet/ returned 1 instead of one of [0]ESC[0m
ESC[1;31mError: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: change from notrun to 0 failed: fuel rel --sync-deployment-tasks --dir /etc/puppet/ returned 1 instead of one of [0]ESC[0m
...
ERROR: Fuel node deployment FAILED! Check /var/log/puppet/bootstrap_admin_node.log for details

I can't login via UI

I can't execute any fuel command:
[root@fuel ~]# fuel --debug --fuel-version
DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient.
Traceback (most recent call last):
  File "/usr/bin/fuel", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 98, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 226, in main
    parser.parse()
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 91, in parse
    parsed_params, _ = self.parser.parse_known_args(self.args[1:])
  File "/usr/lib/python2.6/site-packages/argparse.py", line 1735, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python2.6/site-packages/argparse.py", line 1941, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/usr/lib/python2.6/site-packages/argparse.py", line 1881, in consume_optional
    take_action(action, args, option_string)
  File "/usr/lib/python2.6/site-packages/argparse.py", line 1809, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/arguments.py", line 101, in __call__
    parser.exit(message=APIClient.get_fuel_version())
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 98, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 178, in get_fuel_version
    self.get_request("version"),
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 98, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 146, in get_request
    headers = {'x-auth-token': self.auth_token}
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 57, in auth_token
    if not self.keystone_client.auth_token:
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 76, in keystone_client
    self.initialize_keystone_client()
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 90, in initialize_keystone_client
    tenant_name=self.tenant)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 147, in __init__
    self.authenticate()
  File "/usr/lib/python2.6/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 425, in authenticate
    resp = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 190, in get_raw_token_from_identity_service
    "%s" % e)
keystoneclient.openstack.common.apiclient.exceptions.AuthorizationFailure: Authorization Failed: Bad Gateway (HTTP 502)

Because it still isn't installed:
[root@fuel ~]# dockerctl shell keystone
[root@7b06ae75adc5 ~]# ps aux | grep keystone
root 10 27.4 0.9 322968 77912 ? Sl 12:55 0:03 /usr/bin/ruby /usr/bin/puppet apply --detailed-exitcodes -v /etc/puppet/modules/nailgun/examples/keystone-only.pp
root 425 0.0 0.0 103240 848 ? S+ 12:55 0:00 grep keystone

/var directory is coming

Tags:

Revision history for this message

Leontii Istomin (listomin) wrote on 2015-03-11:

var_log.tar.gz Edit (11.3 MiB, application/x-tar)

Bogdan Dobrelya (bogdando) on 2015-03-12

Changed in fuel:
milestone:	none → 6.1
importance:	Undecided → High
assignee:	nobody → Fuel Library Team (fuel-library)

Revision history for this message

Matthew Mosesohn (raytrac3r) wrote on 2015-03-12:

This is a duplicate of https://bugs.launchpad.net/fuel/+bug/1430348
It was fixed by https://review.openstack.org/#/c/163327/ which was merged after ISO #182 was created. Can you reproduce on the latest stable ISO?

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Matthew Mosesohn (raytrac3r)
status:	New → Incomplete

Revision history for this message

Dina Belova (dbelova) wrote on 2015-03-17:

we did not face this issue in 192 ISO, it looks like it was fixed

Changed in fuel:
status:	Incomplete → Fix Committed

Revision history for this message

Dina Belova (dbelova) wrote on 2015-05-07:

Verified on the scale lab on #233 ISO (6.1)

Changed in fuel:
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

var_log.tar.gz Edit

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.