Mirantis OpenStack

keystoneclient session "Service Unavailable (HTTP 503)"

Bug #1621064 reported by Andrew Kalach on 2016-09-07

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Mirantis OpenStack	Fix Released	High	Andrew Kalach	Mirantis OpenStack 9.2

Bug Description

Description
===========

After MOS9.1 was deployed the Rally test suite run showed many tasks with common error "Service Unavailable (HTTP 503)" in Python keystoneclient session.py module:

2016-09-02 10:21:37.295 21640 DEBUG keystoneclient.session [-] REQ: curl -g -i -X GET http://192.168.0.3:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: python-keystoneclient" _http_log_request /opt/stack/.venv/lib/python2.7/site-packages/keystoneclient/session.py:206
2016-09-02 10:21:37.310 21640 DEBUG keystoneclient.session [-] RESP: [503] Cache-Control: no-cache Connection: close Content-Type: text/html
RESP BODY: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

_http_log_response /opt/stack/.venv/lib/python2.7/site-packages/keystoneclient/session.py:231

First appearance of the error was found in "NeutronNetworks.create_and_delete_port" testcase
(testcase finally finished with "504 Gateway Time-out" error)

CinderVolumes.create_and_list_snapshots
CinderVolumes.create_and_list_volume_backups
CinderVolumes.create_from_volume_and_delete_volume
CinderVolumes.create_snapshot_and_attach_volume
CinderVolumes.modify_volume_metadata
HeatStacks.create_check_delete_stack
HeatStacks.create_stack_and_show_output
HeatStacks.create_update_delete_stack
KeystoneBasic.create_delete_user
MuranoEnvironments.list_environments
NeutronNetworks.create_and_update_subnets
NovaServers.boot_and_associate_floating_ip:
NovaServers.boot_and_live_migrate_server
NovaServers.snapshot_server

Steps to reproduce
==================
1. Deploy MOS9.1
2. Install Rally on masternode
3. Run Rally test suite

Actual result
=============
Many Rally tasks failed with he following error logs:

2016-09-02 10:38:46.609 22815 DEBUG keystoneclient.session [-] Request returned failure status: 503 request /opt/stack/.venv/lib/python2.7/site-packages/keystoneclient/session.py:418
2016-09-02 10:38:46.609 22815 WARNING keystoneclient.auth.identity.generic.base [-] Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.
2016-09-02 10:38:46.610 22815 DEBUG keystoneclient.auth.identity.v2 [-] Making authentication request to http://192.168.0.3:5000/v2.0/tokens get_auth_ref /opt/stack/.venv/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:87
2016-09-02 10:38:46.612 22815 DEBUG keystoneclient.session [-] Request returned failure status: 503 request /opt/stack/.venv/lib/python2.7/site-packages/keystoneclient/session.py:418
Task config is invalid: `Service Unavailable (HTTP 503)`

Environment
===========
Scale 200-node lab (ENV-10):
  3 controllers
  198 compute nodes
  20 ceph nodes
  2 StackLight nodes (Grafana + Kibana)
  DVR enabled
Initial image RC2 build (updated to 9.1):
  fuel-9.0-mos-495-2016-06-16_18-18-00.iso

Diagnostic snapshot:
http://mos-scale-share.mirantis.com/fuelds-160906-e503-anorml.tar.gz

See original description

Tags:

Revision history for this message

Andrew Kalach (akndex) wrote on 2016-09-07:

rally-logs.zip Edit (1.3 MiB, application/zip)

description:

updated

Andrew Kalach (akndex) on 2016-09-07

description:

updated

Michael Semenov (msemenov) on 2016-09-07

Changed in mos:
importance:	Medium → High
milestone:	none → 9.1
assignee:	nobody → MOS Keystone (mos-keystone)

Denis Meltsaykin (dmeltsaykin) on 2016-09-09

Changed in mos:
status:	New → Confirmed
tags:	added: area-keystone

Alexander Makarov (amakarov) on 2016-09-14

Changed in mos:
milestone:	9.1 → 9.2
tags:	added: area-scale

Revision history for this message

Andrew Kalach (akndex) wrote on 2016-09-15:

Excerpt logs from diag snapshot for Keystone and HAProxy:

http://mos-scale-share.mirantis.com/keystone-logs-160906-10-e503.tar.gz
http://mos-scale-share.mirantis.com/haproxy-logs-160906-10-e503.tar.gz

Revision history for this message

Alexander Makarov (amakarov) wrote on 2016-09-15:

A lot of errors in keystone apache log:

2016-09-02T10:34:53.873657+00:00 err: [Fri Sep 02 10:34:46.863439 2016] [wsgi:error] [pid 6972:tid 140385670084352] (11)Resource temporarily unavailable: [client 192.168.0.3:41492] mod_wsgi (pid=6972): Unable to connect to WSGI daemon process 'keystone_main' on '/var/run/apache2/wsgi.6942.0.2.sock' after multiple attempts as listener backlog limit was exceeded.

Indicates keystone doesn't allow such load - need to increase number of workers.

Changed in mos:
assignee:	MOS Keystone (mos-keystone) → Andrew Kalach (akndex)
status:	Confirmed → Triaged

Michael Semenov (msemenov) on 2016-12-07

tags:

added: blocker-for-qa

Andrew Kalach (akndex) on 2017-01-17

Changed in mos:
status:	Triaged → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

rally-logs.zip Edit

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.