Juniper Openstack

Vcenter-as-compute: fab setup_all fails due to Keystone authorization fails

Bug #1689640 reported by Sarath
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.2
Fix Committed
Critical
amudhar
Juniper Openstack r3.2.5.0
R4.0
Fix Committed
Critical
amudhar
Juniper Openstack r4.0.0.0-fcs "r4.0.0.0"
Trunk
Fix Committed
Critical
amudhar
Juniper Openstack r4.1.0.0-fcs "r4.1"

Bug Description

This seen on Sanity setup.
On this HA setup, 3 controller nodes all running keystone server and issue seen on only one node

On restarting ha_proxy, it just start working fine

>> snipped of fab-setup-all failure,

::
::
2017-05-09 12:21:27:661903: Warning: sudo() received nonzero return code 1 while executing 'source /etc/contrail/openstackrc; keystone tenant-list'!
2017-05-09 12:21:27:661903:
2017-05-09 12:21:27:661903: 2017-05-09 12:21:27:661807: Traceback (most recent call last):
2017-05-09 12:21:37:669178: File "/usr/local/lib/python2.7/dist-packages/fabric/main.py", line 743, in main
2017-05-09 12:21:37:669275: *args, **kwargs
2017-05-09 12:21:37:670859: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 375, in execute
2017-05-09 12:21:37:670918: multiprocessing
2017-05-09 12:21:37:671301: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 271, in _execute
2017-05-09 12:21:37:671348: return task.run(*args, **kwargs)
2017-05-09 12:21:37:671535: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run
2017-05-09 12:21:37:671594: return self.wrapped(*args, **kwargs)
2017-05-09 12:21:37:671736: File "/usr/local/lib/python2.7/dist-packages/fabric/decorators.py", line 53, in inner_decorator
2017-05-09 12:21:37:671819: return func(*args, **kwargs)
2017-05-09 12:21:37:672063: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 168, in __call__
2017-05-09 12:21:37:672131: return self.run(*args, **kwargs)
2017-05-09 12:21:37:672286: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run
2017-05-09 12:21:37:672346: return self.wrapped(*args, **kwargs)
2017-05-09 12:21:37:672496: File "/root/fabric-utils/fabfile/tasks/provision.py", line 2833, in setup_all
2017-05-09 12:21:37:672556: execute('setup_orchestrator')
2017-05-09 12:21:37:673762: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 375, in execute
2017-05-09 12:21:37:673808: multiprocessing
2017-05-09 12:21:37:673988: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 271, in _execute
2017-05-09 12:21:37:674042: return task.run(*args, **kwargs)
2017-05-09 12:21:37:674208: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run
2017-05-09 12:21:37:674263: return self.wrapped(*args, **kwargs)
2017-05-09 12:21:37:674397: File "/usr/local/lib/python2.7/dist-packages/fabric/decorators.py", line 53, in inner_decorator
2017-05-09 12:21:37:674452: return func(*args, **kwargs)
2017-05-09 12:21:37:674567: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 168, in __call__
2017-05-09 12:21:37:674621: return self.run(*args, **kwargs)
2017-05-09 12:21:37:674751: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run
2017-05-09 12:21:37:674804: return self.wrapped(*args, **kwargs)
2017-05-09 12:21:37:674933: File "/root/fabric-utils/fabfile/tasks/provision.py", line 2686, in setup_orchestrator
2017-05-09 12:21:37:674986: execute('verify_openstack')
2017-05-09 12:21:37:675703: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 375, in execute
2017-05-09 12:21:37:675758: multiprocessing
2017-05-09 12:21:37:675932: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 271, in _execute
2017-05-09 12:21:37:675985: return task.run(*args, **kwargs)
2017-05-09 12:21:37:676136: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run
2017-05-09 12:21:37:676189: return self.wrapped(*args, **kwargs)
2017-05-09 12:21:37:676322: File "/usr/local/lib/python2.7/dist-packages/fabric/decorators.py", line 53, in inner_decorator
2017-05-09 12:21:37:676377: return func(*args, **kwargs)
2017-05-09 12:21:37:676534: File "/root/fabric-utils/fabfile/tasks/verify.py", line 66, in verify_openstack
2017-05-09 12:21:37:676633: raise OpenStackSetupError(output)
2017-05-09 12:21:37:677017: fabfile.tasks.verify.OpenStackSetupError: Authorization Failed: Unable to establish connection to http://10.87.36.10:5000/v2.0/tokens

>> on problem node, 10.87.36.10

2017-05-09 12:19:53:084516: [root@10.87.36.10] sudo: service keystone status
2017-05-09 12:19:53:084800: [root@10.87.36.10] out: keystone RUNNING pid 14118, uptime 0:11:16

2017-05-09 12:19:53:380078: [root@10.87.36.10] sudo: source /etc/contrail/openstackrc; keystone tenant-list
2017-05-09 12:19:53:380313: [root@10.87.36.10] out: Authorization Failed: Unable to establish connection to http://10.87.36.10:5000/v2.0/tokens

>> on other 2 nodes, .11 & .12

root@5a10s30:~# service keystone status
keystone RUNNING pid 31429, uptime 0:11:48
root@5a10s30:~# source /etc/contrail/openstackrc
root@5a10s30:~#
root@5a10s30:~# keystone token-get
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2017-05-09T20:26:22Z |
| id | 6c44be84a0534bd79025de52e48c99b1 |
| tenant_id | 1b6fd7d5004a4dbdbd4ccb480a14d078 |
| user_id | fb9e4067807b48048ddc50a9ec3da53d |
+-----------+----------------------------------+
r

Jeba Paulaiyan (jebap)
tags: added: blocker
Revision history for this message
amudhar (amudha) wrote :

Please provide testbed.

Thanks,
Amudha

Revision history for this message
Sarath (nsarath) wrote :

_____________________________________________
From: Sarathbabu Narasimhan
Sent: Thursday, May 11, 2017 12:28 PM
To: Amudha R <email address hidden>
Cc: Jeba Paulaiyan <email address hidden>; Rudra Rugge <email address hidden>; Sarathbabu Narasimhan <email address hidden>
Subject: Bug #1689640 : Vcenter-as-compute: fab setup_all fails due to Keystone authorization fails

Hi Amudha,

As per your request for problem setup, have recreated the issue and please find the setup @
10.87.36.10

Now it is in fab setup_all failed state but not actually showing keystone authorization failure but
other keystone issue which not even bringingup keystone

Thanks
*Sarath

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/31676
Submitter: amudha ramachandran (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/31677
Submitter: amudha ramachandran (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/31677
Committed: http://github.com/Juniper/contrail-fabric-utils/commit/67931ba6b587bb2d2ee5eb08a6a4ec0c47214f2a
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 67931ba6b587bb2d2ee5eb08a6a4ec0c47214f2a
Author: Amudha <email address hidden>
Date: Tue May 16 16:55:07 2017 -0700

Fix haproxy restart failure

In HA setup, haproxy fails to restart as openstack services are listening
on the haproxy designated ports.
Fix by stopping the openstack services before restarting haproxy.

Change-Id: Id074ac29da924a898e64f660313eb5b7e26fa1a8
Closes-Bug: #1689640

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/31676
Committed: http://github.com/Juniper/contrail-fabric-utils/commit/eb666d9f9418215ebb74e08aa1b3307e61d3c88d
Submitter: Zuul (<email address hidden>)
Branch: master

commit eb666d9f9418215ebb74e08aa1b3307e61d3c88d
Author: Amudha <email address hidden>
Date: Tue May 16 16:55:07 2017 -0700

Fix haproxy restart failure

In HA setup, haproxy fails to restart as openstack services are listening
on the haproxy designated ports.
Fix by stopping the openstack services before restarting haproxy.

Change-Id: Id074ac29da924a898e64f660313eb5b7e26fa1a8
Closes-Bug: #1689640

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/34172
Submitter: amudha ramachandran (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/34172
Committed: http://github.com/Juniper/contrail-fabric-utils/commit/e862c7db46b2a726fa931d530a6166b0af28787a
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit e862c7db46b2a726fa931d530a6166b0af28787a
Author: Amudha <email address hidden>
Date: Tue May 16 16:55:07 2017 -0700

Fix haproxy restart failure

In HA setup, haproxy fails to restart as openstack services are listening
on the haproxy designated ports.
Fix by stopping the openstack services before restarting haproxy.

Change-Id: Id074ac29da924a898e64f660313eb5b7e26fa1a8
Closes-Bug: #1689640
(cherry picked from commit eb666d9f9418215ebb74e08aa1b3307e61d3c88d)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.