Juniper Openstack

Vcenter-as-compute: After provisioning HA-setup, one of node glance-registry service not bound with errors

  1. Series trunk
  2. Bug #1663438
Bug #1663438 reported by Sarath
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.2
Won't Fix
High
Sarath
R4.0
Incomplete
High
Sarath
R4.1
Incomplete
High
Sarath
Trunk
Incomplete
High
Sarath

Bug Description

This seen on Vcenter-compute setup and hence failing multiples of Sanity testcases on R3.2

Debugged with Senthil and please find below the relevant,

On 3node HA, host-1 & host-3 are okay and issue seen on host-2

>>> host-1 okay

root@a6s32:~/fabric-utils# cat /var/log/contrail/glance-registry-supervisor-stdout.log | more
2017-02-08 15:01:46.431 25144 INFO glance.common.wsgi [-] Starting 4 workers
2017-02-08 15:01:46.447 25144 INFO glance.common.wsgi [-] Started child 25492
2017-02-08 15:01:46.450 25492 INFO eventlet.wsgi.server [-] (25492) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:01:46.450 25144 INFO glance.common.wsgi [-] Started child 25493
2017-02-08 15:01:46.457 25493 INFO eventlet.wsgi.server [-] (25493) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:01:46.479 25494 INFO eventlet.wsgi.server [-] (25494) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:01:46.481 25144 INFO glance.common.wsgi [-] Started child 25494
2017-02-08 15:01:46.484 25144 INFO glance.common.wsgi [-] Started child 25496
2017-02-08 15:01:46.490 25496 INFO eventlet.wsgi.server [-] (25496) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:02:45.154 29311 INFO glance.common.wsgi [-] Starting 4 workers
2017-02-08 15:02:45.171 29311 INFO glance.common.wsgi [-] Started child 29623
2017-02-08 15:02:45.182 29311 INFO glance.common.wsgi [-] Started child 29624
2017-02-08 15:02:45.186 29623 INFO eventlet.wsgi.server [-] (29623) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:02:45.191 29311 INFO glance.common.wsgi [-] Started child 29625
2017-02-08 15:02:45.198 29625 INFO eventlet.wsgi.server [-] (29625) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:02:45.199 29624 INFO eventlet.wsgi.server [-] (29624) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:02:45.217 29626 INFO eventlet.wsgi.server [-] (29626) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:02:45.194 29311 INFO glance.common.wsgi [-] Started child 29626
2017-02-08 15:04:09.253 4085 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admin
_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:04:09.254 4085 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admin
 endpoint
2017-02-08 15:04:09.256 4085 INFO glance.common.wsgi [-] Starting 4 workers
2017-02-08 15:04:09.259 4085 INFO glance.common.wsgi [-] Started child 4144
2017-02-08 15:04:09.260 4144 INFO eventlet.wsgi.server [-] (4144) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:04:09.262 4085 INFO glance.common.wsgi [-] Started child 4145
2017-02-08 15:04:09.264 4145 INFO eventlet.wsgi.server [-] (4145) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:04:09.267 4085 INFO glance.common.wsgi [-] Started child 4148
2017-02-08 15:04:09.269 4148 INFO eventlet.wsgi.server [-] (4148) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:04:09.271 4085 INFO glance.common.wsgi [-] Started child 4149
2017-02-08 15:04:09.272 4149 INFO eventlet.wsgi.server [-] (4149) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:06:59.021 16922 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admi
n_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:06:59.022 16922 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admi
n endpoint

>>> host-3 okay

root@a6s38:/var/log/contrail# cat /var/log/contrail/glance-registry-supervisor-stdout.log | more
2017-02-08 15:16:46.871 11894 INFO glance.common.wsgi [-] Starting 4 workers
2017-02-08 15:16:46.875 11894 INFO glance.common.wsgi [-] Started child 12262
2017-02-08 15:16:46.886 12263 INFO eventlet.wsgi.server [-] (12263) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:16:46.899 11894 INFO glance.common.wsgi [-] Started child 12263
2017-02-08 15:16:46.900 12262 INFO eventlet.wsgi.server [-] (12262) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:16:46.907 11894 INFO glance.common.wsgi [-] Started child 12271
2017-02-08 15:16:46.910 12271 INFO eventlet.wsgi.server [-] (12271) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:16:46.929 11894 INFO glance.common.wsgi [-] Started child 12274
2017-02-08 15:16:46.942 12274 INFO eventlet.wsgi.server [-] (12274) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:17:34.724 15796 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admi
n_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:17:34.726 15796 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admi
n endpoint
2017-02-08 15:17:34.736 15796 INFO glance.common.wsgi [-] Starting 4 workers
2017-02-08 15:17:34.747 15796 INFO glance.common.wsgi [-] Started child 16297
2017-02-08 15:17:34.749 16297 INFO eventlet.wsgi.server [-] (16297) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:17:34.750 15796 INFO glance.common.wsgi [-] Started child 16310
2017-02-08 15:17:34.760 15796 INFO glance.common.wsgi [-] Started child 16311
2017-02-08 15:17:34.761 16310 INFO eventlet.wsgi.server [-] (16310) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:17:34.766 16311 INFO eventlet.wsgi.server [-] (16311) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:17:34.768 15796 INFO glance.common.wsgi [-] Started child 16313
2017-02-08 15:17:34.771 16313 INFO eventlet.wsgi.server [-] (16313) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:19:11.648 24520 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admi
n_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:19:11.661 24520 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admi
n endpoint
2017-02-08 15:19:11.666 24520 INFO glance.common.wsgi [-] Starting 4 workers
2017-02-08 15:19:11.669 24520 INFO glance.common.wsgi [-] Started child 25734
2017-02-08 15:19:11.678 24520 INFO glance.common.wsgi [-] Started child 25740
2017-02-08 15:19:11.682 25734 INFO eventlet.wsgi.server [-] (25734) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:19:11.690 24520 INFO glance.common.wsgi [-] Started child 25744
2017-02-08 15:19:11.692 25740 INFO eventlet.wsgi.server [-] (25740) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:19:11.700 25744 INFO eventlet.wsgi.server [-] (25744) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:19:11.704 24520 INFO glance.common.wsgi [-] Started child 25750
2017-02-08 15:19:11.711 25750 INFO eventlet.wsgi.server [-] (25750) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:19:38.839 27053 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admi
n_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:19:38.840 27053 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admi
n endpoint
2017-02-08 15:19:38.851 27053 INFO glance.common.wsgi [-] Starting 4 workers
2017-02-08 15:19:38.854 27053 INFO glance.common.wsgi [-] Started child 27429
2017-02-08 15:19:38.862 27053 INFO glance.common.wsgi [-] Started child 27430
2017-02-08 15:19:38.868 27429 INFO eventlet.wsgi.server [-] (27429) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:19:38.874 27430 INFO eventlet.wsgi.server [-] (27430) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:19:38.875 27431 INFO eventlet.wsgi.server [-] (27431) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:19:38.866 27053 INFO glance.common.wsgi [-] Started child 27431
2017-02-08 15:19:38.884 27053 INFO glance.common.wsgi [-] Started child 27432
2017-02-08 15:19:38.891 27432 INFO eventlet.wsgi.server [-] (27432) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:20:29.979 31143 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admi

>>> host-2 issue

root@a6s33:~# cat /var/log/contrail/glance-registry-supervisor-stdout.log | more
2017-02-08 15:10:49.652 19865 INFO glance.common.wsgi [-] Starting 4 workers
2017-02-08 15:10:49.655 19865 INFO glance.common.wsgi [-] Started child 20140
2017-02-08 15:10:49.659 19865 INFO glance.common.wsgi [-] Started child 20141
2017-02-08 15:10:49.662 19865 INFO glance.common.wsgi [-] Started child 20142
2017-02-08 15:10:49.665 20141 INFO eventlet.wsgi.server [-] (20141) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:10:49.668 20140 INFO eventlet.wsgi.server [-] (20140) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:10:49.671 20142 INFO eventlet.wsgi.server [-] (20142) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:10:49.673 19865 INFO glance.common.wsgi [-] Started child 20143
2017-02-08 15:10:49.675 20143 INFO eventlet.wsgi.server [-] (20143) wsgi starting up on http://0.0.0.0:9191/
2017-02-08 15:11:37.743 23495 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admi
n_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:11:37.749 23495 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admi
n endpoint
2017-02-08 15:11:42.903 24235 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admi
n_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:11:42.904 24235 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admi
n endpoint
ERROR: Could not bind to 0.0.0.0:9191 after trying for 30 seconds
2017-02-08 15:12:13.712 27246 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admi
n_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:12:13.713 27246 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admi
n endpoint
ERROR: Could not bind to 0.0.0.0:9191 after trying for 30 seconds
2017-02-08 15:12:45.046 30514 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admi
n_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:12:45.047 30514 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admi
n endpoint
2017-02-08 15:13:31.646 1060 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admin
_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:13:31.647 1060 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admin
 endpoint
2017-02-08 15:14:02.076 3747 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admin
_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:14:02.077 3747 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admin
 endpoint
2017-02-08 15:14:44.269 7046 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admin
_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:14:44.278 7046 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admin
 endpoint
2017-02-08 15:14:49.591 7733 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admin
_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:14:49.593 7733 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admin
 endpoint
ERROR: Could not bind to 0.0.0.0:9191 after trying for 30 seconds
2017-02-08 15:15:21.558 10834 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admi
n_tenant_name configuration options is deprecated in favor of auth_plugin and related options and may be removed in a future release.
2017-02-08 15:15:21.559 10834 WARNING keystonemiddleware.auth_token [-] Configuring auth_uri to point to the p

Revision history for this message
Sarath (nsarath) wrote :

WorkAround : To stop & start glance-registry

Revision history for this message
Sarath (nsarath) wrote :

Please find below the Logs and Fab logs

nsarath@ubuntu-build04:/auto/cores/1663438$ ls -ltrd *
-rwxrwxrwx 1 nsarath test 8386560 Feb 9 17:39 Ctrl-A-Utils.tar
-rwxrwxrwx 1 nsarath test 45936640 Feb 9 17:39 Esxi-1-log.tar
-rwxrwxrwx 1 nsarath test 45824000 Feb 9 17:39 Esxi-2-log.tar
-rwxrwxrwx 1 nsarath test 56064000 Feb 9 17:39 Kvm-1-log.tar
-rwxrwxrwx 1 nsarath test 744970240 Feb 9 17:39 Ctrl-A-log.tar
-rwxrwxrwx 1 nsarath test 588267520 Feb 9 17:39 Ctrl-C-log.tar
-rwxrwxrwx 1 nsarath test 884285440 Feb 9 17:39 Ctrl-B-log.tar
nsarath@ubuntu-build04:/auto/cores/1663438$

Please find the testbed.py
https://github.com/Juniper/contrail-tools/blob/master/testbeds/testbed_sanity_a6s32_ha.py.vcenter_compute.ubuntu-14.04

Revision history for this message
Sachchidanand Vaidya (vaidyasd) wrote :

Sarath, are you still seeing this issue ?

Revision history for this message
Sachchidanand Vaidya (vaidyasd) wrote :

Sarath, are you still seeing this issue. This is not vCenter specific..

Revision history for this message
Sarath (nsarath) wrote :

closing the bug as not seen on latest build tests.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.