[Build "Ubuntu 14.04.2 LTS" 3.0.2.0-35~kilo]:Glance api failed to start

[Sandip Dey]

All sanity cases failed here as glance api failed to start

http://10.204.216.50/Docs/logs/3.0.2.0-35_2016_05_11_07_18_47/junit-noframes.html

logs saved at /cs-shared/test_runs/nodei13-vm1/2016_05_11_07_18_47

log
====
2016-05-11 08:08:43,775 INFO success: nova-scheduler entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-05-11 08:08:43,775 INFO success: heat-engine entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-05-11 08:08:43,775 INFO success: glance-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-05-11 08:08:43,775 INFO success: heat-api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-05-11 08:08:43,775 INFO success: cinder-scheduler entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-05-11 08:08:44,104 INFO exited: glance-api (exit status 1; not expected)
2016-05-11 08:08:44,111 INFO spawned: 'glance-api' with pid 8277
2016-05-11 08:08:44,750 INFO exited: glance-api (exit status 1; not expected)
2016-05-11 08:08:45,755 INFO spawned: 'glance-api' with pid 8364
2016-05-11 08:08:46,770 INFO success: glance-api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-05-11 08:08:47,580 INFO exited: glance-api (exit status 1; not expected)
2016-05-11 08:08:47,589 INFO spawned: 'glance-api' with pid 8515
2016-05-11 08:08:48,589 INFO success: glance-api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-05-11 08:08:48,799 INFO exited: glance-api (exit status 1; not expected)
2016-05-11 08:08:48,802 INFO spawned: 'glance-api' with pid 8531
2016-05-11 08:08:49,128 INFO exited: glance-api (exit status 1; not expected)
2016-05-11 08:08:50,318 INFO spawned: 'glance-api' with pid 8613
2016-05-11 08:08:50,620 INFO exited: glance-api (exit status 1; not expected)
2016-05-11 08:08:53,330 INFO spawned: 'glance-api' with pid 8720
2016-05-11 08:08:53,657 INFO exited: glance-api (exit status 1; not expected)
2016-05-11 08:08:57,366 INFO spawned: 'glance-api' with pid 8740
2016-05-11 08:08:58,027 INFO exited: glance-api (exit status 1; not expected)
2016-05-11 08:08:58,374 INFO gave up: glance-api entered FATAL state, too many start retries too quickly
2016-05-11 08:09:04,135 INFO stopped: glance-registry (terminated by SIGKILL)
2016-05-11 08:09:04,139 INFO spawned: 'glance-registry' with pid 9018
2016-05-11 08:09:05,641 INFO success: glance-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-05-11 08:09:05,771 INFO spawned: 'glance-api' with pid 9166
2016-05-11 08:09:06,230 INFO exited: glance-api (exit status 1; not expected)
2016-05-11 08:09:06,848 INFO stopped: keystone (terminated by SIGKILL)
2016-05-11 08:09:06,851 INFO spawned: 'keystone' with pid 9201
2016-05-11 08:09:07,310 INFO spawned: 'glance-api' with pid 9216
2016-05-11 08:09:07,617 INFO exited: glance-api (exit status 1; not expected)
2016-05-11 08:09:08,427 INFO success: keystone entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-05-11 08:09:10,045 INFO spawned: 'glance-api' with pid 9304
2016-05-11 08:09:10,362 INFO exited: glance-api (exit status 1; not expected)
2016-05-11 08:09:13,546 INFO spawned: 'glance-api' with pid 9401
2016-05-11 08:09:13,967 INFO exited: glance-api (exit status 1; not expected)
2016-05-11 08:09:14,364 INFO gave up: glance-api entered FATAL state, too many start retries too quickly

[Atul Moghe (moghea)]

I am unable to create this problem on single box docker setup with a given image. Glance api works fine after fab setup_all. Though glance-manage db_sync gives similar warning during provisioning, That warning is due to deprecated api but should not make glance api service to die. Please recreate the problem and keep the setup in that state.

[Sudheendra Rao (sudheendra-k)]

Same problem is seen on build42, locked the testbed to debug.
Topology :
Config Nodes : [u'nodei13-vm1', u'nodek7-vm1']
Control Nodes : [u'nodei13-vm1', u'nodek7-vm1']
Compute Nodes : [u'nodeg28', u'nodeg29', u'nodeg30']
Openstack Node : nodek7-vm1
WebUI Node : nodei14-vm1
Analytics Nodes : [u'nodei13-vm1', u'nodek7-vm1']
Physical Devices : [u"'hooper'"]

This is virtual testbed, please find the ip address of the virtual nodes:
nodei13-vm1 - 10.204.217.103
nodei14-vm1 - 10.204.217.232
nodek7-vm1 - 10.204.216.110

[Vedamurthy Joshi (vedujoshi)]

Atul,
It seems to be failing due to api.log being owned by root.
Rahul too hit a similar issue on his setup. Changing the owner to glance fixed it.

Looks like it is being randomly hit

Please check.

root@nodek7-vm1:/var/log/glance# ls -l
total 8
-rw-r--r-- 1 root root 0 May 28 06:39 api.log
-rw-r--r-- 1 root root 503 May 28 06:39 api.log.1
-rw-r--r-- 1 glance glance 0 May 28 06:39 registry.log
-rw-r--r-- 1 glance glance 1272 May 28 06:39 registry.log.1
root@nodek7-vm1:/var/log/glance#

[Atul Moghe (moghea)]

yes, problem is consistent due to ownership issue, as we have made a config knob in provisioning that defines log_file, need to provision glance with proper owner credentials "glance:glance"

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20793
Submitter: Atul Moghe (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20794
Submitter: Atul Moghe (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20793
Committed: http://github.org/Juniper/contrail-provisioning/commit/c8f95eb82fb56d63eac3d3f3bd27c76beee6dd0e
Submitter: Zuul
Branch: R3.0

commit c8f95eb82fb56d63eac3d3f3bd27c76beee6dd0e
Author: Atul Moghe <email address hidden>
Date: Tue May 31 14:42:35 2016 -0700

change /var/log/api.log ownership to glance:glance to avoid glance-api service crash.
Since provisioning code explicitly mentioned log_file as /var/log/glance/api.log, ownership sets to
root causing glance to fail while updating a log file.
Closes-Bug: #1580446

Change-Id: I2914bc2b3994dedd568d2d74e70197aba9802f54

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20794
Committed: http://github.org/Juniper/contrail-provisioning/commit/7d778fd1f944864070ef8db82484b2bad958e8b2
Submitter: Zuul
Branch: master

commit 7d778fd1f944864070ef8db82484b2bad958e8b2
Author: Atul Moghe <email address hidden>
Date: Tue May 31 14:52:33 2016 -0700

change /var/log/api.log ownership to glance:glance to avoid glance-api service crash.
Since provisioning code explicitly mentioned log_file as /var/log/glance/api.log, ownership sets to
root causing glance to fail while updating a log file.
Closes-Bug: #1580446

Change-Id: I6bb59204cbe044f9dfac183fa6110fdcdde5636d