Grenade fails with /opt/stack/logs/screen/error.log: Permission denied

[Dan Smith]

Grenade fails to take action on some openstack-owned log file due to a permissions error:

http://logs.openstack.org/37/113337/2/check/check-grenade-dsvm-partial-ncpu/d375363/logs/grenade.sh.txt.gz

2014-08-12 14:27:01.606 | + screen -S stack -X screen -t ceilometer-anotification
2014-08-12 14:27:01.610 | There is a screen on:
2014-08-12 14:27:01.610 | 29046.stack (08/12/2014 02:07:45 PM) (Dead ???)
2014-08-12 14:27:01.611 | Remove dead screens with 'screen -wipe'.
2014-08-12 14:27:01.611 | No screen session found.
2014-08-12 14:27:01.612 | + die 91 'Failure in upgrade-ceilometer'
2014-08-12 14:27:01.612 | + local exitcode=1
2014-08-12 14:27:01.612 | [Call Trace]
2014-08-12 14:27:01.612 | ./grenade.sh:323:upgrade_service
2014-08-12 14:27:01.612 | ./grenade.sh:91:die
2014-08-12 14:27:01.615 | [ERROR] ./grenade.sh:91 Failure in upgrade-ceilometer
2014-08-12 14:27:01.615 | /opt/stack/new/grenade/functions: line 133: /opt/stack/logs/screen/error.log: Permission denied
2014-08-12 14:27:01.615 | Exit code: 1
2014-08-12 14:27:01.643 | World dumping... see /opt/stack/logs/worlddump-2014-08-12-142701.txt for details

[Clark Boylan (cboylan)]

This appears to have failed because ceilometer failed to upgrade. Need more info from grenade and ceilometer if this is an actual infra bug.

[gordon chung (chungg)]

seems pretty intermittent:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcIkZhaWx1cmUgaW4gdXBncmFkZS1jZWlsb21ldGVyXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDkzNDcxMzg2ODF9

[gordon chung (chungg)]

this hasn't occurred recently (assuming my logstash query is correct)... i'll leave this open for now so we have a placeholder if it does come back.

[Chris Dent (cdent)]

It's not always happening on ceilometer. It's basically anything that can't start a screen because a screen was left behind, try this query:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiUmVtb3ZlIGRlYWQgc2NyZWVucyB3aXRoICdzY3JlZW4gLXdpcGUnXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MTE5ODkxMTcyOTZ9

Presumably once the USE_SCREEN=False stuff is turned on we'll be able to inspect this more closely.

In any case, this doesn't appear to be a ceilo problem.

[Chris Dent (cdent)]

A recent look at logstash[1] shows that this is still happening every now and again, even with the USE_SCREEN changes in place. Digging around it appears the problem here is that the value of SCREEN_LOGDIR is different when the err() function is called from when it is used to create the the SCREEN_LOGDIRs.

In a grenade.sh log we see these various assignments:

SCREEN_LOGDIR=/opt/stack/new/logs/screen
SCREEN_LOGDIR=/opt/stack/new/screen-logs
SCREEN_LOGDIR=/opt/stack/old/logs/screen
SCREEN_LOGDIR=/opt/stack/old/screen-logs

But the one that is actually used when calling err() is '/opt/stack/logs/screen/error.log' which is set in grenaderc.

Given the intricacies of where in the stack we are and who is in charge of a env variable at any given moment I'm not sure what the right fix is here.

Note: This error is hiding actual errors, but is not the cause of failure a grenade run itself.

[1] http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiL29wdC9zdGFjay9sb2dzL3NjcmVlbi9lcnJvci5sb2c6IFBlcm1pc3Npb24gZGVuaWVkXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MTI1ODk0ODM5NzB9

[Chris Dent (cdent)]

Actually this ought to fix it: https://review.openstack.org/#/c/125924/

However there may be another bug: Does /opt/stack/logs/screen/error.log need to be one of the stashed files?

[Sean Dague (sdague)]

This grenade bug was last updated over 180 days ago, as grenade
is a fast moving project and we'd like to get the tracker down to
currently actionable bugs, this is getting marked as Invalid. If the
issue still exists, please feel free to reopen it.