/opt/stack/devstack/tools/outfilter.py failing in neutron functional jobs since 8/2

[Matt Riedemann]

Seen here:

https://logs.opendev.org/86/673486/4/gate/neutron-functional-python27/c3fe4df/ara-report/result/28d8d223-313a-49ba-b8aa-8af15fdda973/

++ ./stack.sh:main:500 : /opt/stack/devstack/tools/outfilter.py -v --no-timestamp -o /opt/stack/logs/devstacklog.txt.2019-08-02-160322
Traceback (most recent call last):
  File "/opt/stack/devstack/tools/outfilter.py", line 104, in <module>
    sys.exit(main())
  File "/opt/stack/devstack/tools/outfilter.py", line 61, in main
    outfile = open(opts.outfile, 'ab', 0)
IOError: [Errno 13] Permission denied: '/opt/stack/logs/devstacklog.txt.2019-08-02-160322'

Looks like it's a result of:

https://review.opendev.org/#/c/203698/

Based on logstash data of that failure:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22IOError%3A%20%5BErrno%2013%5D%20Permission%20denied%3A%20'%2Fopt%2Fstack%2Flogs%2Fdevstacklog.txt%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

[Slawek Kaplonski (slaweq)]

Fullstack job is failing for same reason: https://logs.opendev.org/86/673486/4/gate/neutron-fullstack/00b86e2/job-output.txt.gz#_2019-08-02_16_03_36_474424

[Dr. Jens Harbott (j-harbott)]

This is a bug in the neutron jobs uncovered by the devstack change. The log directory is created as user zuul

https://logs.opendev.org/86/673486/4/gate/neutron-functional-python27/c3fe4df/ara-report/result/efee2f33-0b24-48ea-a37a-b23b4f1a2b11/

and later devstack is being run as user "stack":

https://logs.opendev.org/86/673486/4/gate/neutron-functional-python27/c3fe4df/ara-report/file/162babb8-0600-4bc9-80d0-659f30edb864/#line-1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/674426

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/674426
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=86feb7916b6a69a7d4076cb3fc59e392ab1b73c1
Submitter: Zuul
Branch: master

commit 86feb7916b6a69a7d4076cb3fc59e392ab1b73c1
Author: Jens Harbott <email address hidden>
Date: Fri Aug 2 21:21:52 2019 +0000

    Fix log directory permissions for functional job

    After [0] devstack no longer changes the ownership of the whole
    /opt/stack tree to the stack user unconditionally. So we need to create
    /opt/stack/logs with the correct owner directly.

    [0] https://review.opendev.org/203698

    Change-Id: I4597c83bfce6267ef93bb8f7b0de62fa3ac94db4
    Closes-Bug: 1838811

[Ghanshyam Mann (ghanshyammann)]

It was owner issue on neutron job. Thanks Jens. - https://review.opendev.org/203698

Anything we need to fix on devstack side ? I think we are good here.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.