Periodic Jobs failing at tempest config while creating image(with swift backend)

[yatin]

Failure logs:-
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/ea77cda/undercloud/home/jenkins/tempest_output.log.txt.gz
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/b4a4255/undercloud/home/jenkins/tempest_output.log.txt.gz
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/a19ca83/undercloud/home/jenkins/tempest_output.log.txt.gz

Glance is failing to upload to swift: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/ea77cda/overcloud-controller-foo-0/var/log/containers/glance/api.log.txt.gz#_2018-02-01_04_15_58_209

Last passing jobs are from 26th January:-
https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/25/consoleText
https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/289/consoleText

[Alan Pevec (apevec)]

In failing job openstack-swift-2.16.1-0.20180131025146.d800c17.el7
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/ea77cda/overcloud-controller-foo-0/var/log/extra/rpm-list.txt.gz

In Jan 26 passing jobs I do not see overcloud logs under https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/eb23456/
but in https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/eb23456/rpm-qa.txt.gz
Swift version is openstack-swift-2.15.2-0.20180118220641.449d83f.el7

[Alan Pevec (apevec)]

Jan 26 was when Swift was unpinned https://review.rdoproject.org/r/11248

[Christian Schwede (cschwede)]

Looks like Swift can't create data in the given directory (/srv/node/d1).

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/ea77cda/overcloud-controller-foo-0/var/log/swift/swift.log.txt.gz#_Feb__1_04_15_26

There were two required fixes for t-h-t and puppet-tripleo that merged on master ~ 3 weeks ago:

https://review.openstack.org/#/c/517374/
https://review.openstack.org/#/c/517373/

However, it seems they are already in?

Any chance to SSH into a deployment to check this directly? Or any other debugging possibility to check the /srv/node/d1 directory?

[wes hayutin (weshayutin)]

@Christian, we can't hold environments however using rdo-cloud and this script you should get the same exact environment up

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/ea77cda/reproducer-quickstart.sh

The very latest failure and repro is here
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/0daa56c/reproducer-quickstart.sh

from:
https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/

[Bogdan Dobrelya (bogdando)]

The logs show 'ERROR Insufficient Storage'

[Alan Pevec (apevec)]

There should be enough space
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/ea77cda/overcloud-controller-foo-0/var/log/host_info.txt.gz

+ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda2 80G 8.9G 72G 12% /

[Alan Pevec (apevec)]

Actually proxy-server: ERROR Insufficient Storage 172.19.0.17:6002/d1 is the same as in https://bugs.launchpad.net/tripleo/+bug/1729569 b/c of which Swift was pinned back in November.
Looks like THT fix https://review.openstack.org/517374 is not enough?

[Alan Pevec (apevec)]

> Looks like THT fix https://review.openstack.org/517374 is not enough?

@Christian that changed puppet/services/swift-storage.yaml shouldn't also docker/services/swift-storage.yaml be adjusted?

[Christian Schwede (cschwede)]

> Looks like THT fix https://review.openstack.org/517374 is not enough?

> @Christian that changed puppet/services/swift-storage.yaml shouldn't also docker/services/swift-> storage.yaml be adjusted?

Isn't this automatically used because of the imported SwiftStorageBase?
Even if not, the setting is true by default in puppet-tripleo, so this can't be the only reason.

I'm trying to reproduce this on rdo-cloud; if someone can check in the meantime if /srv/node/d1 does exist and is writeable by the Swift containers, that would be helpful.

[Sagi (Sergey) Shnaidman (sshnaidm)]

Is any progress on that?
It's main blocking promotion bug atm
thanks

[Christian Schwede (cschwede)]

Using the reproducer script on RDO wasn't successful so far - it run into a timeout two times in a row, third time failed just now when deploying the overcloud. Now trying something different with OOOq.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/541662

[Christian Schwede (cschwede)]

So the /srv/node/d1 directory was missing, i'm still wondering why because puppet-tripleo should create this automatically (see https://review.openstack.org/#/c/517374/). And in fact it is creating this when running OOOq on master.

It's just not working when using the reproducer on RDO cloud - which seems strange to me.

I proposed an upstream fix, but this one doesn't work on the reproducer either: https://review.openstack.org/541662

[Christian Schwede (cschwede)]

Also interesting: puppet-swift seems to create the directory d1 on the reproducer, but it's not there once containers are running. I see this in /var/log/messages:

Notice: /Stage[main]/Tripleo::Profile::Base::Swift::Storage/File[/srv/node/d1]/ensure: created

I also see that the command from my posted fix is executed:

mkdir -p /srv/node/d1

Still, this directory does not exist once containers are up and running?

[Bogdan Dobrelya (bogdando)]

I think we want the /srv/node/d1 created with host prep tasks and bind-mounted for the container

[Christian Schwede (cschwede)]

Updated my patch - please have a look!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/541662
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=bc8618126f12d3a0d41aa59701c0575c70e17879
Submitter: Zuul
Branch: master

commit bc8618126f12d3a0d41aa59701c0575c70e17879
Author: Christian Schwede <email address hidden>
Date: Wed Feb 7 10:56:15 2018 +0100

    Fix missing Swift d1 directory

    The /srv/node/d1 directory was missing, thus creating it in advance.

    Note: there is a related change that merged earlier (f6108f5d) but
    for some reason didn't work as expected.

    Closes-Bug: 1746734
    Change-Id: Iabaa2033d065c9da653f7ba9e25430c3554a1169

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0rc1 release candidate.