tripleo

Periodic jobs using swift as glance backend failing

Bug #1729569 reported by Alfredo Moralejo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Emilien Macchi
tripleo queens-3

Bug Description

TripleO jobs using swift as glance backend are failing when trying to upload images. Following error is found in glance API log [1]:

2017-11-02 05:23:07.763 51205 ERROR glance.common.wsgi Got error from Swift: Container PUT failed: http://192.168.24.8:8080/v1/AUTH_b8ca634b5fda4802a9630eb165a6973f/glance 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<.

Following swift error is seen swift log [2]:

Nov 2 05:23:07 upstream-centos-7-2-node-rdo-cloud-tripleo-30449-15322 proxy-server: ERROR Insufficient Storage 192.168.24.13:6002/d1 (txn: tx6257de642c74474db261b-0059faabbb)

[1] https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset006-master/63d2919/subnode-2/var/log/glance/api.log.txt.gz
[2] https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset006-master/63d2919/subnode-2/var/log/swift/swift.log.txt.gz

Tags: ci tech-debt
Revision history for this message
Alfredo Moralejo (amoralej) wrote :

Seems caused by https://review.openstack.org/#/c/500158

Changed in tripleo:
importance: Undecided → Critical
tags: added: alert ci promotion-blocker
Changed in tripleo:
milestone: none → queens-2
status: New → Triaged
Revision history for this message
Alfredo Moralejo (amoralej) wrote :

cschwede is working to fix this in puppet-swift/tripleo

As workaroung we'll propose to pin swift in RDO until fix is merged.

Revision history for this message
Ben Nemec (bnemec) wrote :

Can we not always alert on promotion blockers? This is literally just promotion doing it's job, and these types of failures do not block either developer or user deployments of TripleO. I'm totally fine with making these critical bugs and potentially even alerting on them if they block promotion for a long time, but I don't think we should be alerting on them as soon as they pop up.

In any case, I'm dropping the alert on this one since it doesn't even seem to be something that TripleO can fix so there's no point spamming the channel about it.

tags: removed: alert
Revision history for this message
Alfredo Moralejo (amoralej) wrote :

My understanding is that we MUST use alert tag for promotion blockers, specially if they are related to deployment framework (as this case that will be somewhere between puppet-swift/tripleo iiuc). if i'm wrong, what's the policy about what tags to apply for tripleo bugs?.

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

As workaround, swift has been pinned in RDO, https://review.rdoproject.org/r/#/c/10385/

tags: added: tech-debt
removed: ci promotion-blocker
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to puppet-tripleo (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/517373

Changed in tripleo:
assignee: nobody → Christian Schwede (cschwede)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/517374

Alex Schultz (alex-schultz)
Changed in tripleo:
milestone: queens-2 → queens-3
Revision history for this message
Alan Pevec (apevec) wrote :

This is getting critical and almost dropped out of radar because it was prematurely de-escalated.

tags: added: alert
Bogdan Dobrelya (bogdando)
tags: added: ci
Alan Pevec (apevec)
tags: removed: alert
OpenStack Infra (hudson-openstack)
Changed in tripleo:
assignee: Christian Schwede (cschwede) → Emilien Macchi (emilienm)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to puppet-tripleo (master)

Reviewed: https://review.openstack.org/517373
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=ffd524d7f1d9205ff463457e5e54d562e723f7e4
Submitter: Zuul
Branch: master

commit ffd524d7f1d9205ff463457e5e54d562e723f7e4
Author: Christian Schwede <email address hidden>
Date: Thu Nov 2 17:02:04 2017 +0000

    Create Swift directory d1 if needed

    Swift added a requirement that storage directories must exist before
    using them. In case of the d1 directory in TripleO - used when there are
    no "real disks" - it has to be created by TripleO in advance.

    Related-Bug: 1729569
    Change-Id: I49e395ac379ced01adb60d8d9f951c08718b1c61

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/517374
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f6108f5dab9501ec4a91449e2faaae5ec894c379
Submitter: Zuul
Branch: master

commit f6108f5dab9501ec4a91449e2faaae5ec894c379
Author: Christian Schwede <email address hidden>
Date: Thu Nov 2 17:04:52 2017 +0000

    Create Swift directory d1 if required

    Swift added a requirement that storage directories must exist before
    using them. In case of the d1 directory in TripleO - used when there are
    no "real disks" - it has to be created by TripleO in advance.

    This change forwards the SwiftUseLocalDir parameter to puppet-tripleo,
    which creates the directory with the correct permissions.

    Closes-Bug: 1729569
    Depends-On: I49e395ac379ced01adb60d8d9f951c08718b1c61
    Change-Id: I82b783541c6097f6b8747b63349720c47f3b3f94

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.0.0.0b3

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.