Glance failed to upload image under high load

Bug #1653165 reported by Mikhail Chernik
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Confirmed
High
MOS Glance

Bug Description

When running rally scenario http://paste.openstack.org/show/593606/ against MOS 9.2 snapshot #688, task failed with Gateway Time-out error due to internal error in Glance: "Failed to upload image data due to internal error"

glance logs: http://paste.openstack.org/show/593604/ http://paste.openstack.org/show/593602/

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Mikhail, could you please provide more logs? Specifically, haproxy logs from all controller nodes.

Changed in mos:
status: New → Incomplete
assignee: nobody → Mikhail Chernik (mchernik)
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

We need to attach diagnostic snapshot, the bottleneck can be in Glance API / HAProxy / Keystone API / DataBase and we need to get more logs to identify the root of the issue.

Revision history for this message
Mikhail Chernik (mchernik) wrote :

Diagnostic snapshot: http://mos-scale-share.mirantis.com/env_10_scale_run_build_688_snapshot/

The test passed when was run once again with Cinder/Glance debugging was turned off.

Changed in mos:
status: Incomplete → New
assignee: Mikhail Chernik (mchernik) → nobody
Changed in mos:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → MOS Glance (mos-glance)
tags: added: move-to-9.3
Changed in mos:
milestone: 9.2 → 9.3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.