Undercloud running out of disk space in CI

Bug #1649615 reported by Ben Nemec
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned

Bug Description

This is a continuation of https://bugs.launchpad.net/tripleo/+bug/1649252 because the original problem there has been fixed, but now we're hitting a different one.

The relevant bit is Sagi's comment:

"""
There is no enough space on our undercloud (nodepool images), which fails ironic deployments:

http://logs.openstack.org/43/407943/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/680096e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2016-12-13_01_17_25_959

2016-12-13 01:17:25.961 24965 ERROR nova.compute.manager [instance: fa3c962d-9cbd-4712-a6a2-4bb9012cbf96] InstanceDeployFailure: Failed to provision instance fa3c962d-9cbd-4712-a6a2-4bb9012cbf96: Failed to deploy. Error: Disk volume where '/var/lib/ironic/master_images/tmpU78MIq' is located doesn't have enough disk space. Required 5116 MiB, only 4302 MiB available space present.

http://logs.openstack.org/43/407943/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/680096e/logs/undercloud/var/log/ironic/ironic-conductor.txt.gz#_2016-12-13_01_12_33_251

2016-12-13 01:12:33.251 25629 ERROR ironic.conductor.manager [req-5048c3cb-6863-455c-9fc8-effd2137b2da 79b54a43398b431c8bac6439afdb8595 f2474735437e49b38b0225c8d603af18 - default -] Error in deploy of node cf0d9729-5740-47c9-a1ed-171e80f6a787: Disk volume where '/var/lib/ironic/master_images/tmpiizphO' is located doesn't have enough disk space. Required 5116 MiB, only 4330 MiB available space present.
"""

It's entirely possible we just need to increase the size of our undercloud vms. Note that this is somewhat problematic because the undercloud vms are a standard infra flavor, and changing the disk size would give us another snowflake trait.

Tags: ci
Revision history for this message
Emilien Macchi (emilienm) wrote :
tags: removed: alert
Changed in tripleo:
status: Triaged → Fix Released
milestone: none → ocata-2
Revision history for this message
Emilien Macchi (emilienm) wrote :
Changed in tripleo:
status: Fix Released → In Progress
tags: added: alert
Changed in tripleo:
status: In Progress → Fix Committed
Revision history for this message
Emilien Macchi (emilienm) wrote :

I'll move it to Fix Released once we're sure this bug is fixed.

Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

The patch faild verificatio0n, reverifying it again.

I have another patch as well that should free up to 8GB for us: https://review.openstack.org/#/c/410323/

tags: removed: alert
Changed in tripleo:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.