Error booting vm when specifying xfs as ephemeral file system

Bug #1850610 reported by Martin Schuppert
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Medium
Martin Schuppert
Rocky
Fix Released
Medium
Radosław Piliszek
Stein
Fix Released
Medium
Martin Schuppert
Train
Fix Released
Medium
Martin Schuppert
tripleo
Fix Released
Undecided
Martin Schuppert

Bug Description

When using ephemeral disks instances are ending up in error state if asking to format with xfs.
~~
| fault | {u'message': u"Unexpected error while running command.\nCommand: mkfs -t xfs -L ephemeral0 /var/lib/nova/instances/_base/ephemeral_1_0706d66\nExit code: 1\nStdout: u''\nStderr: u'mkfs.xfs: No such file or directory\\n'", u'code': 500, u'details': u'Traceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1858, in _do_build_and_run_instance\n filter_properties, request_spec)\n File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2142, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\nRescheduledException: Build of instance 194a0199-5b34-468e-8ae6-c87cd22afdf8 was re-scheduled: Unexpected error while running command.\nCommand: mkfs -t xfs -L ephemeral0 /var/lib/nova/instances/_base/ephemeral_1_0706d66\nExit code: 1\nStdout: u\'\'\nStderr: u\'mkfs.xfs: No such file or directory\\n\'\n', u'created': u'2019-10-23T11:48:10Z'} |
~~

I noticed that the issue occurred only on the first we are trying to spawn the instance on a specific hypervisor.
After investigation the mkfs.xfs binary is missing in the rhosp13/openstack-nova-compute docker image leading in the error reported in this case.
~~
mkfs.xfs: No such file or directory
~~

If we do the same with ext4 we don't have the issue. Also if I look in the nova-compute docker, there is no mkfs.xfs but there is a mkfs.ext4

e.g.
nova --debug boot --poll --nic net-id=728c92e2-954c-4e10-9bf1-e67ee6265743 --flavor acs.small --image circos --ephemeral size=1,format=xfs --availability-zone tu-mec-3e:ciahob2ck37c018.tu-mec-3.cloud.redhat.net test-ephemeral

Changed in tripleo:
assignee: nobody → Martin Schuppert (mschuppert)
Revision history for this message
Marcin Juszkiewicz (hrw) wrote :
Changed in kolla:
milestone: none → 9.0.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/692091

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/692092

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/692093

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.opendev.org/692075
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=9b1a243a3a1669532f25bbdcbc56b78f0f72882c
Submitter: Zuul
Branch: master

commit 9b1a243a3a1669532f25bbdcbc56b78f0f72882c
Author: Martin Schuppert <email address hidden>
Date: Wed Oct 30 09:43:23 2019 +0100

    Add xfsprogs to nova-compute

    xfsprogs is required for formatting XFS ephemeral disk partitions
    when format=xfs is specified as instance create failes with
    'mkfs.xfs: No such file or directory' due to missing xfsprogs
    package in the nova-compute container.

    Closes-Bug: #1850610

    Change-Id: Iaf3414464f3dd747427247339c6b201b352063cb

Changed in kolla:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/queens)

Reviewed: https://review.opendev.org/692093
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=b103210a86dfafbaf9e10d114d27acf5c3c319aa
Submitter: Zuul
Branch: stable/queens

commit b103210a86dfafbaf9e10d114d27acf5c3c319aa
Author: Martin Schuppert <email address hidden>
Date: Wed Oct 30 09:43:23 2019 +0100

    Add xfsprogs to nova-compute

    xfsprogs is required for formatting XFS ephemeral disk partitions
    when format=xfs is specified as instance create failes with
    'mkfs.xfs: No such file or directory' due to missing xfsprogs
    package in the nova-compute container.

    Closes-Bug: #1850610

    Conflicts:
          docker/nova/nova-compute/Dockerfile.j2

    Change-Id: Iaf3414464f3dd747427247339c6b201b352063cb
    (cherry picked from commit 9b1a243a3a1669532f25bbdcbc56b78f0f72882c)
    (cherry picked from commit 23e488dfe73a9273698f2a33d8b74f770d9a83f6)
    (cherry picked from commit f0705fcfbd38564d3cb2efa666a67e6b7b757d18)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/stein)

Reviewed: https://review.opendev.org/692091
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=23e488dfe73a9273698f2a33d8b74f770d9a83f6
Submitter: Zuul
Branch: stable/stein

commit 23e488dfe73a9273698f2a33d8b74f770d9a83f6
Author: Martin Schuppert <email address hidden>
Date: Wed Oct 30 09:43:23 2019 +0100

    Add xfsprogs to nova-compute

    xfsprogs is required for formatting XFS ephemeral disk partitions
    when format=xfs is specified as instance create failes with
    'mkfs.xfs: No such file or directory' due to missing xfsprogs
    package in the nova-compute container.

    Closes-Bug: #1850610

    Change-Id: Iaf3414464f3dd747427247339c6b201b352063cb
    (cherry picked from commit 9b1a243a3a1669532f25bbdcbc56b78f0f72882c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/rocky)

Reviewed: https://review.opendev.org/692092
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=d1731da6330c11f80e9cb875d642d5208b81fa98
Submitter: Zuul
Branch: stable/rocky

commit d1731da6330c11f80e9cb875d642d5208b81fa98
Author: Martin Schuppert <email address hidden>
Date: Wed Oct 30 09:43:23 2019 +0100

    Add xfsprogs to nova-compute

    xfsprogs is required for formatting XFS ephemeral disk partitions
    when format=xfs is specified as instance create failes with
    'mkfs.xfs: No such file or directory' due to missing xfsprogs
    package in the nova-compute container.

    Closes-Bug: #1850610

    Conflicts:
          docker/nova/nova-compute/Dockerfile.j2

    Change-Id: Iaf3414464f3dd747427247339c6b201b352063cb
    (cherry picked from commit 9b1a243a3a1669532f25bbdcbc56b78f0f72882c)
    (cherry picked from commit 23e488dfe73a9273698f2a33d8b74f770d9a83f6)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 9.0.0.0rc1

This issue was fixed in the openstack/kolla 9.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 7.1.0

This issue was fixed in the openstack/kolla 7.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 8.0.2

This issue was fixed in the openstack/kolla 8.0.2 release.

wes hayutin (weshayutin)
Changed in tripleo:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla queens-eol

This issue was fixed in the openstack/kolla queens-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.