Containers: Creating VM with ephemeral disk fails at mkfs for epheramal disk

[Yang Liu]

Brief Description
-----------------
VM creation fails when launch from a flavor with ephemeral and swap disks

Severity
--------
Major

Steps to Reproduce
------------------
- Create a flavor with ephemeral (1G) and swap disks (512M) set
- Create a cinder volume
- Attempt to launch a vm using above flavor and volume

Expected Behavior
------------------
- VM launches successfully

Actual Behavior
----------------
- VM launch failed

Reproducibility
---------------
Intermittent
This issue is reproducible in sx sanity when the test is first executed, however I tried to reproduce it after a day, and it was passing.

System Configuration
--------------------
One node system

Branch/Pull Time/Commit
-----------------------
master as of "2019-01-15_20-18-00"

Timestamp/Logs
--------------
{"log":"2019-01-17 20:50:52,778.778 104572 INFO nova.compute.resource_tracker [req-365a2f35-9e7b-4d9b-b8af-5126f71cf886 a5b04eb361bc491a9b7e1e8bc0056b7a 4cf1217899db4bfc9db68d57ab6c89b5 - default default] [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] sign=1, id=instance-00000006, name=tenant2-vol_ephemswap-6, vm_mode=building, task_state=None, power_state=pending, numa_topology=node:0, 1024MB, pgsize:2M, vcpus:0, unallocated, pol:sha, \n","stream":"stdout","time":"2019-01-17T20:50:52.779414496Z"}
{"log":"2019-01-17 20:50:52,951.951 104572 WARNING nova.virt.libvirt.driver [req-365a2f35-9e7b-4d9b-b8af-5126f71cf886 a5b04eb361bc491a9b7e1e8bc0056b7a 4cf1217899db4bfc9db68d57ab6c89b5 - default default] [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] Ignoring supplied device name: /dev/vda. Libvirt can't honour user-supplied dev names\n","stream":"stdout","time":"2019-01-17T20:50:52.952304237Z"}
{"log":"2019-01-17 20:50:53,056.056 104572 INFO nova.virt.block_device [req-365a2f35-9e7b-4d9b-b8af-5126f71cf886 a5b04eb361bc491a9b7e1e8bc0056b7a 4cf1217899db4bfc9db68d57ab6c89b5 - default default] [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] Booting with blank volume at /dev/vda\n","stream":"stdout","time":"2019-01-17T20:50:53.057197308Z"}
{"log":"2019-01-17 20:50:53,900.900 104572 INFO nova.virt.libvirt.driver [req-365a2f35-9e7b-4d9b-b8af-5126f71cf886 a5b04eb361bc491a9b7e1e8bc0056b7a 4cf1217899db4bfc9db68d57ab6c89b5 - default default] [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] Creating image\n","stream":"stdout","time":"2019-01-17T20:50:53.900963861Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [req-365a2f35-9e7b-4d9b-b8af-5126f71cf886 a5b04eb361bc491a9b7e1e8bc0056b7a 4cf1217899db4bfc9db68d57ab6c89b5 - default default] [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] Instance failed to spawn: ProcessExecutionError: Unexpected error while running command.\n","stream":"stdout","time":"2019-01-17T20:50:53.945060693Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] Traceback (most recent call last):\n","stream":"stdout","time":"2019-01-17T20:50:53.945095726Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py\", line 2408, in _build_resources\n","stream":"stdout","time":"2019-01-17T20:50:53.945098695Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] yield resources\n","stream":"stdout","time":"2019-01-17T20:50:53.945101586Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/nova/compute/manager.py\", line 2158, in _build_and_run_instance\n","stream":"stdout","time":"2019-01-17T20:50:53.945104392Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] block_device_info=block_device_info)\n","stream":"stdout","time":"2019-01-17T20:50:53.945107326Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/nova/compute/resource_tracker.py\", line 870, in spawn_instance_and_update_tracker\n","stream":"stdout","time":"2019-01-17T20:50:53.945110053Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] block_device_info=block_device_info)\n","stream":"stdout","time":"2019-01-17T20:50:53.945118281Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py\", line 2968, in spawn\n","stream":"stdout","time":"2019-01-17T20:50:53.945121444Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] block_device_info=block_device_info)\n","stream":"stdout","time":"2019-01-17T20:50:53.945125012Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py\", line 3557, in _create_image\n","stream":"stdout","time":"2019-01-17T20:50:53.945127884Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] specified_fs=specified_fs)\n","stream":"stdout","time":"2019-01-17T20:50:53.945130821Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py\", line 244, in cache\n","stream":"stdout","time":"2019-01-17T20:50:53.945133464Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] *args, **kwargs)\n","stream":"stdout","time":"2019-01-17T20:50:53.945136312Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py\", line 599, in create_image\n","stream":"stdout","time":"2019-01-17T20:50:53.945138995Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] prepare_template(target=base, *args, **kwargs)\n","stream":"stdout","time":"2019-01-17T20:50:53.945141956Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_concurrency/lockutils.py\", line 271, in inner\n","stream":"stdout","time":"2019-01-17T20:50:53.945144849Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] return f(*args, **kwargs)\n","stream":"stdout","time":"2019-01-17T20:50:53.945148096Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py\", line 239, in fetch_func_sync\n","stream":"stdout","time":"2019-01-17T20:50:53.94515264Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] fetch_func(target=target, *args, **kwargs)\n","stream":"stdout","time":"2019-01-17T20:50:53.945155763Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/nova/virt/libvirt/driver.py\", line 3310, in _create_ephemeral\n","stream":"stdout","time":"2019-01-17T20:50:53.945158769Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] specified_fs=specified_fs)\n","stream":"stdout","time":"2019-01-17T20:50:53.945161778Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/nova/virt/disk/api.py\", line 119, in mkfs\n","stream":"stdout","time":"2019-01-17T20:50:53.945166929Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] utils.mkfs(specified_fs, target, fs_label, run_as_root=run_as_root)\n","stream":"stdout","time":"2019-01-17T20:50:53.945169903Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/nova/utils.py\", line 1076, in mkfs\n","stream":"stdout","time":"2019-01-17T20:50:53.945172666Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] execute(*args, run_as_root=run_as_root)\n","stream":"stdout","time":"2019-01-17T20:50:53.94517559Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/nova/utils.py\", line 517, in execute\n","stream":"stdout","time":"2019-01-17T20:50:53.945178403Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] return processutils.execute(*cmd, **kwargs)\n","stream":"stdout","time":"2019-01-17T20:50:53.945181286Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_concurrency/processutils.py\", line 419, in execute\n","stream":"stdout","time":"2019-01-17T20:50:53.945184023Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] cmd=sanitized_cmd)\n","stream":"stdout","time":"2019-01-17T20:50:53.94518687Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] ProcessExecutionError: Unexpected error while running command.\n","stream":"stdout","time":"2019-01-17T20:50:53.945189687Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] Command: mkfs -t ext4 -F -L ephemeral0 /var/lib/nova/instances/_base/ephemeral_1_40d1d2c\n","stream":"stdout","time":"2019-01-17T20:50:53.945192724Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] Exit code: 1\n","stream":"stdout","time":"2019-01-17T20:50:53.945195615Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] Stdout: u''\n","stream":"stdout","time":"2019-01-17T20:50:53.945198296Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] Stderr: u'mkfs.ext4: No such file or directory\\n'\n","stream":"stdout","time":"2019-01-17T20:50:53.94520099Z"}
{"log":"2019-01-17 20:50:53,941.941 104572 ERROR nova.compute.manager [instance: f2ddbbb3-5258-4cf3-9ea5-e0b161d7bd6c] \n","stream":"stdout","time":"2019-01-17T20:50:53.945203789Z"}

[Ghada Khalil (gkhalil)]

Marking as release gating; issue is related to the containerization env

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/631883

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/631883
Committed: https://git.openstack.org/cgit/openstack/stx-upstream/commit/?id=9e649fabd82b062fe6fdea3e5b0e93385d48e361
Submitter: Zuul
Branch: master

commit 9e649fabd82b062fe6fdea3e5b0e93385d48e361
Author: Don Penney <email address hidden>
Date: Fri Jan 18 16:03:45 2019 -0500

    Add e2fsprogs to stx-nova package list

    This update adds e2fsprogs to the pike and master docker
    image build directives files for stx-nova, to provide the
    mkfs utilities required for ephemeral storage support.

    Change-Id: If9b901696169d7d157a37d6b96f7b8c4db0a24a5
    Closes-Bug: 1812432
    Signed-off-by: Don Penney <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/stein
Review: https://review.openstack.org/632814

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/632814
Committed: https://git.openstack.org/cgit/openstack/stx-upstream/commit/?id=679b884c76fe00c649cb143bddcf42b3e16b0d85
Submitter: Zuul
Branch: f/stein

commit cf672440526fa0b3d4d27c9868523dedd294a0cb
Author: Don Penney <email address hidden>
Date: Wed Jan 23 11:47:26 2019 -0500

    Update master stx-ceilometer to use panko-5.0.0

    This commit updates the master docker image file for
    ceilometer to specify install of panko-5.0.0 explicitly.

    Depends-On: I854b75577b6dbc3f0a8171190f5a1aa839412dc8
    Change-Id: I4db80c4bcaaa09046285e9ee0af34db1be54a606
    Story: 2004520
    Task: 29047
    Signed-off-by: Don Penney <email address hidden>

commit 6e74844f720eee4bd6d7b4067d3b6db3b40caa49
Author: Gerry Kopec <email address hidden>
Date: Wed Jan 9 20:52:05 2019 -0500

    Update helm charts to support cold migration

    To enable cold migration, need to update nova charts in openstack-helm
    and helm-toolkit chart in openstack-helm-infra. These changes build
    on existing upstream components which attempt to add a second container
    to the nova-compute pod which creates a sshd process listening on port
    8022.

    Nova chart changes include:
    - Fix bug in ssh-config mapping so config file is generated properly in
      /root/.ssh/config in nova-compute container.
    - Move private key from sshd container to nova-compute container.
    - Map private and public ssh keys to new configmap-ssh which will
      default to acceptable file permissions (400) for ssh. Keys will be
      provided in overrides.
    - Add additional config to /etc/ssh/sshd_config to allow passwordless
      root logins over appropriate subnet passed in from overrides. This
      is the same as what is done in nova puppet currently.
    - Remove chmods from sshd bash script as they are failing. Function is
      replaced by configmap-ssh.

    To enable cold migration in nova helm chart, we need to allow multiple
    containers within the same daemonset pod. This requires a patch to
    the helm-toolkit _daemonset_overrides template to remove upstream
    restriction. This issue is tracked upstream by storyboard 2003876.

    These changes should be upstreamed but may require further refinement.

    Story: 2003909
    Task: 28927
    Change-Id: Id789ba051cec019e8b7564c713cf1b5296ecf9f6
    Signed-off-by: Gerry Kopec <email address hidden>

commit 9e649fabd82b062fe6fdea3e5b0e93385d48e361
Author: Don Penney <email address hidden>
Date: Fri Jan 18 16:03:45 2019 -0500

    Add e2fsprogs to stx-nova package list

    This update adds e2fsprogs to the pike and master docker
    image build directives files for stx-nova, to provide the
    mkfs utilities required for ephemeral storage support.

    Change-Id: If9b901696169d7d157a37d6b96f7b8c4db0a24a5
    Closes-Bug: 1812432
    Signed-off-by: Don Penney <email address hidden>

commit 2462f3015a5601a9398400beef2fadc56cbaa89f
Author: Tao Liu <email address hidden>
Date: Sat Jan 12 23:21:53 2019 -0500

    Clean up the stale files

    The keystone-api pod gets stuck in CrashLoopBackOff...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos76
Review: https://review.openstack.org/636367

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/636367
Committed: https://git.openstack.org/cgit/openstack/stx-upstream/commit/?id=e0ae063497c04bd0cc9878b3e471e4f2e0eae516
Submitter: Zuul
Branch: f/centos76

commit ab3945fcc78cc912b0db0dc0e8ecffc75cef4568
Author: Chris Friesen <email address hidden>
Date: Wed Feb 6 17:47:47 2019 -0600

    fix type error to streamline single-replica mariadb startup

    The mariadb startup script was trying to optimize the single-replica
    case but missed the fact that the variable it was checking was a
    string rather than an int.

    Converting it to an int before doing the comparison makes it work
    as expected.

    Change-Id: I0f920b52c5cc92672a71ee3db3d7f8e5700fb709
    Signed-off-by: Chris Friesen <email address hidden>
    Story: 2004712
    Task: 29385

commit aac168b3984f715f06661686869e70f1b44b57c0
Author: Irina Mihai <email address hidden>
Date: Mon Feb 4 18:06:14 2019 +0000

    Glance chart: add images pool replication

    - pass new pool replication parameter to the storage init script
    - set images pool replication in the storage init script

    Change-Id: If0ecaeb12afd78caacdbb9b85cda8d6cfda00dea
    Story: 2004520
    Task: 29324
    Depends-on: I120e196ec3acca1f57b717cc4548c7c549fac738
    Signed-off-by: Irina Mihai <email address hidden>

commit 185708cded74645cdf589009d3324d5d9adf1437
Author: Kristal Dale <email address hidden>
Date: Thu Jan 31 16:06:26 2019 -0800

    Update config for release notes to include project name

    Update conf.py for release notes to include the project
    variable, set to the project name. This is so the string
    above the left nav renders the project name.

    Story: 2004900
    Task: 29234

    Change-Id: Id6b86d8c14628c0e04729e3e54b1718a51fe178b
    Signed-off-by: Kristal Dale <email address hidden>

commit ff598b27e1f4dbc9b993d9c9360fbe06e2d2ed8c
Author: Tao Liu <email address hidden>
Date: Thu Jan 31 09:41:48 2019 -0600

    Configurable Host HTTP/HTTPS Port Binding

    change platform horizon to bind to port 8008

    Story: 2004642
    Task: 29265
    Depends-On: https://review.openstack.org/#/c/634237/

    Change-Id: Id6890c8e62d31225969e4c0bb23c78ccb66a3517
    Signed-off-by: Tao Liu <email address hidden>

commit cf672440526fa0b3d4d27c9868523dedd294a0cb
Author: Don Penney <email address hidden>
Date: Wed Jan 23 11:47:26 2019 -0500

    Update master stx-ceilometer to use panko-5.0.0

    This commit updates the master docker image file for
    ceilometer to specify install of panko-5.0.0 explicitly.

    Depends-On: I854b75577b6dbc3f0a8171190f5a1aa839412dc8
    Change-Id: I4db80c4bcaaa09046285e9ee0af34db1be54a606
    Story: 2004520
    Task: 29047
    Signed-off-by: Don Penney <email address hidden>

commit 6e74844f720eee4bd6d7b4067d3b6db3b40caa49
Author: Gerry Kopec <email address hidden>
Date: Wed Jan 9 20:52:05 2019 -0500

    Update helm charts to support cold migration

    To enable cold migration, need to update nova charts in openstack-helm
    and helm-toolkit chart in openstack-helm-infra. These changes build
    o...