train ovb jobs fail modify image (repo-setup) wrong fs type, bad option, bad superblock on /dev/nbd0

Bug #2019192 reported by Marios Andreou
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
In Progress
Critical
Unassigned

Bug Description

At [1][2] the periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train fails during repo setup/modify image (injecting repos into the deploy image) with trace that looks like (also fs35 at [3])

        * 2023-05-10 12:35:31.637924 | primary | TASK [modify-image : Mount image] **********************************************
2023-05-10 12:35:31.637982 | primary | Wednesday 10 May 2023 12:35:31 +0000 (0:00:03.196) 0:09:15.541 *********
2023-05-10 12:35:32.976326 | primary | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "set -ex\nif type tripleo-mount-image >/dev/null; then\n tripleo-mount-image -a /home/zuul/overcloud-full.raw -m /tmp/tmp.b5XbFYcE6J\nelse\n # stable branches do not have tripleo-mount-image, and only use\n # partition images\n modprobe nbd\n if qemu-img info --output json /home/zuul/overcloud-full.raw |grep '\"format\": \"raw\"' ; then\n image_format='--format raw'\n elif qemu-img info --output json /home/zuul/overcloud-full.raw |grep '\"format\": \"qcow2\"' ; then\n image_format='--format qcow2'\n else\n image_format=''\n fi\n qemu-nbd $image_format --connect /dev/nbd0 /home/zuul/overcloud-full.raw\n mount /dev/nbd0 /tmp/tmp.b5XbFYcE6J\nfi\n", "delta": "0:00:00.114529", "end": "2023-05-10 12:35:32.811237", "msg": "non-zero return code", "rc": 32, "start": "2023-05-10 12:35:32.696708", "stderr": "+ type tripleo-mount-image\n/bin/sh: line 1: type: tripleo-mount-image: not found\n+ modprobe nbd\n+ qemu-img info --output json /home/zuul/overcloud-full.raw\n+ grep '\"format\": \"raw\"'\n+ image_format='--format raw'\n+ qemu-nbd --format raw --connect /dev/nbd0 /home/zuul/overcloud-full.raw\n+ mount /dev/nbd0 /tmp/tmp.b5XbFYcE6J\nmount: /tmp/tmp.b5XbFYcE6J: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.", "stderr_lines": ["+ type tripleo-mount-image", "/bin/sh: line 1: type: tripleo-mount-image: not found", "+ modprobe nbd", "+ qemu-img info --output json /home/zuul/overcloud-full.raw", "+ grep '\"format\": \"raw\"'", "+ image_format='--format raw'", "+ qemu-nbd --format raw --connect /dev/nbd0 /home/zuul/overcloud-full.raw", "+ mount /dev/nbd0 /tmp/tmp.b5XbFYcE6J", "mount: /tmp/tmp.b5XbFYcE6J: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error."], "stdout": " \"format\": \"raw\",", "stdout_lines": [" \"format\": \"raw\","]}

        * 2023-05-10 12:35:33.176489 | primary | "stderr": "+ type tripleo-mount-image\n/bin/sh: line 1: type: tripleo-mount-image: not found\n+ modprobe nbd\n+ qemu-img info --output json /home/zuul/overcloud-full.raw\n+ grep '\"format\": \"raw\"'\n+ image_format='--format raw'\n+ qemu-nbd --format raw --connect /dev/nbd0 /home/zuul/overcloud-full.raw\n+ mount /dev/nbd0 /tmp/tmp.b5XbFYcE6J\nmount: /tmp/tmp.b5XbFYcE6J: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.",

[1] https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train/16b7186/job-output.txt
[2] https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train/0c592ce/job-output.txt
[3] https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/a3b5664/job-output.txt

Revision history for this message
Douglas Viroel (dviroel) wrote :

Latest buildset didn't reproduce the error reported [1]. We will keep an eye on next runs before actually close this bug.

[1] https://review.rdoproject.org/zuul/buildset/03f8575bd6074a7d99b72fde5b203bf0

Revision history for this message
Douglas Viroel (dviroel) wrote :
Revision history for this message
Soniya Murlidhar Vyas (svyas) wrote (last edit ):
Revision history for this message
Soniya Murlidhar Vyas (svyas) wrote :
Revision history for this message
Soniya Murlidhar Vyas (svyas) wrote :
Revision history for this message
Harald Jensås (harald-jensas) wrote :

Notes:
{
  "changed": true,
  "cmd": "set -ex\nif type tripleo-mount-image >/dev/null; then\n tripleo-mount-image -a /home/zuul/overcloud-full.raw -m /tmp/tmp.hgGhG1Uhzi\nelse\n # stable branches do not have tripleo-mount-image, and only use\n # partition images\n modprobe nbd\n if qemu-img info --output json /home/zuul/overcloud-full.raw |grep '\"format\": \"raw\"' ; then\n image_format='--format raw'\n elif qemu-img info --output json /home/zuul/overcloud-full.raw |grep '\"format\": \"qcow2\"' ; then\n image_format='--format qcow2'\n else\n image_format=''\n fi\n qemu-nbd --connect /dev/nbd0 /home/zuul/overcloud-full.raw\n mount /dev/nbd0 /tmp/tmp.hgGhG1Uhzi\nfi\n",
  "delta": "0:00:00.124561",
  "end": "2023-05-29 12:32:03.920319",
  "msg": "non-zero return code",
  "rc": 32,
  "start": "2023-05-29 12:32:03.795758",
  "stderr": "+ type tripleo-mount-image\n/bin/sh: line 1: type: tripleo-mount-image: not found\n+ modprobe nbd\n+ qemu-img info --output json /home/zuul/overcloud-full.raw\n+ grep '\"format\": \"raw\"'\n+ image_format='--format raw'\n+ qemu-nbd --format raw --connect /dev/nbd0 /home/zuul/overcloud-full.raw\n+ mount /dev/nbd0 /tmp/tmp.hgGhG1Uhzi\nmount: /tmp/tmp.hgGhG1Uhzi: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.",
  "stderr_lines": [
    "+ type tripleo-mount-image",
    "/bin/sh: line 1: type: tripleo-mount-image: not found",
    "+ modprobe nbd",
    "+ qemu-img info --output json /home/zuul/overcloud-full.raw",
    "+ grep '\"format\": \"raw\"'",
    "+ image_format='--format raw'",
    "+ qemu-nbd --format raw --connect /dev/nbd0 /home/zuul/overcloud-full.raw",
    "+ mount /dev/nbd0 /tmp/tmp.hgGhG1Uhzi",
    "mount: /tmp/tmp.hgGhG1Uhzi: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error."
  ],
  "stdout": " \"format\": \"raw\",",
  "stdout_lines": [
    " \"format\": \"raw\","
  ]
}

Revision history for this message
Harald Jensås (harald-jensas) wrote :

I tried to reproduce this using - https://images.rdoproject.org/centos8/train/rdo_trunk/tripleo-ci-testing/overcloud-full.tar

The image can be converted and mounted using nbd just fine on my system.

On my system I see this logged in dmesg when connecting the nbd device:
[146056.667136] nbd0: detected capacity change from 0 to 10547200

And then this in dmesg when mounting the filesystem:
[146078.599160] XFS (nbd0): Mounting V5 Filesystem 19d3e5c1-45f0-43f4-8cad-ee89fb197cdb
[146078.606937] XFS (nbd0): Ending clean mount

I cannot see anything related to nbd in:
  https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/72722f2/logs/undercloud/var/log/extra/dmesg.txt.gz

Can we reproduce and hold the node?

Revision history for this message
Soniya Murlidhar Vyas (svyas) wrote :

Sure, we can hold the node for you

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-quickstart-extras (master)
Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
Soniya Murlidhar Vyas (svyas) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-quickstart-extras (master)

Change abandoned by "Harald Jensås <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/884993

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.