swift mount - nobarrier not available with xfs in the modern kernels

Bug #1800132 reported by Eric Miller
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Medium
Radosław Piliszek
Rocky
Fix Released
Medium
Radosław Piliszek
Stein
Fix Released
Medium
Radosław Piliszek
Train
Fix Released
Medium
Radosław Piliszek
Ussuri
Fix Released
Medium
Radosław Piliszek

Bug Description

I think I figured out the issue while I was investigating while writing this, but there is still an issue.

It looks like the options for mounting, configured here:
https://github.com/openstack/kolla-ansible/blob/6b372d3b751c8dff9e806d77e858b9224118753a/ansible/roles/swift/tasks/start.yml#L27

has the "nobarrier" option. This option is deprecated, and actually removed in the latest kernel (4.19.0-1.el7.elrepo.x86_64), which I'm testing against. This is described here:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1c02d502c20809a2a5f71ec16a930a61ed779b81

I left my original explanation I was going to submit below, in case it helps to see the results of the problem (it obviously didn't have anything to do with partition 2 not existing).

Eric

I wasn't sure how to word the summary, but essentially, it looks like a recent change to the kolla_toolbox (maybe?) automatically assumes that partition 2 will be assigned as a journal partition.

We have single partitions for every Swift object storage drive, labeled as KOLLA_SWIFT_DATA (as described here: https://docs.openstack.org/kolla-ansible/latest/reference/swift-guide.html#disks-with-a-partition-table-recommended

This worked perfectly until I tried the Kolla-Ansible and Kolla release for 7.0.0.

We have failures during deploy in the task "[swift : Mounting Swift disks]".

Below (end of this message) are two log entries, the first is when it succeeded on October 15, 2018, the second failed today (October 26, 2018).

Note that this is the output from "parted" on the failed node which shows the formatted partition (but only a single partition):

Model: ATA HGST HUH721010AL (scsi)
Disk /dev/sda: 10.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name Flags
 1 1049kB 10.0TB 10.0TB xfs KOLLA_SWIFT_DATA

Did something change that requires a second partition for the journal?

Eric

Succeeded:

ok: [compute001] => (item={u'fs_uuid': u'5266599c-1d9b-48d3-b036-3eae11fd95d7', u'journal_device': u'/dev/sda', u'journal': u'/dev/sda2', u'partition': u'/dev/sda1', u'partition_num': u'1', u'journal_num': 2, u'fs_label': u'd0', u'device': u'/dev/sda', u'partition_label': u'KOLLA_SWIFT_DATA', u'external_journal': False}) => {
    "changed": false,
    "dump": "0",
    "fstab": "/etc/fstab",
    "fstype": "xfs",
    "invocation": {
        "module_args": {
            "backup": false,
            "boot": true,
            "dump": null,
            "fstab": null,
            "fstype": "xfs",
            "name": "/srv/node/d0",
            "opts": "noatime,nodiratime,nobarrier,logbufs=8",
            "passno": null,
            "path": "/srv/node/d0",
            "src": "UUID=5266599c-1d9b-48d3-b036-3eae11fd95d7",
            "state": "mounted"
        }
    },
    "item": {
        "device": "/dev/sda",
        "external_journal": false,
        "fs_label": "d0",
        "fs_uuid": "5266599c-1d9b-48d3-b036-3eae11fd95d7",
        "journal": "/dev/sda2",
        "journal_device": "/dev/sda",
        "journal_num": 2,
        "partition": "/dev/sda1",
        "partition_label": "KOLLA_SWIFT_DATA",
        "partition_num": "1"
    },
    "name": "/srv/node/d0",
    "opts": "noatime,nodiratime,nobarrier,logbufs=8",
    "passno": "0",
    "src": "UUID=5266599c-1d9b-48d3-b036-3eae11fd95d7"
}

Failed:

failed: [compute001](item={
 u'fs_uuid': u'1548d474-72f1-4872-ab1f-6be07e44061d',
 u'journal_device': u'/dev/sda',
 u'journal': u'/dev/sda2',
 u'partition': u'/dev/sda1',
 u'partition_num': u'1',
 u'journal_num': 2,
 u'fs_label': u'd0',
 u'device': u'/dev/sda',
 u'partition_label': u'KOLLA_SWIFT_DATA',
 u'external_journal': False
})=>{
 "changed": false,
 "item": {
  "device": "/dev/sda",
  "external_journal": false,
  "fs_label": "d0",
  "fs_uuid": "1548d474-72f1-4872-ab1f-6be07e44061d",
  "journal": "/dev/sda2",
  "journal_device": "/dev/sda",
  "journal_num": 2,
  "partition": "/dev/sda1",
  "partition_label": "KOLLA_SWIFT_DATA",
  "partition_num": "1"
 },
 "msg": "Error mounting /srv/node/d0: mount: wrong fs type, bad option, bad superblock on /dev/sda1,\n missing codepage or helper program, or other error\n\n In some cases useful info is found in syslog - try\n dmesg | tail or so.\n"
}

Changed in kolla-ansible:
status: New → Triaged
summary: - Kolla Ansible 7.0.0 release requires two partitions for Swift object
- disks
+ swift mount - nobarrier not available with xfs
Revision history for this message
Radosław Piliszek (yoctozepto) wrote : Re: swift mount - nobarrier not available with xfs

Affects debian buster which is a supported platform (train), and will affect centos 8 (ussuri).

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

CentOS 8 only warns about it though.

Anyways, nobarrier is noop in all distros we support - best remove.

summary: - swift mount - nobarrier not available with xfs
+ swift mount - nobarrier not available with xfs in the modern kernels
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/696757

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/696757
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=21488710d68450eac4ace21bea4c5f929e0189d4
Submitter: Zuul
Branch: master

commit 21488710d68450eac4ace21bea4c5f929e0189d4
Author: Radosław Piliszek <email address hidden>
Date: Fri Nov 29 20:26:16 2019 +0100

    Fix Swift with kernel 4.19 and later

    We mount Swift volumes with xfs.
    The 'nobarrier' option we used was made noop [1]
    and deprecated [2] (with warning) in kernel 4.10.
    In 4.19 it was removed [3] resulting in an error
    when using e.g. Debian Buster as host.
    The noop patch was backported to CentOS 7 so
    it is safe to remove this option with no behavior
    change and backport to where needed.
    Ubuntu Bionic uses 4.15 which only warns.
    CentOS 8 uses 4.18 which only warns as well.
    Debian Buster uses 4.19 exactly which breaks.

    [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2291dab2c9d1880efd19469df2042e2277c8b7a4
    [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4cf4573d899cd80d8578c050061dc342f99f3a32
    [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1c02d502c20809a2a5f71ec16a930a61ed779b81

    Change-Id: I006dea21321146c7fc738d0b41c401b72d271a99
    Closes-bug: #1800132

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/697255

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/697257

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/rocky)

Reviewed: https://review.opendev.org/697257
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=bebb9bc72332b85a176e8529bbcf4fb24dc2d8fb
Submitter: Zuul
Branch: stable/rocky

commit bebb9bc72332b85a176e8529bbcf4fb24dc2d8fb
Author: Radosław Piliszek <email address hidden>
Date: Fri Nov 29 20:26:16 2019 +0100

    Fix Swift with kernel 4.19 and later

    We mount Swift volumes with xfs.
    The 'nobarrier' option we used was made noop [1]
    and deprecated [2] (with warning) in kernel 4.10.
    In 4.19 it was removed [3] resulting in an error
    when using e.g. Debian Buster as host.
    The noop patch was backported to CentOS 7 so
    it is safe to remove this option with no behavior
    change and backport to where needed.
    Ubuntu Bionic uses 4.15 which only warns.
    CentOS 8 uses 4.18 which only warns as well.
    Debian Buster uses 4.19 exactly which breaks.

    [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2291dab2c9d1880efd19469df2042e2277c8b7a4
    [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4cf4573d899cd80d8578c050061dc342f99f3a32
    [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1c02d502c20809a2a5f71ec16a930a61ed779b81

    Change-Id: I006dea21321146c7fc738d0b41c401b72d271a99
    Closes-bug: #1800132
    (cherry picked from commit 21488710d68450eac4ace21bea4c5f929e0189d4)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/stein)

Reviewed: https://review.opendev.org/697256
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=021f55b2fa7b66aa37d2859331e48b1383474da7
Submitter: Zuul
Branch: stable/stein

commit 021f55b2fa7b66aa37d2859331e48b1383474da7
Author: Radosław Piliszek <email address hidden>
Date: Fri Nov 29 20:26:16 2019 +0100

    Fix Swift with kernel 4.19 and later

    We mount Swift volumes with xfs.
    The 'nobarrier' option we used was made noop [1]
    and deprecated [2] (with warning) in kernel 4.10.
    In 4.19 it was removed [3] resulting in an error
    when using e.g. Debian Buster as host.
    The noop patch was backported to CentOS 7 so
    it is safe to remove this option with no behavior
    change and backport to where needed.
    Ubuntu Bionic uses 4.15 which only warns.
    CentOS 8 uses 4.18 which only warns as well.
    Debian Buster uses 4.19 exactly which breaks.

    [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2291dab2c9d1880efd19469df2042e2277c8b7a4
    [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4cf4573d899cd80d8578c050061dc342f99f3a32
    [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1c02d502c20809a2a5f71ec16a930a61ed779b81

    Change-Id: I006dea21321146c7fc738d0b41c401b72d271a99
    Closes-bug: #1800132
    (cherry picked from commit 21488710d68450eac4ace21bea4c5f929e0189d4)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/train)

Reviewed: https://review.opendev.org/697255
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=8a3afd4ea53601ca24ad5f6e873c0552fb0dd139
Submitter: Zuul
Branch: stable/train

commit 8a3afd4ea53601ca24ad5f6e873c0552fb0dd139
Author: Radosław Piliszek <email address hidden>
Date: Fri Nov 29 20:26:16 2019 +0100

    Fix Swift with kernel 4.19 and later

    We mount Swift volumes with xfs.
    The 'nobarrier' option we used was made noop [1]
    and deprecated [2] (with warning) in kernel 4.10.
    In 4.19 it was removed [3] resulting in an error
    when using e.g. Debian Buster as host.
    The noop patch was backported to CentOS 7 so
    it is safe to remove this option with no behavior
    change and backport to where needed.
    Ubuntu Bionic uses 4.15 which only warns.
    CentOS 8 uses 4.18 which only warns as well.
    Debian Buster uses 4.19 exactly which breaks.

    [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2291dab2c9d1880efd19469df2042e2277c8b7a4
    [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4cf4573d899cd80d8578c050061dc342f99f3a32
    [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1c02d502c20809a2a5f71ec16a930a61ed779b81

    Change-Id: I006dea21321146c7fc738d0b41c401b72d271a99
    Closes-bug: #1800132
    (cherry picked from commit 21488710d68450eac4ace21bea4c5f929e0189d4)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 9.0.0.0rc2

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 7.2.0

This issue was fixed in the openstack/kolla-ansible 7.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 8.1.0

This issue was fixed in the openstack/kolla-ansible 8.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.