drbd filesystems not resized during bootstrap

Bug #1920245 reported by David Sullivan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
David Sullivan

Bug Description

Brief Description
-----------------
drbd filesystems are under sized after the bootstrap playbook is run. While the underlying lvs are sized correctly the drbd filesystem remains unchanged. eg platform will remain at 1gb instead of 10gb.

Severity
--------
Major

Steps to Reproduce
------------------
Bootstrap a system. Check lvs vs drbd-overview

Expected Behavior
------------------
The reported sizes for the lvs and drbd filesystem should match.

Actual Behavior
----------------
The drbd filesystem remain at a smaller size

Reproducibility
---------------
Reproducible

System Configuration
--------------------
All systems

Branch/Pull Time/Commit
-----------------------
2021-03-18_00-00-06

Last Pass
---------
StX4

Timestamp/Logs
--------------
2021-03-15T22:34:02.000 localhost ansible-command: info Invoked with warn=True executable=None _uses_shell=False _raw_params=drbdadm -- --assume-peer-has-space resize drbd-pgsql/0 removes=None argv=None creates=None chdir=None stdin=None
2021-03-15T22:34:05.000 localhost ansible-command: info Invoked with warn=True executable=None _uses_shell=False _raw_params=drbdadm -- --assume-peer-has-space resize drbd-platform/0 removes=None argv=None creates=None chdir=None stdin=None
2021-03-15T22:34:09.000 localhost ansible-command: info Invoked with warn=True executable=None _uses_shell=False _raw_params=drbdadm -- --assume-peer-has-space resize drbd-etcd/0 removes=None argv=None creates=None chdir=None stdin=None
2021-03-15T22:34:12.000 localhost ansible-command: info Invoked with warn=True executable=None _uses_shell=False _raw_params=drbdadm -- --assume-peer-has-space resize drbd-dockerdistribution/0 removes=None argv=None creates=None chdir=None stdin=None

Test Activity
-------------
Regression Testing

Workaround
----------
Manually run resize2fs /dev/drbdX

Revision history for this message
David Sullivan (dsullivanwr) wrote :
Changed in starlingx:
assignee: nobody → David Sullivan (dsullivanwr)
status: New → In Progress
status: In Progress → Fix Released
Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Medium
tags: added: stx.5.0 stx.config
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (master)

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/784860
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/5b286734637ca6f503a62131b309829d2f308fed
Submitter: "Zuul (22348)"
Branch: master

commit 5b286734637ca6f503a62131b309829d2f308fed
Author: David Sullivan <email address hidden>
Date: Mon Apr 5 17:01:48 2021 -0500

    Remove drbd resize actions from bootstrap playbook

    drbd resize actions can cause failures during unlock. Avoid all drbd
    resize actions during the bootstrap playbook. Pass the correct
    filesystem sizes to the bootstrap manifest. Use the collected drbd
    filesystem sizes during the restore/upgrade bootstrap manifest.

    Closes-Bug: 1920245
    Signed-off-by: David Sullivan <email address hidden>
    Change-Id: I235575ad40ba84298d3db5b39ce7861a143c78a8

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to integ (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/integ/+/787294

Changed in starlingx:
status: Fix Released → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to integ (master)

Reviewed: https://review.opendev.org/c/starlingx/integ/+/787294
Committed: https://opendev.org/starlingx/integ/commit/e62b1a53b9148738a7c36355b19607d6e6f3d0d7
Submitter: "Zuul (22348)"
Branch: master

commit e62b1a53b9148738a7c36355b19607d6e6f3d0d7
Author: David Sullivan <email address hidden>
Date: Tue Apr 20 17:32:45 2021 -0500

    Unmount all targets during drbd stop

    When stopping drbd, we need to unmount targets from each device.
    Devices with multiple mountpoints can fail to unmount, leading to
    metadata corruption. Add --all-targets to the umount command.

    Closes-Bug: 1920245
    Signed-off-by: David Sullivan <email address hidden>
    Change-Id: Ic1b4583c72a0dd256724b8672dbb59126273330b

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
Ghada Khalil (gkhalil) wrote :

@David Sullivan, This is tagged for stx.5.0. Please cherrypick this change to the r/stx.5.0 release branch asap and no later than April 30

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to integ (r/stx.5.0)

Fix proposed to branch: r/stx.5.0
Review: https://review.opendev.org/c/starlingx/integ/+/787760

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-playbooks (r/stx.5.0)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (f/centos8)

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/785199
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/87cc6e9c9fe3ebe37fb1f364f4ced2a818a60e98
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 87cc6e9c9fe3ebe37fb1f364f4ced2a818a60e98
Author: David Sullivan <email address hidden>
Date: Fri Mar 19 10:36:55 2021 -0500

    drbd filesystems not resized during bootstrap

    Remove drbd resize_result check on resize2fs operation. Both operations
    should run when requested. These commands will return 0 when the disk is
    resized and will return 0 if the disk is already correctly sized. Any
    non-zero return code should fail the playbook.

    Closes-Bug: 1920245
    Signed-off-by: David Sullivan <email address hidden>
    Change-Id: I946f63b2886b5377494e658a59586005c27ec2d2
    (cherry picked from commit 2ffd9c4bff6bd736f35cc1fb2fd3b0e25c9ef8f2)

tags: added: in-f-centos8
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (r/stx.5.0)

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/787769
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/9dac0f83b5225f32d0687f8f44277e5a99cf2443
Submitter: "Zuul (22348)"
Branch: r/stx.5.0

commit 9dac0f83b5225f32d0687f8f44277e5a99cf2443
Author: David Sullivan <email address hidden>
Date: Mon Apr 5 17:01:48 2021 -0500

    Remove drbd resize actions from bootstrap playbook

    drbd resize actions can cause failures during unlock. Avoid all drbd
    resize actions during the bootstrap playbook. Pass the correct
    filesystem sizes to the bootstrap manifest. Use the collected drbd
    filesystem sizes during the restore/upgrade bootstrap manifest.

    Closes-Bug: 1920245
    Signed-off-by: David Sullivan <email address hidden>
    Change-Id: I235575ad40ba84298d3db5b39ce7861a143c78a8

Ghada Khalil (gkhalil)
tags: added: stx.cherrypickneeded
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to integ (r/stx.5.0)

Reviewed: https://review.opendev.org/c/starlingx/integ/+/787760
Committed: https://opendev.org/starlingx/integ/commit/a72cceb1bf796984ba0db103b77187da98a15dfa
Submitter: "Zuul (22348)"
Branch: r/stx.5.0

commit a72cceb1bf796984ba0db103b77187da98a15dfa
Author: David Sullivan <email address hidden>
Date: Tue Apr 20 17:32:45 2021 -0500

    Unmount all targets during drbd stop

    When stopping drbd, we need to unmount targets from each device.
    Devices with multiple mountpoints can fail to unmount, leading to
    metadata corruption. Add --all-targets to the umount command.

    Closes-Bug: 1920245
    Signed-off-by: David Sullivan <email address hidden>
    Change-Id: Ic1b4583c72a0dd256724b8672dbb59126273330b

Ghada Khalil (gkhalil)
tags: added: in-r-stx50
removed: stx.cherrypickneeded
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-playbooks (f/centos8)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to integ (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/integ/+/793754

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-playbooks (f/centos8)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ansible-playbooks (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/792195

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (f/centos8)
Download full text (52.5 KiB)

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794324
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/163ec9989cc7360dba4c572b2c43effd10306048
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 4e96b762f549aadb0291cc9bcf3352ae923e94eb
Author: Mihnea Saracin <email address hidden>
Date: Sat May 22 15:48:19 2021 +0000

    Revert "Restore host filesystems with collected sizes"

    This reverts commit 255488739efa4ac072424b19f2dbb7a3adb0254e.

    Reason for revert: Did a rework to fix https://bugs.launchpad.net/starlingx/+bug/1926591. The original problem was in puppet, and this fix in ansible was not good enough, it generated some other problems.

    Change-Id: Iea79701a874effecb7fe995ac468d50081d1a84f
    Depends-On: I55ae6954d24ba32e40c2e5e276ec17015d9bba44

commit c064aacc377c8bd5336ceab825d4bcbf5af0b5e8
Author: Angie Wang <email address hidden>
Date: Fri May 21 21:28:02 2021 -0400

    Ensure apiserver keys are present before extract from tarball

    This is to fix the upgrade playbook issue that happens during
    AIO-SX upgrade from stx4.0 to stx5.0 which introduced by
    https://review.opendev.org/c/starlingx/ansible-playbooks/+/792093.
    The apiserver keys are not available in stx4.0 side so we need
    to ensure the keys under /etc/kubernetes/pki are present in the
    backed-up tarball before extracting, otherwise playbook fails
    because the keys are not found in the archive.

    Change-Id: I8602f07d1b1041a7fd3fff21e6f9a422b9784ab5
    Closes-Bug: 928925
    Signed-off-by: Angie Wang <email address hidden>

commit 0261f22ff7c23d2a8608fe3b51725c9f29931281
Author: Don Penney <email address hidden>
Date: Thu May 20 23:09:07 2021 -0400

    Update SX to DX migration to wait for coredns config

    This commit updates the SX to DX migration playbook to wait after
    modifying the system mode to duplex until the runtime manifest that
    updates coredns config has completed. The playbook will wait for up to
    20 minutes to allow for the possibilty that sysinv has multiple
    runtime manifests queued up, each of which could take several minutes.

    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/792494
    Depends-On: https://review.opendev.org/c/starlingx/config/+/792496
    Change-Id: I3bf94d3493ae20eeb16b3fdcb27576ee18c0dc4d
    Closes-Bug: 1929148
    Signed-off-by: Don Penney <email address hidden>

commit 7c4f17bd0d92fc1122823211e1c9787829d206a9
Author: Daniel Safta <email address hidden>
Date: Wed May 19 09:08:16 2021 +0000

    Fixed missing apiserver-etcd-client certs

    When controller-1 is the active controller
    the backup archive does not contain
    /etc/etcd/apiserver-etcd-client.{crt, key}

    This change adds a new task which brings
    the certs from /etc/kubernetes/pki

    Closes-bug: 1928925
    Signed-off-by: Daniel Safta <email address hidden>
    Change-Id: I3c68377603e1af9a71d104e5b1108e9582497a09

commit e221ef8fbe51aa6ca229b584fb5632fe512ad5cb
Author: David Sullivan <email address hidden>
Date: Wed May 19 16:01:27 2021 -0500

    Support boo...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to integ (f/centos8)
Download full text (37.0 KiB)

Reviewed: https://review.opendev.org/c/starlingx/integ/+/793754
Committed: https://opendev.org/starlingx/integ/commit/a13966754d4e19423874ca31bf1533f057380c52
Submitter: "Zuul (22348)"
Branch: f/centos8

commit b310077093fd567944c6a46b7d0adcabe1f2b4b9
Author: Mihnea Saracin <email address hidden>
Date: Sat May 22 18:19:54 2021 +0300

    Fix resize of filesystems in puppet logical_volume

    After system reinstalls there is stale data on the disk
    and puppet fails when resizing, reporting some wrong filesystem
    types. In our case docker-lv was reported as drbd when
    it should have been xfs.

    This problem was solved in some cases e.g:
    when doing a live fs resize we wipe the last 10MB
    at the end of partition:
    https://opendev.org/starlingx/stx-puppet/src/branch/master/puppet-manifests/src/modules/platform/manifests/filesystem.pp#L146

    Our issue happened here:
    https://opendev.org/starlingx/stx-puppet/src/branch/master/puppet-manifests/src/modules/platform/manifests/filesystem.pp#L65
    Resize can happen at unlock when a bigger size is detected for the
    filesystem and the 'logical_volume' will resize it.
    To fix this we have to wipe the last 10MB of the partition after the
    'lvextend' cmd in the 'logical_volume' module.

    Tested the following scenarios:

    B&R on SX with default sizes of filesystems and cgts-vg.

    B&R on SX with with docker-lv of size 50G, backup-lv also 50G and
    cgts-vg with additional physical volumes:

    - name: cgts-vg
        physicalVolumes:
        - path: /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0
        size: 50
        type: partition
        - path: /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0
        size: 30
        type: partition
        - path: /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0
        type: disk

    B&R on DX system with backup of size 70G and cgts-vg
    with additional physical volumes:

    physicalVolumes:
    - path: /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0
        size: 50
        type: partition
    - path: /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0
        size: 30
        type: partition
    - path: /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0
        type: disk

    Closes-Bug: 1926591
    Change-Id: I55ae6954d24ba32e40c2e5e276ec17015d9bba44
    Signed-off-by: Mihnea Saracin <email address hidden>

commit 3225570530458956fd642fa06b83360a7e4e2e61
Author: Mihnea Saracin <email address hidden>
Date: Thu May 20 14:33:58 2021 +0300

    Execute once the ceph services script on AIO

    The MTC client manages ceph services via ceph.sh which
    is installed on all node types in
    /etc/service.d/{controller,worker,storage}/ceph.sh

    Since the AIO controllers have both controller and worker
    personalities, the MTC client will execute the ceph script
    twice (/etc/service.d/worker/ceph.sh,
    /etc/service.d/controller/ceph.sh).
    This behavior will generate some issues.

    We fix this by exiting the ceph script if it is the one from
    /etc/services.d/worker on AIO systems.

    Closes-Bug: 1928934
    Change-Id: I3e4dc313cc3764f870b8f6c640a60338...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.