systemd sends tons of useless PropertiesChanged messages when a mount happens

Bug #1924691 reported by Li Zhou
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Li Zhou

Bug Description

Brief Description
-----------------
It appears that a single "mount" operation on a single file system (including a "mount -o remount") makes systemd send 2 PropertiesChanged message per mount point seen on the system (even if the mount point is not touched).
This creates a huge systemd load, and potentially a huge load on DBus.

This is a known issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1793527

Severity
--------
Major

Steps to Reproduce
------------------
Refer to [https://bugzilla.redhat.com/show_bug.cgi?id=1793527]

Expected Behavior
------------------
Make systemd performance more independent of the number of mounts

Actual Behavior
----------------
Each mount point creates extreme amount of systemd sendmsg and cause a heavy load on it.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
One node system

Test Activity
-------------
Developer Testing

Workaround
----------
None

Li Zhou (lzhou2)
Changed in starlingx:
assignee: nobody → Li Zhou (lzhou2)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to integ (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/integ/+/786600

Changed in starlingx:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tools (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/tools/+/786601

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config-files (master)
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Marking for stx.6.0 as this is an issue with very large deployments

tags: added: stx.distro.other
Changed in starlingx:
importance: Undecided → Medium
tags: added: stx.6.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tools (master)

Reviewed: https://review.opendev.org/c/starlingx/tools/+/786601
Committed: https://opendev.org/starlingx/tools/commit/e31e0dda7a4c09143d41cd518ab97ea6112d7fb5
Submitter: "Zuul (22348)"
Branch: master

commit e31e0dda7a4c09143d41cd518ab97ea6112d7fb5
Author: Li Zhou <email address hidden>
Date: Tue Apr 13 04:53:50 2021 -0400

    systemd: Upgrade to version 219-78.el7_9.3

    Refer the lst entries to the new version.

    Partial-Bug: #1924691
    Signed-off-by: Li Zhou <email address hidden>
    Change-Id: I557eff6a47f341cc67de02fd59024b28bb6cac84

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to integ (master)

Reviewed: https://review.opendev.org/c/starlingx/integ/+/786600
Committed: https://opendev.org/starlingx/integ/commit/4850ab86da1cecca239d2ffa6dded4c0946e8a43
Submitter: "Zuul (22348)"
Branch: master

commit 4850ab86da1cecca239d2ffa6dded4c0946e8a43
Author: Li Zhou <email address hidden>
Date: Tue Apr 13 04:34:32 2021 -0400

    systemd: Upgrade to version 219-78.el7_9.3

    This fixes the issue of systemd sending tons of useless
    PropertiesChanged messages when a mount happens as described in:
    https://bugzilla.redhat.com/show_bug.cgi?id=1793527

    Depends-On: https://review.opendev.org/c/starlingx/tools/+/786601
    Partial-Bug: #1924691
    Signed-off-by: Li Zhou <email address hidden>
    Change-Id: I3596303d77211a135e8559a05806395328725cde

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config-files (master)

Reviewed: https://review.opendev.org/c/starlingx/config-files/+/786602
Committed: https://opendev.org/starlingx/config-files/commit/efb718e03171580c43702a01f7c103e590832ab7
Submitter: "Zuul (22348)"
Branch: master

commit efb718e03171580c43702a01f7c103e590832ab7
Author: Li Zhou <email address hidden>
Date: Tue Apr 13 04:48:46 2021 -0400

    systemd: Upgrade to version 219-78.el7_9.3

    Change the BuildRequires to refer to the new systemd version.

    Depends-On: https://review.opendev.org/c/starlingx/tools/+/786601
    Closes-Bug: #1924691
    Signed-off-by: Li Zhou <email address hidden>
    Change-Id: I76169b7fd85069e26cfb37de8889cea006c57238

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tools (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/tools/+/792229

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tools (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/tools/+/792229
Reason: Updated merge coming

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tools (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/tools/+/793627

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config-files (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/config-files/+/793634

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to integ (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/integ/+/793754

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config-files (f/centos8)
Download full text (4.3 KiB)

Reviewed: https://review.opendev.org/c/starlingx/config-files/+/793634
Committed: https://opendev.org/starlingx/config-files/commit/03c3f68b2a1477da3dbc7d351e8bf9e2cff2acf1
Submitter: "Zuul (22348)"
Branch: f/centos8

commit e82d1b9e70dd50fbec76db7cfc51e433c5b6bf9e
Author: Jim Gauld <email address hidden>
Date: Thu May 6 12:14:39 2021 -0400

    Configure io-scheduler udev rules for ETCD and HW-RAID

    This configures io-scheduler udev rules for etcd and hw-raid
    performance.

    This sets the io-scheduler to 'cfq' tuned parameters for 'controller'
    nodetype with root file-system disk.

    This sets io-scheduler to 'noop' for HW-RAID Dell PowerEdge R720,
    this was a missing commit from pre-starlingx.

    Partial-Bug: 1927515
    Depends-On: https://review.opendev.org/c/starlingx/utilities/+/790094
    Signed-off-by: Jim Gauld <email address hidden>
    Change-Id: Iaf1de8d962d1e8d253c72e680370666a2aed8c8e

commit efb718e03171580c43702a01f7c103e590832ab7
Author: Li Zhou <email address hidden>
Date: Tue Apr 13 04:48:46 2021 -0400

    systemd: Upgrade to version 219-78.el7_9.3

    Change the BuildRequires to refer to the new systemd version.

    Depends-On: https://review.opendev.org/c/starlingx/tools/+/786601
    Closes-Bug: #1924691
    Signed-off-by: Li Zhou <email address hidden>
    Change-Id: I76169b7fd85069e26cfb37de8889cea006c57238

commit 7877dbc6baec4e3214a12ac0ae44db5491a22e9d
Author: Andy Ning <email address hidden>
Date: Fri Apr 16 10:46:13 2021 -0400

    Enforce "cannot reuse the last 2 passwords" password rule

    Currently the "remember" attribute in pam_pwhistory configuration
    is set to "2", which enforces "cannot reuse the last 1 passwords"
    in history instead of "cannot reuse the last 2 passwords" stated
    in security document.

    This update changed "remember" attribute to "3" so that the rule
    complies with the document.

    Closes-Bug: 1924772
    Signed-off-by: Andy Ning <email address hidden>
    Change-Id: I340152f8b8a572bc1e86f1eb4a14eb8e392f6334

commit e87383f6c328efeab2a9407daa33076a85739b96
Author: Eric MacDonald <email address hidden>
Date: Tue Apr 6 08:44:26 2021 -0400

    Comment out 'dateext' setting in logrotate.conf file

    This update comments out the 'dateext' setting to avoid
    log files being rotated with date as a default.

    Test Plan:

    PASS: Verify log rotation config files that don't
          specifically set dateext option are rotated
          by number rather than date.
    PASS: Verify system install

    Partial-Bug: 1918979
    Signed-off-by: Eric MacDonald <email address hidden>
    Change-Id: Ib68d86d1ec3f15abedce4c4059c3a8ec34b7d196

commit 35160afbdada2efe0ff567dd94ca1419903c87ad
Author: Nicolas Alvarez <email address hidden>
Date: Tue Dec 8 17:26:18 2020 -0300

    Disable SNMP Host-Based from config-files repo.

    Due to SNMP is going to be containerized, we disable
    it from starlingx/config-files repo.

    Story: 2008132
    Task: 41381
    Depends-On: https://review.opendev.org/765381
    Signed-off-by: Nicolas Alvarez <nic...

Read more...

tags: added: in-f-centos8
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tools (f/centos8)
Download full text (30.4 KiB)

Reviewed: https://review.opendev.org/c/starlingx/tools/+/793627
Committed: https://opendev.org/starlingx/tools/commit/d701c6f896dfe440566cc942e3dd71be1f19ae5d
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 7b5f3a45e663866a3c0ca3ca86eb3c92bc7f0210
Author: Scott Little <email address hidden>
Date: Wed May 5 09:56:33 2021 -0400

    fix bad flockflock url pt 2

    A stray '}' character found it's way into my prior update
    titled 'fix bad flockflock url' after testing. The result was
    the following error

    sed: -e expression #1, char 15: unexpected `}'

    This removes the unwanted '}', restoring the prior update
    to its intended form.

    Closes-bug: 1926987
    Signed-off-by: Scott Little <email address hidden>
    Change-Id: I48f4721ccaf121679916b01747243deedf5836cd

commit ac05493480f6df6f31d071d29380c1b4f35b70a9
Author: Scott Little <email address hidden>
Date: Tue May 4 12:42:36 2021 -0400

    fix git-review within docker build environment

    'tb create' fails to create a build environment since
    upstream git-review was updated of Apr 26.

    Fix is to install/update pbr ahead of git-review.

    Also, to reduce the likelyhood of this recurring, lock
    down specific versions of the pypi supplied tools we
    know to work.

    Closes-bug: 1927137
    Signed-off-by: Scott Little <email address hidden>
    Change-Id: Ib9fe6fd33de4d637f254ac421cc0427ee6131b65

commit b96ebc83d859a4a7802a462504817ecec6182a7b
Author: Scott Little <email address hidden>
Date: Mon May 3 13:16:53 2021 -0400

    fix bad flockflock url

    download_mirror.sh fails due to a bad path containing
    ‘stx-tools/centos-mirror-tools/config/centos/flockflock’

    The path is constructed, and the trigger is when an EOL is missing
    from a centos_build_layer.cfg file, causing 'cat' to merge the last
    line of the offending file with the first line of the next file.

    Switch 'cat' to 'grep', which will always ensure an EOL is present.
    Along the way, we can filter out empty lines and comments.

    Closes-bug: 1926987
    Signed-off-by: Scott Little <email address hidden>
    Change-Id: I2404b3415f0f3e2f395c2bcb7a527aa01a488f61

commit 4c3ee114bcbff710c2049626044dd1ddc756cbd9
Author: Joe Slater <email address hidden>
Date: Tue Apr 27 18:50:53 2021 -0400

    screen: fix CVE-2021-26937 segfault

    Advance to screen-4.1.0-0.27.20120314git3c2946.el7_9.x86_64.rpm.

    Closes-bug: 1926372
    Change-Id: I41834e7b1e16153b0632751f59f7ac9f503389da
    Signed-off-by: Joe Slater <email address hidden>

commit e31e0dda7a4c09143d41cd518ab97ea6112d7fb5
Author: Li Zhou <email address hidden>
Date: Tue Apr 13 04:53:50 2021 -0400

    systemd: Upgrade to version 219-78.el7_9.3

    Refer the lst entries to the new version.

    Partial-Bug: #1924691
    Signed-off-by: Li Zhou <email address hidden>
    Change-Id: I557eff6a47f341cc67de02fd59024b28bb6cac84

commit 26db2859dd3a5c060c337b886fd16c4d2d9f93af
Author: Scott Little <email address hidden>
Date: Mon Apr 12 11:21:31 2021 -0400

    Replace basearch references in y...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to integ (f/centos8)
Download full text (37.0 KiB)

Reviewed: https://review.opendev.org/c/starlingx/integ/+/793754
Committed: https://opendev.org/starlingx/integ/commit/a13966754d4e19423874ca31bf1533f057380c52
Submitter: "Zuul (22348)"
Branch: f/centos8

commit b310077093fd567944c6a46b7d0adcabe1f2b4b9
Author: Mihnea Saracin <email address hidden>
Date: Sat May 22 18:19:54 2021 +0300

    Fix resize of filesystems in puppet logical_volume

    After system reinstalls there is stale data on the disk
    and puppet fails when resizing, reporting some wrong filesystem
    types. In our case docker-lv was reported as drbd when
    it should have been xfs.

    This problem was solved in some cases e.g:
    when doing a live fs resize we wipe the last 10MB
    at the end of partition:
    https://opendev.org/starlingx/stx-puppet/src/branch/master/puppet-manifests/src/modules/platform/manifests/filesystem.pp#L146

    Our issue happened here:
    https://opendev.org/starlingx/stx-puppet/src/branch/master/puppet-manifests/src/modules/platform/manifests/filesystem.pp#L65
    Resize can happen at unlock when a bigger size is detected for the
    filesystem and the 'logical_volume' will resize it.
    To fix this we have to wipe the last 10MB of the partition after the
    'lvextend' cmd in the 'logical_volume' module.

    Tested the following scenarios:

    B&R on SX with default sizes of filesystems and cgts-vg.

    B&R on SX with with docker-lv of size 50G, backup-lv also 50G and
    cgts-vg with additional physical volumes:

    - name: cgts-vg
        physicalVolumes:
        - path: /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0
        size: 50
        type: partition
        - path: /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0
        size: 30
        type: partition
        - path: /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0
        type: disk

    B&R on DX system with backup of size 70G and cgts-vg
    with additional physical volumes:

    physicalVolumes:
    - path: /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0
        size: 50
        type: partition
    - path: /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0
        size: 30
        type: partition
    - path: /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0
        type: disk

    Closes-Bug: 1926591
    Change-Id: I55ae6954d24ba32e40c2e5e276ec17015d9bba44
    Signed-off-by: Mihnea Saracin <email address hidden>

commit 3225570530458956fd642fa06b83360a7e4e2e61
Author: Mihnea Saracin <email address hidden>
Date: Thu May 20 14:33:58 2021 +0300

    Execute once the ceph services script on AIO

    The MTC client manages ceph services via ceph.sh which
    is installed on all node types in
    /etc/service.d/{controller,worker,storage}/ceph.sh

    Since the AIO controllers have both controller and worker
    personalities, the MTC client will execute the ceph script
    twice (/etc/service.d/worker/ceph.sh,
    /etc/service.d/controller/ceph.sh).
    This behavior will generate some issues.

    We fix this by exiting the ceph script if it is the one from
    /etc/services.d/worker on AIO systems.

    Closes-Bug: 1928934
    Change-Id: I3e4dc313cc3764f870b8f6c640a60338...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.