40-vm-hotadd.rules attempts to set non-existent sysfs parameters

Bug #1876018 reported by Jose Miguel Parrella on 2020-04-30
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Undecided
Unassigned
Xenial
Medium
Dan Streetman
Bionic
Medium
Dan Streetman
Focal
Medium
Dan Streetman

Bug Description

[impact]

40-vm-hotadd.rules unconditionally tries onlining memory, which results in logged error messages if the memory is already online

[test case]

since this rules file restricts operation to only hyper-v or xen guests, boot a hyper-v or xen vm guest, and check for logged error msgs like:

Apr 29 22:36:46 focal01 systemd-udevd[266]: memory7: /usr/lib/udev/rules.d/40-vm-hotadd.rules:9 Failed to write ATTR{/sys/devices/system/memory/memory7/state}, ignoring: Invalid argument

alternately, to test on a vm guest other than hyper-v or xen, comment/remove the 'GOTO="vm_hotadd_end"' line from the rules file and reboot.

[regression potential]

as this adds a check before attempting to online memory for hyper-v and xen vm guests, any regression would likely involve failure to correctly online all memory on those guest platforms.

[scope]

this rule has been around for a long time, so is needed for x/b/f/g.

[original description]

In focal, udev's 40-vm-hotadd.rules (from debian/extra/rules-ubuntu) tries to write to invalid (as of 5.4.0-1010-azure) sysfs nodes resulting in warnings such as:

Apr 29 22:36:46 focal01 systemd-udevd[266]: memory7: /usr/lib/udev/rules.d/40-vm-hotadd.rules:9 Failed to write ATTR{/sys/devices/system/memory/memory7/state}, ignoring: Invalid argument

Perhaps 40-vm-hotadd.rules needs to be updated for 5.4 semantics, removed, or something else. This behavior is present on systems upgraded from 18.04 (via d-r-u) as well as new focal systems, upon first reboot of the VM.

udev: 245.4-4ubuntu3

Christian Reis (kiko) wrote :

Note that same udevd rule exists as far back as 16.04 LTS:

ubuntu@xenial-test:~$ md5sum /lib/udev/rules.d/40-vm-hotadd.rules
d8569b53b5bd1874a86b743b82a2a5b4 /lib/udev/rules.d/40-vm-hotadd.rules

ubuntu@focal-test:~$ md5sum /usr/lib/udev/rules.d/40-vm-hotadd.rules
d8569b53b5bd1874a86b743b82a2a5b4 /usr/lib/udev/rules.d/40-vm-hotadd.rules

There is a previous bug, https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1518457 which addressed a different symptom but may be related.

Dan Streetman (ddstreet) wrote :

> Apr 29 22:36:46 focal01 systemd-udevd[266]: memory7: /usr/lib/udev/rules.d/40-vm-hotadd.rules:9 Failed to write ATTR{/sys/devices/system/memory/memory7/state}, ignoring: Invalid argument

This isn't because the file doesn't exist, because the udev rule checks for its existence before trying to write to it. The error is because the kernel is rejecting the write to its state file for some reason. In a really quick test, it seems like this is because it's already online; the kernel default has been CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y since at least 4.15.

I think the error message is harmless, since the memory is already online, but the udev rule should be fixed to check if it's offline before actually trying to online it.

Can you test with the systemd from this ppa to see if the error message is gone?
https://launchpad.net/~ddstreet/+archive/ubuntu/lp1876018

Also please make sure all your memory and cpus are online after booting/hotadding.

Note that I might just remove this rule entirely, since it really shouldn't be needed anymore, now that the kernel default is to online hotplugged cpus/memory, but I want to verify that first.

Balint Reczey (rbalint) wrote :

@ddstreet, thanks, the fix works and I'll include it in the next upload to Groovy, in the next days.

Changed in systemd (Ubuntu):
status: New → In Progress
tags: added: id-5eac701c52dbda63e8c9b00c
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 245.5-3ubuntu1

---------------
systemd (245.5-3ubuntu1) groovy; urgency=medium

  [ Dan Streetman ]
  * Hotadd only offline memory and CPUs (LP: #1876018)
    File: debian/extra/rules-ubuntu/40-vm-hotadd.rules
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=dc68569fb46b04791202a40326003799becae859

  [ Balint Reczey ]
  * Merge to Ubuntu from Debian unstable
  - Dropped changes:
    * Cherry-pick fix from upstream master to adjust UseGateway= default
  * journald: stream pid change newline fix (LP: #1875708)
    Files:
    - debian/patches/debian/fsckd-daemon-for-inter-fsckd-communication.patch
    - debian/patches/lp1875708/journald-Increase-stdout-buffer-size-sooner-when-almost-f.patch
    - debian/patches/lp1875708/journald-rework-end-of-line-marker-handling-to-use-a-fiel.patch
    - debian/patches/lp1875708/journald-rework-pid-change-handling.patch
    - debian/patches/lp1875708/journald-use-log_warning_errno-where-appropriate.patch
    - debian/patches/lp1875708/journald-use-the-fact-that-client_context_release-returns.patch
    - debian/patches/lp1875708/man-document-the-new-_LINE_BREAK-type.patch
    - debian/patches/lp1875708/socket-util-introduce-type-safe-dereferencing-wrapper-CMS.patch
    - debian/patches/lp1875708/test-Add-a-test-case-for-15654.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=95302fbe53202bdac33f86031aa56026444a92d3

  [ Dimitri John Ledkov ]
  * meson: initialize time-epoch to reproducible builds compatible value
    (LP: #1878969)
    File: debian/patches/meson-initialize-time-epoch-to-reproducible-builds-compat.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6785dfd98f81dfe3e3d5273511ec33b196ddb0c3

systemd (245.5-3) unstable; urgency=medium

  [ Michael Biebl ]
  * Bump priority of libnss-systemd to standard
  * logind: avoid shadow lookups when doing userdb client side
  * Disable DNSSEC support by default in resolved.
    The upstream default, DNSSEC=allow-downgrade can lead to compatibility
    issues with certain network access points. Previously, DNSSEC support
    was only turned off when built for a stable Debian release, but it is
    safer and more consistent to just generally change the default to
    DNSSEC=no. (Closes: #959996)
  * Bump debhelper compatibility level to 13.
    Use the debhelper-compat package to declare the compatibility level and
    drop debian/compat.
  * Convert to dh_installsystemd and disable dh_installsystemduser
  * Drop custom initramfs update code.
    Now handled by dh_installinitramfs which is enabled by default in compat
    level 12 and above.

  [ Dan Streetman ]
  * Cherry-pick fix from upstream master to adjust UseGateway= default
    - network: change UseGateway= default to UseRoutes= setting
    - network: honor SetDNSRoutes= even if UseGateway=False
    (LP: #1867375)

  [ Topi Miettinen ]
  * Delete empty lines at end of file.
    Upstream commit hooks don't allow empty lines and of course they serve no
    purpose.

 -- Balint Reczey <email address hidden> Tue, 19 May 2020 19:43:50 +0200

Changed in systemd (Ubuntu):
status: In Progress → Fix Released
Balint Reczey (rbalint) wrote :

@ddstreet Could you please include this fix in the next upload to Focal?

Dan Streetman (ddstreet) wrote :

will do, thnx

Dan Streetman (ddstreet) on 2020-06-30
description: updated
Changed in systemd (Ubuntu Xenial):
status: New → In Progress
Changed in systemd (Ubuntu Bionic):
status: New → In Progress
Changed in systemd (Ubuntu Focal):
importance: Undecided → Medium
Changed in systemd (Ubuntu Bionic):
importance: Undecided → Medium
Changed in systemd (Ubuntu Xenial):
importance: Undecided → Medium
Changed in systemd (Ubuntu Focal):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Xenial):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Focal):
status: New → In Progress
Changed in systemd (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)

Hello Jose, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.2) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

apt/unknown (armhf)
indicator-session/17.3.20+19.10.20190921-0ubuntu1 (arm64)
dovecot/1:2.3.7.2-1ubuntu3.1 (armhf)
postgresql-12/unknown (armhf)
mir/unknown (armhf)
systemd/245.4-4ubuntu3.2 (amd64)
umockdev/unknown (armhf)
policykit-1/unknown (armhf)
asterisk/unknown (armhf)
anbox/unknown (armhf)
php7.4/unknown (armhf)
ksystemlog/unknown (armhf)
polkit-qt-1/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Dan Streetman (ddstreet) wrote :

focal:

with the reproducer hack from the description (comment out the 'goto' line in the udev rule), and the system rebooted:

ubuntu@lp1876018-f:~$ dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.1 amd64 system and service manager
ubuntu@lp1876018-f:~$ journalctl -g vm-hotadd
-- Logs begin at Wed 2020-07-22 18:11:53 UTC, end at Wed 2020-07-22 18:46:23 UTC. --
Jul 22 18:45:15 lp1876018-f sudo[12380]: ubuntu : TTY=ttyS0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/vi /lib/udev/rules.d/40-vm-hotadd.rules
-- Reboot --
Jul 22 18:45:48 lp1876018-f systemd-udevd[457]: memory0: /usr/lib/udev/rules.d/40-vm-hotadd.rules:9 Failed to write ATTR{/sys/devices/system/memory/memory0/state}, ignoring: Invalid argument
Jul 22 18:45:48 lp1876018-f systemd-udevd[457]: memory1: /usr/lib/udev/rules.d/40-vm-hotadd.rules:9 Failed to write ATTR{/sys/devices/system/memory/memory1/state}, ignoring: Invalid argument
Jul 22 18:45:48 lp1876018-f systemd-udevd[457]: memory10: /usr/lib/udev/rules.d/40-vm-hotadd.rules:9 Failed to write ATTR{/sys/devices/system/memory/memory10/state}, ignoring: Invalid argument
...(many more)...

and with the udev rule again edited after the systemd upgrade (since it replaces the edited file), and the system rebooted:

ubuntu@lp1876018-f:~$ dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.2 amd64 system and service manager
ubuntu@lp1876018-f:~$ journalctl -b -g vm-hotadd
-- Logs begin at Wed 2020-07-22 18:11:53 UTC, end at Wed 2020-07-22 18:50:00 UT>
-- No entries --

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 245.4-4ubuntu3.2

---------------
systemd (245.4-4ubuntu3.2) focal; urgency=medium

   [ Dan Streetman ]
   * Hotadd only offline memory and CPUs (LP: #1876018)
     File: debian/extra/rules-ubuntu/40-vm-hotadd.rules
     https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=72d815471596056b7727be5b10f87513ff1d5757
   * Lock swap blockdevice while calling mkswap (LP: #1838329)
     Files:
     - d/p/lp1838329/0001-blockdev-propagate-one-more-unexpected-error.patch
     - d/p/lp1838329/0002-makefs-log-about-OOM-condition.patch
     - d/p/lp1838329/0003-dissect-use-log_debug_errno-where-appropriate.patch
     - d/p/lp1838329/0004-blockdev-add-helper-for-locking-whole-block-device.patch
     - d/p/lp1838329/0005-makefs-lock-device-while-we-operate.patch
     - d/p/lp1838329/0006-makefs-normalize-logging-a-bit.patch
     - d/p/lp1838329/0007-cryptsetup-generator-use-systemd-makefs-for-implemen.patch
     https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=c81b75c4297cbb04554488b070b6f79996b8cceb

   [ Balint Reczey ]
   * debian/udev.postinst: Allow kvm to be an already present non-system group
     (LP: #1880541)
     File: debian/udev.postinst
     https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8b5c31828d4323ddb719326b1316c179b7cdbdef
   * d/p/hwdb-Mask-rfkill-event-from-intel-hid-on-HP-platforms.patch:
     hwdb: Mask rfkill event from intel-hid on HP platforms
     (LP: #1883846)
     https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=164c016b466210c7d6d05963fd753eccf4679844
   * journald: stream pid change newline fix (LP: #1875708)
     Files:
     - debian/patches/lp1875708/journald-Increase-stdout-buffer-size-sooner-when-almost-f.patch
     - debian/patches/lp1875708/journald-rework-end-of-line-marker-handling-to-use-a-fiel.patch
     - debian/patches/lp1875708/journald-rework-pid-change-handling.patch
     - debian/patches/lp1875708/journald-use-log_warning_errno-where-appropriate.patch
     - debian/patches/lp1875708/journald-use-the-fact-that-client_context_release-returns.patch
     - debian/patches/lp1875708/man-document-the-new-_LINE_BREAK-type.patch
     - debian/patches/lp1875708/socket-util-introduce-type-safe-dereferencing-wrapper-CMS.patch
     - debian/patches/lp1875708/test-Add-a-test-case-for-15654.patch
     https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=2dc19f7ae4aad7277e9d89849182453ff1d046dc

 -- Dan Streetman <email address hidden> Mon, 06 Jul 2020 17:38:31 -0400

Changed in systemd (Ubuntu Focal):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Hello Jose, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.42 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
removed: verification-done
Dan Streetman (ddstreet) wrote :

bionic verified, with udev rule changed per description testcase:

ii systemd 237-3ubuntu10.42 amd64 system and service manager
ubuntu@lp1876018-b:~$ journalctl -b |grep vm-hot
Jul 30 16:25:23 lp1876018-b systemd-udevd[509]: Reading rules file: /lib/udev/rules.d/40-vm-hotadd.rules

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic

All autopkgtests for the newly accepted systemd (237-3ubuntu10.42) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

linux-hwe-5.0/5.0.0-58.62~18.04.1 (armhf)
linux-aws-edge/5.0.0-1019.21~18.04.1 (amd64)
linux-hwe-5.4/5.4.0-42.46~18.04.1 (arm64, armhf)
lxc/3.0.3-0ubuntu1~18.04.1 (arm64)
dovecot/1:2.2.33.2-1ubuntu4.5 (armhf)
casync/2+61.20180112-1 (amd64)
linux-raspi2-5.3/5.3.0-1030.32~18.04.2 (armhf)
umockdev/0.11.1-1 (armhf)
netplan.io/0.99-0ubuntu3~18.04.3 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers