| 2019-06-21 07:14:33 |
kkm |
bug |
|
|
added bug |
| 2019-06-21 09:07:59 |
kkm |
description |
Xfer: https://github.com/GoogleCloudPlatform/compute-image-packages/issues/783
I initially reported the bug there, but it appears the file is owned by you guys?
I see this bug in Google Cloud images of 18.04 in --image=ubuntu-1804-bionic-v20190514 --image-project=gce-uefi-images.
What happens is, the image contains the file /lib/udev/rules.d/99-gce.rules with the following rule:
# Switch to using NOOP as the default scheduler per GCE request
SUBSYSTEM=="block", ACTION=="add|change", ENV{ID_VENDOR}=="*Google*", ATTR{queue/scheduler}="noop"
The rule matches both devices (/sda) and partitions (/sda1), but the scheduler is a device property and does not apply to partition. These lines are logged multiple times during the first boot of the image, when the partition and the filesystem is grown, and once on every subsequent boot, once per every partition:
Jun 3 04:46:49 toy-sec-1 systemd-udevd[1442]: error opening ATTR{/sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/sda1/queue/scheduler} for writing: No such file or directory
Jun 3 04:46:49 toy-sec-1 systemd-udevd[1438]: error opening ATTR{/sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/sda15/queue/scheduler} for writing: No such file or directory
Jun 3 04:46:49 toy-sec-1 systemd-udevd[1437]: error opening ATTR{/sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/sda14/queue/scheduler} for writing: No such file or directory
To repro, no GCE necessary; you can boot any VM with the guest using the virtio driver, drop in this file, and run e. g. parted, or any program opening the raw device, as it triggers kernel uevents. Start parted, and the messages are logged. Quit parted, and they are logged again.
This issue is harmless, but when you ingest logs, you'd rather have them as error-level message free as possible.
I can think of 3 ways to solve this issue:
1. Make the rule not match partitions. I drop-replace this file in all my images with the following:
SUBSYSTEM=="block", ENV{DEVTYPE}!="partition", ACTION=="add|change", ENV{ID_VENDOR}=="*Google*", ATTR{queue/scheduler}="noop"
2. Since Ubuntu is providing GCE images, kernel command line option 'elevator=none' sets the I/O scheduler to all applicable devices by default; no udev integration necessary. The default is not locked, so if anyone needs to change it (e. g. for a physical disk directly attached to a VM, not a GCE setup but in a local VM it's possible), they can select a different elevator strategy with udev rules. This is the setting widely recommended by other Linux-based system, e. g. there is a RHEL support page recommending that. It obviously a better choice shift the I/O elevation job to the host, as it handles requests from all guests, and can prioritize I/O much better, as it has all consolidated information available at any moment for the physical device actually doing the block I/O.
3. Since these GCE images come with a special kernel build (it has a '-gcp' version suffix), the default of none can be simply selected at compile time. It also make sense to compile in virtio into the kernel; as it is, the device is probed from initramfs. Since all VM boot drives are virtio, it is probably a sensible choice to have it compiled-in; definitely so for the GCP-specific kernel build.
Thanks, you'll probably know better than me which of these (or maybe other options I could not think of right now), as you probably understand all the implications I'm likely unaware of, so I'm just sharing my thoughts on this issue, not preferring any of these.
I did not check other images available from the same GCE project, but I'm sure if the rule is there, the result will be identical--it's a kernel thing, and partitions do not have I/O schedulers by design. |
Xfer: https://github.com/GoogleCloudPlatform/compute-image-packages/issues/783
I initially reported the bug there, but it appears the file is owned by you guys?
I see this bug in Google Cloud images of 18.04 in --image=ubuntu-1804-bionic-v20190514 --image-project=gce-uefi-images.
What happens is, the image contains the file /lib/udev/rules.d/99-gce.rules with the following rule:
# Switch to using NOOP as the default scheduler per GCE request
SUBSYSTEM=="block", ACTION=="add|change", ENV{ID_VENDOR}=="*Google*", ATTR{queue/scheduler}="noop"
The rule matches both devices (/sda) and partitions (/sda1), but the scheduler is a device property and does not apply to partition. These lines are logged multiple times during the first boot of the image, when the partition and the filesystem is grown, and once on every subsequent boot, once per every partition:
Jun 3 04:46:49 toy-sec-1 systemd-udevd[1442]: error opening ATTR{/sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/sda1/queue/scheduler} for writing: No such file or directory
Jun 3 04:46:49 toy-sec-1 systemd-udevd[1438]: error opening ATTR{/sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/sda15/queue/scheduler} for writing: No such file or directory
Jun 3 04:46:49 toy-sec-1 systemd-udevd[1437]: error opening ATTR{/sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/sda14/queue/scheduler} for writing: No such file or directory
To repro, no GCE necessary; you can boot any VM with the guest using the virtio driver, drop in this file, and run e. g. parted, or any program opening the raw device, as it triggers kernel uevents. Start parted, and the messages are logged. Quit parted, and they are logged again.
This issue is harmless, but when you ingest logs, you'd rather have them as error-level message free as possible.
I can think of 3 ways to solve this issue:
1. Make the rule not match partitions. I drop-replace this file in all my images with the following:
SUBSYSTEM=="block", ENV{DEVTYPE}!="partition", ACTION=="add|change", ENV{ID_VENDOR}=="*Google*", ATTR{queue/scheduler}="noop"
2. Since Ubuntu is providing GCE images, kernel command line option 'elevator=none' sets the I/O scheduler to all applicable devices by default; no udev integration necessary. The default is not locked, so if anyone needs to change it (e. g. for a physical disk directly attached to a VM, not a GCE setup but in a local VM it's possible), they can select a different elevator strategy with udev rules. This is the setting widely recommended by other Linux-based system, e. g. there is a RHEL support page recommending that. It obviously a better choice shift the I/O elevation job to the host, as it handles requests from all guests, and can prioritize I/O much better, as it has all consolidated information available at any moment for the physical device actually doing the block I/O.
3. Since these GCE images come with a special kernel build (it has a '-gcp' version suffix), the default of none can be simply selected at compile time. [STRIKEOUT]It also make sense to compile in virtio into the kernel; as it is, the device is probed from initramfs. Since all VM boot drives are virtio, it is probably a sensible choice to have it compiled-in; definitely so for the GCP-specific kernel build.[/STRIKEOUT] EDIT: Sorry, just noticed virtio IS compiled
into this kernel. But that was a side note besides the main point.
Thanks, you'll probably know better than me which of these (or maybe other options I could not think of right now), as you probably understand all the implications I'm likely unaware of, so I'm just sharing my thoughts on this issue, not preferring any of these.
I did not check other images available from the same GCE project, but I'm sure if the rule is there, the result will be identical--it's a kernel thing, and partitions do not have I/O schedulers by design. |
|
| 2019-08-15 20:35:33 |
Balint Reczey |
description |
Xfer: https://github.com/GoogleCloudPlatform/compute-image-packages/issues/783
I initially reported the bug there, but it appears the file is owned by you guys?
I see this bug in Google Cloud images of 18.04 in --image=ubuntu-1804-bionic-v20190514 --image-project=gce-uefi-images.
What happens is, the image contains the file /lib/udev/rules.d/99-gce.rules with the following rule:
# Switch to using NOOP as the default scheduler per GCE request
SUBSYSTEM=="block", ACTION=="add|change", ENV{ID_VENDOR}=="*Google*", ATTR{queue/scheduler}="noop"
The rule matches both devices (/sda) and partitions (/sda1), but the scheduler is a device property and does not apply to partition. These lines are logged multiple times during the first boot of the image, when the partition and the filesystem is grown, and once on every subsequent boot, once per every partition:
Jun 3 04:46:49 toy-sec-1 systemd-udevd[1442]: error opening ATTR{/sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/sda1/queue/scheduler} for writing: No such file or directory
Jun 3 04:46:49 toy-sec-1 systemd-udevd[1438]: error opening ATTR{/sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/sda15/queue/scheduler} for writing: No such file or directory
Jun 3 04:46:49 toy-sec-1 systemd-udevd[1437]: error opening ATTR{/sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/sda14/queue/scheduler} for writing: No such file or directory
To repro, no GCE necessary; you can boot any VM with the guest using the virtio driver, drop in this file, and run e. g. parted, or any program opening the raw device, as it triggers kernel uevents. Start parted, and the messages are logged. Quit parted, and they are logged again.
This issue is harmless, but when you ingest logs, you'd rather have them as error-level message free as possible.
I can think of 3 ways to solve this issue:
1. Make the rule not match partitions. I drop-replace this file in all my images with the following:
SUBSYSTEM=="block", ENV{DEVTYPE}!="partition", ACTION=="add|change", ENV{ID_VENDOR}=="*Google*", ATTR{queue/scheduler}="noop"
2. Since Ubuntu is providing GCE images, kernel command line option 'elevator=none' sets the I/O scheduler to all applicable devices by default; no udev integration necessary. The default is not locked, so if anyone needs to change it (e. g. for a physical disk directly attached to a VM, not a GCE setup but in a local VM it's possible), they can select a different elevator strategy with udev rules. This is the setting widely recommended by other Linux-based system, e. g. there is a RHEL support page recommending that. It obviously a better choice shift the I/O elevation job to the host, as it handles requests from all guests, and can prioritize I/O much better, as it has all consolidated information available at any moment for the physical device actually doing the block I/O.
3. Since these GCE images come with a special kernel build (it has a '-gcp' version suffix), the default of none can be simply selected at compile time. [STRIKEOUT]It also make sense to compile in virtio into the kernel; as it is, the device is probed from initramfs. Since all VM boot drives are virtio, it is probably a sensible choice to have it compiled-in; definitely so for the GCP-specific kernel build.[/STRIKEOUT] EDIT: Sorry, just noticed virtio IS compiled
into this kernel. But that was a side note besides the main point.
Thanks, you'll probably know better than me which of these (or maybe other options I could not think of right now), as you probably understand all the implications I'm likely unaware of, so I'm just sharing my thoughts on this issue, not preferring any of these.
I did not check other images available from the same GCE project, but I'm sure if the rule is there, the result will be identical--it's a kernel thing, and partitions do not have I/O schedulers by design. |
[Impact]
* The following messages can be observed in the journal:
systemd-udevd[1876]: sda14: Failed to open ATTR{/sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/sda14/queue/scheduler} for writing: No such file or directory
* This is not harmful, but annoying and does not show how we thrive to provide amazing experience with Ubuntu.
[Test Case]
* Run parted then exit:
$ echo p | sudo parted
* Observe the "Failed to open ATTR" messages not appearing in the journal in fixed versions.
$ journalctl -a
...
[Regression Potential]
* Absolutely minimal, the fix just adds a filter to not change scheduler on partitions.
[Original Bug Text]
Xfer: https://github.com/GoogleCloudPlatform/compute-image-packages/issues/783
I initially reported the bug there, but it appears the file is owned by you guys?
I see this bug in Google Cloud images of 18.04 in --image=ubuntu-1804-bionic-v20190514 --image-project=gce-uefi-images.
What happens is, the image contains the file /lib/udev/rules.d/99-gce.rules with the following rule:
# Switch to using NOOP as the default scheduler per GCE request
SUBSYSTEM=="block", ACTION=="add|change", ENV{ID_VENDOR}=="*Google*", ATTR{queue/scheduler}="noop"
The rule matches both devices (/sda) and partitions (/sda1), but the scheduler is a device property and does not apply to partition. These lines are logged multiple times during the first boot of the image, when the partition and the filesystem is grown, and once on every subsequent boot, once per every partition:
Jun 3 04:46:49 toy-sec-1 systemd-udevd[1442]: error opening ATTR{/sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/sda1/queue/scheduler} for writing: No such file or directory
Jun 3 04:46:49 toy-sec-1 systemd-udevd[1438]: error opening ATTR{/sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/sda15/queue/scheduler} for writing: No such file or directory
Jun 3 04:46:49 toy-sec-1 systemd-udevd[1437]: error opening ATTR{/sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/sda14/queue/scheduler} for writing: No such file or directory
To repro, no GCE necessary; you can boot any VM with the guest using the virtio driver, drop in this file, and run e. g. parted, or any program opening the raw device, as it triggers kernel uevents. Start parted, and the messages are logged. Quit parted, and they are logged again.
This issue is harmless, but when you ingest logs, you'd rather have them as error-level message free as possible.
I can think of 3 ways to solve this issue:
1. Make the rule not match partitions. I drop-replace this file in all my images with the following:
SUBSYSTEM=="block", ENV{DEVTYPE}!="partition", ACTION=="add|change", ENV{ID_VENDOR}=="*Google*", ATTR{queue/scheduler}="noop"
2. Since Ubuntu is providing GCE images, kernel command line option 'elevator=none' sets the I/O scheduler to all applicable devices by default; no udev integration necessary. The default is not locked, so if anyone needs to change it (e. g. for a physical disk directly attached to a VM, not a GCE setup but in a local VM it's possible), they can select a different elevator strategy with udev rules. This is the setting widely recommended by other Linux-based system, e. g. there is a RHEL support page recommending that. It obviously a better choice shift the I/O elevation job to the host, as it handles requests from all guests, and can prioritize I/O much better, as it has all consolidated information available at any moment for the physical device actually doing the block I/O.
3. Since these GCE images come with a special kernel build (it has a '-gcp' version suffix), the default of none can be simply selected at compile time. [STRIKEOUT]It also make sense to compile in virtio into the kernel; as it is, the device is probed from initramfs. Since all VM boot drives are virtio, it is probably a sensible choice to have it compiled-in; definitely so for the GCP-specific kernel build.[/STRIKEOUT] EDIT: Sorry, just noticed virtio IS compiled
into this kernel. But that was a side note besides the main point.
Thanks, you'll probably know better than me which of these (or maybe other options I could not think of right now), as you probably understand all the implications I'm likely unaware of, so I'm just sharing my thoughts on this issue, not preferring any of these.
I did not check other images available from the same GCE project, but I'm sure if the rule is there, the result will be identical--it's a kernel thing, and partitions do not have I/O schedulers by design. |
|
| 2019-08-15 20:55:41 |
Launchpad Janitor |
gce-compute-image-packages (Ubuntu): status |
New |
Fix Released |
|
| 2019-08-20 16:34:28 |
Brian Murray |
gce-compute-image-packages (Ubuntu Disco): status |
New |
Fix Committed |
|
| 2019-08-20 16:34:30 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-08-20 16:34:33 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2019-08-20 16:34:39 |
Brian Murray |
tags |
|
verification-needed verification-needed-disco |
|
| 2019-08-20 16:45:39 |
Brian Murray |
gce-compute-image-packages (Ubuntu Bionic): status |
New |
Fix Committed |
|
| 2019-08-20 16:45:46 |
Brian Murray |
tags |
verification-needed verification-needed-disco |
verification-needed verification-needed-bionic verification-needed-disco |
|
| 2019-08-20 16:48:01 |
Brian Murray |
gce-compute-image-packages (Ubuntu Xenial): status |
New |
Fix Committed |
|
| 2019-08-20 16:48:08 |
Brian Murray |
tags |
verification-needed verification-needed-bionic verification-needed-disco |
verification-needed verification-needed-bionic verification-needed-disco verification-needed-xenial |
|
| 2019-11-19 20:29:12 |
Brian Murray |
tags |
verification-needed verification-needed-bionic verification-needed-disco verification-needed-xenial |
removal-candidate verification-needed verification-needed-bionic verification-needed-disco verification-needed-xenial |
|
| 2019-11-20 02:06:00 |
kkm |
gce-compute-image-packages (Ubuntu Bionic): status |
Fix Committed |
Confirmed |
|
| 2019-11-20 02:07:06 |
kkm |
attachment added |
|
testlog-bionic.log https://bugs.launchpad.net/ubuntu/+source/gce-compute-image-packages/+bug/1833660/+attachment/5306541/+files/testlog-bionic.log |
|
| 2019-11-20 02:21:59 |
kkm |
gce-compute-image-packages (Ubuntu Xenial): status |
Fix Committed |
Confirmed |
|
| 2019-11-20 02:24:42 |
kkm |
gce-compute-image-packages (Ubuntu Xenial): status |
Confirmed |
Fix Committed |
|
| 2019-11-20 02:24:46 |
kkm |
gce-compute-image-packages (Ubuntu Bionic): status |
Confirmed |
Fix Committed |
|
| 2019-11-20 02:25:34 |
kkm |
tags |
removal-candidate verification-needed verification-needed-bionic verification-needed-disco verification-needed-xenial |
removal-candidate verification-done-bionic verification-done-xenial verification-needed verification-needed-disco |
|
| 2019-11-20 02:26:04 |
kkm |
attachment added |
|
testlog-xenial.log https://bugs.launchpad.net/ubuntu/+source/gce-compute-image-packages/+bug/1833660/+attachment/5306545/+files/testlog-xenial.log |
|
| 2019-11-20 02:41:54 |
kkm |
attachment added |
|
testlog-disco.log https://bugs.launchpad.net/ubuntu/+source/gce-compute-image-packages/+bug/1833660/+attachment/5306548/+files/testlog-disco.log |
|
| 2019-11-20 02:42:53 |
kkm |
tags |
removal-candidate verification-done-bionic verification-done-xenial verification-needed verification-needed-disco |
verification-done-bionic verification-done-disco verification-done-xenial |
|
| 2019-12-17 20:01:06 |
Launchpad Janitor |
gce-compute-image-packages (Ubuntu Disco): status |
Fix Committed |
Fix Released |
|
| 2019-12-17 20:01:14 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2019-12-17 20:01:39 |
Launchpad Janitor |
gce-compute-image-packages (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2019-12-17 20:01:57 |
Launchpad Janitor |
gce-compute-image-packages (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
