| 2023-10-16 23:42:10 |
Brett Holman |
bug |
|
|
added bug |
| 2023-10-17 01:06:38 |
Robie Basak |
bug |
|
|
added subscriber Robie Basak |
| 2023-10-17 09:19:57 |
Alberto Contreras |
bug |
|
|
added subscriber Alberto Contreras |
| 2023-10-17 09:54:53 |
Andrew Cloke |
bug |
|
|
added subscriber Andrew Cloke |
| 2023-10-17 22:02:44 |
Chad Smith |
cloud-init (Ubuntu): status |
New |
Fix Committed |
|
| 2023-10-17 23:28:02 |
Chad Smith |
cloud-init (Ubuntu): importance |
Undecided |
High |
|
| 2023-10-17 23:28:05 |
Chad Smith |
cloud-init (Ubuntu): importance |
High |
Critical |
|
| 2023-10-17 23:28:21 |
Chad Smith |
cloud-init (Ubuntu): assignee |
|
Brett Holman (holmanb) |
|
| 2023-10-19 18:32:31 |
Chad Smith |
description |
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
The symptom we can see on long-delayed systemd-users-sessions.service is that ssh and a non-root user will get this console error message until cloud-config.service has completed in early boot:
`System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8).`
We really want to reduce that time to ssh for non-root users, so expected short-term fix in downstream Ubuntu is to revert the commit b3c9b6a7.
Long-term will be a resolution in cloud-init upstream to avoid strict systemd unit ordering After=snapd.seeded.service for cloud-config.service by only blocking on snap seeding if the user-data/vendor-data provided to the instance depends on snaps being present.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
|
| 2023-10-19 18:53:28 |
Chad Smith |
description |
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
The symptom we can see on long-delayed systemd-users-sessions.service is that ssh and a non-root user will get this console error message until cloud-config.service has completed in early boot:
`System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8).`
We really want to reduce that time to ssh for non-root users, so expected short-term fix in downstream Ubuntu is to revert the commit b3c9b6a7.
Long-term will be a resolution in cloud-init upstream to avoid strict systemd unit ordering After=snapd.seeded.service for cloud-config.service by only blocking on snap seeding if the user-data/vendor-data provided to the instance depends on snaps being present.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
=== Begin SRU Template ===
[Impact]
In 23.3.1, system unit ordering changes moved the configuration Before=systemd-user-sessions.service out of the earlier cloud-init.service boot stage and into the subsequent cloud-config.service boot stage to ensure all users have been configured before providing a login prompt on the console. This was originally intended to fix LP: #2013403 for emulated riscv environments, but the general time to SSH/login cost is too great at this stage to leave the impact active in most images where snap seeding is being perfomed on first boot.
Leaving this current extended delay to login unresolved also breaks tooling which uses `uvt-kvm wait` which is present in some continuous integration testing.
The fix is to revert https://github.com/canonical/cloud-init/commit/b3c9b6a7.
[Test Case]
Launch a daily image and a daily-image with proposed cloud-init and compare the following data points:
- time to SSH: number of retries required to successfully SSH into the VM
- validate time to ssh by sampling: systemd-analyze critical-chain systemd-user-sessions.service to see total time until login was unblocked from systemd perspective
- systemctl show -p Before,After cloud-config.service cloud-init.service --no-pager
- systemd-analyze blame: # validate systemd-user-sessions is not one of the longest blocks to boot
[Regression Potential]
This is a revert to functionality that was working for years. It will regress emulated riscV users per LP: #2013403 as they may be able to see a login prompt that will show up before cloud-config completes and could reject their correct configured password as invalid until the cloud-config.service completes setup on first boot.
[Other info]
LP: #2013403
LP: #2039441
[Original Description]
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
|
| 2023-10-19 18:53:33 |
Chad Smith |
description |
=== Begin SRU Template ===
[Impact]
In 23.3.1, system unit ordering changes moved the configuration Before=systemd-user-sessions.service out of the earlier cloud-init.service boot stage and into the subsequent cloud-config.service boot stage to ensure all users have been configured before providing a login prompt on the console. This was originally intended to fix LP: #2013403 for emulated riscv environments, but the general time to SSH/login cost is too great at this stage to leave the impact active in most images where snap seeding is being perfomed on first boot.
Leaving this current extended delay to login unresolved also breaks tooling which uses `uvt-kvm wait` which is present in some continuous integration testing.
The fix is to revert https://github.com/canonical/cloud-init/commit/b3c9b6a7.
[Test Case]
Launch a daily image and a daily-image with proposed cloud-init and compare the following data points:
- time to SSH: number of retries required to successfully SSH into the VM
- validate time to ssh by sampling: systemd-analyze critical-chain systemd-user-sessions.service to see total time until login was unblocked from systemd perspective
- systemctl show -p Before,After cloud-config.service cloud-init.service --no-pager
- systemd-analyze blame: # validate systemd-user-sessions is not one of the longest blocks to boot
[Regression Potential]
This is a revert to functionality that was working for years. It will regress emulated riscV users per LP: #2013403 as they may be able to see a login prompt that will show up before cloud-config completes and could reject their correct configured password as invalid until the cloud-config.service completes setup on first boot.
[Other info]
LP: #2013403
LP: #2039441
[Original Description]
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
=== Begin SRU Template ===
[Impact]
In 23.3.1, system unit ordering changes moved the configuration Before=systemd-user-sessions.service out of the earlier cloud-init.service boot stage and into the subsequent cloud-config.service boot stage to ensure all users have been configured before providing a login prompt on the console. This was originally intended to fix LP: #2013403 for emulated riscv environments, but the general time to SSH/login cost is too great at this stage to leave the impact active in most images where snap seeding is being perfomed on first boot.
Leaving this current extended delay to login unresolved also breaks tooling which uses `uvt-kvm wait` which is present in some continuous integration testing.
The fix is to revert https://github.com/canonical/cloud-init/commit/b3c9b6a7.
[Test Case]
Launch a daily image and a daily-image with proposed cloud-init and compare the following data points:
- time to SSH: number of retries required to successfully SSH into the VM
- validate time to ssh by sampling: systemd-analyze critical-chain systemd-user-sessions.service to see total time until login was unblocked from systemd perspective
- systemctl show -p Before,After cloud-config.service cloud-init.service --no-pager
- systemd-analyze blame: # validate systemd-user-sessions is not one of the longest blocks to boot
[Regression Potential]
This is a revert to functionality that was working for years. It will regress emulated riscV users per LP: #2013403 as they may be able to see a login prompt that will show up before cloud-config completes and could reject their correct configured password as invalid until the cloud-config.service completes setup on first boot.
[Other info]
LP: #2013403
LP: #2039441
[Original Description]
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
|
| 2023-10-24 14:50:51 |
Chad Smith |
nominated for series |
|
Ubuntu Lunar |
|
| 2023-10-24 14:50:51 |
Chad Smith |
bug task added |
|
cloud-init (Ubuntu Lunar) |
|
| 2023-10-24 14:50:51 |
Chad Smith |
nominated for series |
|
Ubuntu Focal |
|
| 2023-10-24 14:50:51 |
Chad Smith |
bug task added |
|
cloud-init (Ubuntu Focal) |
|
| 2023-10-24 14:50:51 |
Chad Smith |
nominated for series |
|
Ubuntu Jammy |
|
| 2023-10-24 14:50:51 |
Chad Smith |
bug task added |
|
cloud-init (Ubuntu Jammy) |
|
| 2023-10-24 14:50:51 |
Chad Smith |
nominated for series |
|
Ubuntu Mantic |
|
| 2023-10-24 14:50:51 |
Chad Smith |
bug task added |
|
cloud-init (Ubuntu Mantic) |
|
| 2023-10-24 21:44:25 |
Ubuntu Archive Robot |
bug |
|
|
added subscriber James Falcon |
| 2023-10-30 19:15:41 |
Chad Smith |
cloud-init (Ubuntu Focal): status |
New |
Fix Committed |
|
| 2023-10-30 19:15:45 |
Chad Smith |
cloud-init (Ubuntu Jammy): status |
New |
Fix Committed |
|
| 2023-10-30 19:15:47 |
Chad Smith |
cloud-init (Ubuntu Lunar): status |
New |
Fix Committed |
|
| 2023-10-30 19:15:51 |
Chad Smith |
cloud-init (Ubuntu Mantic): status |
New |
Fix Committed |
|
| 2023-11-11 11:40:30 |
Launchpad Janitor |
cloud-init (Ubuntu): status |
Fix Committed |
Fix Released |
|
| 2023-11-16 20:02:04 |
Andreas Hasenack |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2023-11-16 20:02:14 |
Andreas Hasenack |
bug |
|
|
added subscriber SRU Verification |
| 2023-11-16 20:02:19 |
Andreas Hasenack |
tags |
|
verification-needed verification-needed-mantic |
|
| 2023-11-16 20:04:30 |
Andreas Hasenack |
tags |
verification-needed verification-needed-mantic |
verification-needed verification-needed-lunar verification-needed-mantic |
|
| 2023-11-16 20:07:10 |
Andreas Hasenack |
tags |
verification-needed verification-needed-lunar verification-needed-mantic |
verification-needed verification-needed-jammy verification-needed-lunar verification-needed-mantic |
|
| 2023-11-16 20:08:01 |
Andreas Hasenack |
tags |
verification-needed verification-needed-jammy verification-needed-lunar verification-needed-mantic |
verification-needed verification-needed-focal verification-needed-jammy verification-needed-lunar verification-needed-mantic |
|
| 2023-11-17 20:21:16 |
Chad Smith |
description |
=== Begin SRU Template ===
[Impact]
In 23.3.1, system unit ordering changes moved the configuration Before=systemd-user-sessions.service out of the earlier cloud-init.service boot stage and into the subsequent cloud-config.service boot stage to ensure all users have been configured before providing a login prompt on the console. This was originally intended to fix LP: #2013403 for emulated riscv environments, but the general time to SSH/login cost is too great at this stage to leave the impact active in most images where snap seeding is being perfomed on first boot.
Leaving this current extended delay to login unresolved also breaks tooling which uses `uvt-kvm wait` which is present in some continuous integration testing.
The fix is to revert https://github.com/canonical/cloud-init/commit/b3c9b6a7.
[Test Case]
Launch a daily image and a daily-image with proposed cloud-init and compare the following data points:
- time to SSH: number of retries required to successfully SSH into the VM
- validate time to ssh by sampling: systemd-analyze critical-chain systemd-user-sessions.service to see total time until login was unblocked from systemd perspective
- systemctl show -p Before,After cloud-config.service cloud-init.service --no-pager
- systemd-analyze blame: # validate systemd-user-sessions is not one of the longest blocks to boot
[Regression Potential]
This is a revert to functionality that was working for years. It will regress emulated riscV users per LP: #2013403 as they may be able to see a login prompt that will show up before cloud-config completes and could reject their correct configured password as invalid until the cloud-config.service completes setup on first boot.
[Other info]
LP: #2013403
LP: #2039441
[Original Description]
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
=== Begin SRU Template ===
[Impact]
In 23.3.1, system unit ordering changes moved the configuration Before=systemd-user-sessions.service out of the earlier cloud-init.service boot stage and into the subsequent cloud-config.service boot stage to ensure all users have been configured before providing a login prompt on the console. This was originally intended to fix LP: #2013403 for emulated riscv environments, but the general time to SSH/login cost is too great at this stage to leave the impact active in most images where snap seeding is being perfomed on first boot.
Leaving this current extended delay to login unresolved also breaks tooling which uses `uvt-kvm wait` which is present in some continuous integration testing.
The fix is to revert https://github.com/canonical/cloud-init/commit/b3c9b6a7.
[Test Case]
Launch a daily image and a daily-image with proposed cloud-init and compare the following data points:
- time to SSH: number of retries required to successfully SSH into the VM
- assert presence of console messages about Unprivileged users inability to login on current cloud-init
- assert absence of console messages about Unprivileged users inability to login on -proposed cloud-init 23.3.3
- validate time to ssh by sampling: systemd-analyze critical-chain systemd-user-sessions.service to see total time until login was unblocked from systemd perspective
- systemctl show -p Before,After cloud-config.service cloud-init.service --no-pager
- systemd-analyze blame: # validate systemd-user-sessions is not one of the longest blocks to boot
test procedure:
RANDOMSEED=$(petname)
cat <<EOF > meta-data-${RANDOMSEED}
instance-id: iid-local01
local-hostname: cloudimg
EOF
PLAIN_TEXT_PASSWORD=passw0rd
ENCRYPTED_PASSWORD=$(mkpasswd --method=SHA-512 --rounds=4096 "${PLAIN_TEXT_PASSWORD}")
cat <<EOF > user-data-${RANDOMSEED}
#cloud-config
ssh_import_id: [chad.smith]
system_info:
default_user:
lock_passwd: false
passwd: ${ENCRYPTED_PASSWORD}
ssh_pwauth: true
EOF
cat <<EOF > setup_proposed.sh
#!/bin/bash
mirror=http://archive.ubuntu.com/ubuntu
echo deb \$mirror \$(lsb_release -sc)-proposed main | tee /etc/apt/sources.list.d/proposed.list
apt-get update -q
apt-get install -t \$(lsb_release -sc)-proposed -qy cloud-init
EOF
cloud-localds seed-${RANDOMSEED}.img user-data-${RANDOMSEED} meta-data-${RANDOMSEED}
wait_for_ssh() {
port=$1
while true; do
sleep 5
if ssh -o "StrictHOstKeyChecking no" -o "UserKnownHostsFile=/dev/null" ubuntu@localhost -p $port -- cloud-init status --long --wait; then
echo "got it"
break
else
echo "Awaiting SSH access..."
fi
done
}
cmd(){
port=$1
shift
echo "---- $@ -----"
ssh -o "StrictHOstKeyChecking no" -o "UserKnownHostsFile=/dev/null" ubuntu@localhost -p $port -- $@
}
for RELEASE in mantic; do
RELEASE_URL=https://cloud-images.ubuntu.com/$RELEASE/current
RELEASE_IMG=$RELEASE-server-cloudimg-amd64.img
if [ ! -f $RELEASE_IMG ]; then
echo $RELEASE_IMG
wget $RELEASE_URL/$RELEASE_IMG
fi
echo "Creating a disk backed by the image..."
qemu-img create -f qcow2 -F qcow2 -b $RELEASE_IMG boot-disk-${RANDOMSEED}.img
echo "Launching current cloud-init to confirm expected time to SSH delay"
echo --- expect to see some messages Unprivileged users are not permitted to log in yet ---
qemu-system-x86_64 -enable-kvm -daemonize -drive file=boot-disk-${RANDOMSEED}.img,if=ide,index=0 -drive file=seed-${RANDOMSEED}.img,if=ide -net nic -net user,hostfwd=tcp::2222-:22 -s5
wait_for_ssh 2222
echo --- assert systemd-user-sessions.service is blocked by later cloud-config.service
cmd 2222 systemd-analyze critical-chain systemd-user-sessions.service | grep cloud
cmd 2222 systemctl show -p Before cloud-config.service | grep systemd-user-sessions.service || echo FAILURE: expected cloud-config.service ordered Before=systemd-user-sessions && ecs
echo "Creating a disk backed by the image with cloud-init upgraded to -proposed..."
qemu-img create -f qcow2 -F qcow2 -b $RELEASE_IMG boot-disk-proposed-${RANDOMSEED}.img
sudo mount-image-callback boot-disk-proposed-${RANDOMSEED}.img -- sh -c 'cp ./setup_proposed.sh ${MOUNTPOINT}/setup_proposed.sh'
echo "Upgrading cloud-init from $RELEASE-proposed"
sudo mount-image-callback boot-disk-proposed-${RANDOMSEED}.img --system-resolvconf -- mchroot sh -c 'bash /setup_proposed.sh'
echo --- expect to see NO messages concerning Unprivileged users are not permitted to log in yet ---
qemu-system-x86_64 -enable-kvm -daemonize -drive file=boot-disk-proposed-${RANDOMSEED}.img,if=ide,index=0 -drive file=seed-${RANDOMSEED}.img,if=ide -net nic -net user,hostfwd=tcp::225
wait_for_ssh 2223
echo --- assert cloud-init is 23.3.3 based
cmd 2223 cloud-init --version
echo --- assert systemd-user-sessions.service is blocked by earlier cloud-init.service
cmd 2223 systemd-analyze critical-chain systemd-user-sessions.service | grep cloud
echo --- assert cloud-init.service blocks systemd-user-sessions.service
cmd 2223 systemctl show -p Before cloud-init.service
cmd 2223 systemctl show -p Before cloud-init.service | grep systemd-user-sessions.service || echo FAILURE: expected cloud-init.service ordered Before=systemd-user-sessions && echo Ss
done
[Regression Potential]
This is a revert to functionality that was working for years. It will regress emulated riscV users per LP: #2013403 as they may be able to see a login prompt that will show up before cloud-config completes and could reject their correct configured password as invalid until the cloud-config.service completes setup on first boot.
[Other info]
LP: #2013403
LP: #2039441
[Original Description]
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
|
| 2023-11-17 20:25:45 |
Chad Smith |
attachment added |
|
sru.sh https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2039505/+attachment/5720592/+files/sru.sh |
|
| 2023-11-17 20:27:33 |
Chad Smith |
tags |
verification-needed verification-needed-focal verification-needed-jammy verification-needed-lunar verification-needed-mantic |
verification-done-mantic verification-needed verification-needed-focal verification-needed-jammy verification-needed-lunar |
|
| 2023-11-17 20:37:29 |
Chad Smith |
tags |
verification-done-mantic verification-needed verification-needed-focal verification-needed-jammy verification-needed-lunar |
verification-done-jammy verification-done-mantic verification-needed verification-needed-focal verification-needed-lunar |
|
| 2023-11-17 20:47:58 |
Chad Smith |
tags |
verification-done-jammy verification-done-mantic verification-needed verification-needed-focal verification-needed-lunar |
verification-done-jammy verification-done-lunar verification-done-mantic verification-needed verification-needed-focal |
|
| 2023-11-17 20:53:03 |
Chad Smith |
tags |
verification-done-jammy verification-done-lunar verification-done-mantic verification-needed verification-needed-focal |
verification-done verification-done-focal verification-done-jammy verification-done-lunar verification-done-mantic |
|
| 2023-11-17 21:21:05 |
Chad Smith |
attachment removed |
sru.sh https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2039505/+attachment/5720592/+files/sru.sh |
|
|
| 2023-11-17 21:43:27 |
Chad Smith |
attachment added |
|
sru.sh https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2039505/+attachment/5720596/+files/sru.sh |
|
| 2023-11-22 19:41:16 |
Simon Déziel |
bug |
|
|
added subscriber Simon Déziel |
| 2023-11-27 18:36:42 |
Launchpad Janitor |
cloud-init (Ubuntu Mantic): status |
Fix Committed |
Fix Released |
|
| 2023-11-27 18:36:51 |
Timo Aaltonen |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2023-11-27 18:37:08 |
Launchpad Janitor |
cloud-init (Ubuntu Lunar): status |
Fix Committed |
Fix Released |
|
| 2023-11-27 18:37:21 |
Launchpad Janitor |
cloud-init (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2023-11-27 18:37:29 |
Launchpad Janitor |
cloud-init (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
