Activity log for bug #1910835

Date Who What changed Old value New value Message
2021-01-09 03:29:43 Chad Smith bug added bug
2021-01-09 06:09:25 Rakesh Ginjupalli bug added subscriber Rakesh Ginjupalli
2021-01-11 13:41:44 Richard Harding summary Auzre IMDS publicKeys contain \r\n which prevents ssh access to vms using cloud-generated ssh keys. Azure IMDS publicKeys contain \r\n which prevents ssh access to vms using cloud-generated ssh keys.
2021-01-11 22:38:38 Chad Smith bug task added cloud-init (Ubuntu)
2021-01-11 22:38:54 Chad Smith nominated for series Ubuntu Hirsute
2021-01-11 22:38:54 Chad Smith bug task added cloud-init (Ubuntu Hirsute)
2021-01-11 22:38:54 Chad Smith nominated for series Ubuntu Focal
2021-01-11 22:38:54 Chad Smith bug task added cloud-init (Ubuntu Focal)
2021-01-11 22:38:54 Chad Smith nominated for series Ubuntu Xenial
2021-01-11 22:38:54 Chad Smith bug task added cloud-init (Ubuntu Xenial)
2021-01-11 22:38:54 Chad Smith nominated for series Ubuntu Bionic
2021-01-11 22:38:54 Chad Smith bug task added cloud-init (Ubuntu Bionic)
2021-01-11 22:38:54 Chad Smith nominated for series Ubuntu Groovy
2021-01-11 22:38:54 Chad Smith bug task added cloud-init (Ubuntu Groovy)
2021-01-11 22:57:44 Chad Smith description cloud-init 20.4 or later will incorrectly add Azure publicKeys to .ssh/authorized_keys preventing ssh access for cloud-generated keys. To reproduce: launch an ubuntu VM from the portal.azure.com choosing to generate new ssh key. When the instance is launched you can see that the ssh-rsa content provided in the metadata publicKeys value contains CRLF characters (\r\n) thus splitting the content of the pubkey onto multiple lines when it is rendered into .ssh/authorized_keys. the solution is either for IMDS to stop adding the CRLF characters or cloud-init to strip them out. Here is the IMDS value provided to cloud-init cloud-init query --format '{{ds.meta_data.imds.compute.publicKeys}}' [{'keyData': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d\r\nk/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R\r\n9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW\r\nlkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq\r\n4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7\r\n6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu\r\niKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht\r\n6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9\r\nS2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure\r\n', 'path': '/home/ubuntu/.ssh/authorized_keys'}] cloud-init renders this directly to .ssh/authorized_keys without processing the string, resulting in an invalid keyline: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d k/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R^M 9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW^M lkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq^M 4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7^M 6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu^M iKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht^M 6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9^M S2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure this prevents ssh from actually reading the right key from azure: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys If we strip the CRLF (^M) characters and reparse with ssh-keygenm we see the proper key registered: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys 3072 SHA256:PQ9EKxTKONJKFC2N56UpL6+Oc/cujfA9HpsF5VW2QDI generated-by-azure (RSA) If cloud-init (or IMDS) were to strip those \r\n characters from each line ssh == Begin SRU Template == [Impact] This release is only a single functional cherry-pick which solely affects Azure platform. It is a critical bug we wish to release as soon as possible * Azure: cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys (#760) (LP: #1910835) [Test Case] The following development and SRU process was followed: https://wiki.ubuntu.com/CloudinitUpdates The cloud-init team will be in charge of attaching the artifacts and console output of the appropriate run to the bug. cloud-init team members will not mark ‘verification-done’ until this has happened. * Automated Test Results <TODO: attach automated cloud-init-proposed test artifacts from tests for each release with lxd artifacts> <TODO: attach automated cloud-init-proposed test artifacts from tests for each release with kvm artifacts> <TODO: attach automated curtin vmtest with cloud-init proposed> <TODO: attach Solutions Testing team test results for each LTS> * Manual Test Results <TODO: attach manual cloud-init-proposed test artifacts from tests for each release on ec2 datasource> <TODO: attach manual cloud-init-proposed test artifacts from tests for each release on gce datasource> <TODO: attach manual cloud-init-proposed test artifacts from tests for each release on azure datasource> [Regression Potential] In order to mitigate the regression potential, the results of the aforementioned integration tests are attached to this bug. [Discussion] This should only affect public Azure VM launched which use Azure to --generate-ssh-keys either from the dashboard or from the `az cli` Any other cloud-platform is not affected by this change. == End SRU Template == * cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys (#760) (LP: #1910835) == Original Description == cloud-init 20.4 or later will incorrectly add Azure publicKeys to .ssh/authorized_keys preventing ssh access for cloud-generated keys. To reproduce: launch an ubuntu VM from the portal.azure.com choosing to generate new ssh key. When the instance is launched you can see that the ssh-rsa content provided in the metadata publicKeys value contains CRLF characters (\r\n) thus splitting the content of the pubkey onto multiple lines when it is rendered into .ssh/authorized_keys. the solution is either for IMDS to stop adding the CRLF characters or cloud-init to strip them out. Here is the IMDS value provided to cloud-init cloud-init query --format '{{ds.meta_data.imds.compute.publicKeys}}' [{'keyData': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d\r\nk/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R\r\n9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW\r\nlkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq\r\n4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7\r\n6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu\r\niKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht\r\n6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9\r\nS2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure\r\n', 'path': '/home/ubuntu/.ssh/authorized_keys'}] cloud-init renders this directly to .ssh/authorized_keys without processing the string, resulting in an invalid keyline: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d k/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R^M 9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW^M lkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq^M 4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7^M 6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu^M iKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht^M 6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9^M S2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure this prevents ssh from actually reading the right key from azure: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys If we strip the CRLF (^M) characters and reparse with ssh-keygenm we see the proper key registered: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys 3072 SHA256:PQ9EKxTKONJKFC2N56UpL6+Oc/cujfA9HpsF5VW2QDI generated-by-azure (RSA) If cloud-init (or IMDS) were to strip those \r\n characters from each line ssh
2021-01-11 23:01:00 Chad Smith description == Begin SRU Template == [Impact] This release is only a single functional cherry-pick which solely affects Azure platform. It is a critical bug we wish to release as soon as possible * Azure: cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys (#760) (LP: #1910835) [Test Case] The following development and SRU process was followed: https://wiki.ubuntu.com/CloudinitUpdates The cloud-init team will be in charge of attaching the artifacts and console output of the appropriate run to the bug. cloud-init team members will not mark ‘verification-done’ until this has happened. * Automated Test Results <TODO: attach automated cloud-init-proposed test artifacts from tests for each release with lxd artifacts> <TODO: attach automated cloud-init-proposed test artifacts from tests for each release with kvm artifacts> <TODO: attach automated curtin vmtest with cloud-init proposed> <TODO: attach Solutions Testing team test results for each LTS> * Manual Test Results <TODO: attach manual cloud-init-proposed test artifacts from tests for each release on ec2 datasource> <TODO: attach manual cloud-init-proposed test artifacts from tests for each release on gce datasource> <TODO: attach manual cloud-init-proposed test artifacts from tests for each release on azure datasource> [Regression Potential] In order to mitigate the regression potential, the results of the aforementioned integration tests are attached to this bug. [Discussion] This should only affect public Azure VM launched which use Azure to --generate-ssh-keys either from the dashboard or from the `az cli` Any other cloud-platform is not affected by this change. == End SRU Template == * cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys (#760) (LP: #1910835) == Original Description == cloud-init 20.4 or later will incorrectly add Azure publicKeys to .ssh/authorized_keys preventing ssh access for cloud-generated keys. To reproduce: launch an ubuntu VM from the portal.azure.com choosing to generate new ssh key. When the instance is launched you can see that the ssh-rsa content provided in the metadata publicKeys value contains CRLF characters (\r\n) thus splitting the content of the pubkey onto multiple lines when it is rendered into .ssh/authorized_keys. the solution is either for IMDS to stop adding the CRLF characters or cloud-init to strip them out. Here is the IMDS value provided to cloud-init cloud-init query --format '{{ds.meta_data.imds.compute.publicKeys}}' [{'keyData': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d\r\nk/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R\r\n9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW\r\nlkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq\r\n4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7\r\n6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu\r\niKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht\r\n6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9\r\nS2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure\r\n', 'path': '/home/ubuntu/.ssh/authorized_keys'}] cloud-init renders this directly to .ssh/authorized_keys without processing the string, resulting in an invalid keyline: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d k/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R^M 9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW^M lkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq^M 4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7^M 6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu^M iKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht^M 6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9^M S2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure this prevents ssh from actually reading the right key from azure: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys If we strip the CRLF (^M) characters and reparse with ssh-keygenm we see the proper key registered: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys 3072 SHA256:PQ9EKxTKONJKFC2N56UpL6+Oc/cujfA9HpsF5VW2QDI generated-by-azure (RSA) If cloud-init (or IMDS) were to strip those \r\n characters from each line ssh == Begin SRU Template == [Impact] This release is only a single functional cherry-pick which solely affects Azure platform. It is a critical bug we wish to release as soon as possible   * Azure: cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys     (#760) (LP: #1910835) The functional changeset here introduces a raise KeyError exception which forces cloud-init to revert to previous released logic of the previous cloud-init public release 20.3. [Test Case] The following development and SRU process was followed: https://wiki.ubuntu.com/CloudinitUpdates The cloud-init team will be in charge of attaching the artifacts and console output of the appropriate run to the bug. cloud-init team members will not mark ‘verification-done’ until this has happened. * Automated Test Results <TODO: attach automated cloud-init-proposed test artifacts from tests for each release with lxd artifacts> <TODO: attach automated cloud-init-proposed test artifacts from tests for each release with kvm artifacts> <TODO: attach automated curtin vmtest with cloud-init proposed> <TODO: attach Solutions Testing team test results for each LTS> * Manual Test Results <TODO: attach manual cloud-init-proposed test artifacts from tests for each release on ec2 datasource> <TODO: attach manual cloud-init-proposed test artifacts from tests for each release on gce datasource> <TODO: attach manual cloud-init-proposed test artifacts from tests for each release on azure datasource> [Regression Potential] In order to mitigate the regression potential, the results of the aforementioned integration tests are attached to this bug. [Discussion] This should only affect public Azure VM launched which use Azure to --generate-ssh-keys either from the dashboard or from the `az cli` Any other cloud-platform is not affected by this change. == End SRU Template ==   * cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys     (#760) (LP: #1910835) == Original Description == cloud-init 20.4 or later will incorrectly add Azure publicKeys to .ssh/authorized_keys preventing ssh access for cloud-generated keys. To reproduce: launch an ubuntu VM from the portal.azure.com choosing to generate new ssh key. When the instance is launched you can see that the ssh-rsa content provided in the metadata publicKeys value contains CRLF characters (\r\n) thus splitting the content of the pubkey onto multiple lines when it is rendered into .ssh/authorized_keys. the solution is either for IMDS to stop adding the CRLF characters or cloud-init to strip them out. Here is the IMDS value provided to cloud-init cloud-init query --format '{{ds.meta_data.imds.compute.publicKeys}}' [{'keyData': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d\r\nk/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R\r\n9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW\r\nlkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq\r\n4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7\r\n6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu\r\niKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht\r\n6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9\r\nS2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure\r\n', 'path': '/home/ubuntu/.ssh/authorized_keys'}] cloud-init renders this directly to .ssh/authorized_keys without processing the string, resulting in an invalid keyline: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d k/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R^M 9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW^M lkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq^M 4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7^M 6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu^M iKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht^M 6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9^M S2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure this prevents ssh from actually reading the right key from azure: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys If we strip the CRLF (^M) characters and reparse with ssh-keygenm we see the proper key registered: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys 3072 SHA256:PQ9EKxTKONJKFC2N56UpL6+Oc/cujfA9HpsF5VW2QDI generated-by-azure (RSA) If cloud-init (or IMDS) were to strip those \r\n characters from each line ssh
2021-01-11 23:08:04 Chris Halse Rogers cloud-init (Ubuntu Xenial): status New Fix Committed
2021-01-11 23:08:09 Chris Halse Rogers bug added subscriber Ubuntu Stable Release Updates Team
2021-01-11 23:08:13 Chris Halse Rogers bug added subscriber SRU Verification
2021-01-11 23:08:19 Chris Halse Rogers tags verification-needed verification-needed-xenial
2021-01-11 23:09:22 Chris Halse Rogers cloud-init (Ubuntu Bionic): status New Fix Committed
2021-01-11 23:09:34 Chris Halse Rogers tags verification-needed verification-needed-xenial verification-needed verification-needed-bionic verification-needed-xenial
2021-01-11 23:19:26 Chris Halse Rogers cloud-init (Ubuntu Groovy): status New Fix Committed
2021-01-11 23:19:36 Chris Halse Rogers tags verification-needed verification-needed-bionic verification-needed-xenial verification-needed verification-needed-bionic verification-needed-groovy verification-needed-xenial
2021-01-11 23:38:16 Chris Halse Rogers cloud-init (Ubuntu Focal): status New Fix Committed
2021-01-11 23:38:28 Chris Halse Rogers tags verification-needed verification-needed-bionic verification-needed-groovy verification-needed-xenial verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-needed-xenial
2021-01-12 14:44:05 Dan Watkins description == Begin SRU Template == [Impact] This release is only a single functional cherry-pick which solely affects Azure platform. It is a critical bug we wish to release as soon as possible   * Azure: cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys     (#760) (LP: #1910835) The functional changeset here introduces a raise KeyError exception which forces cloud-init to revert to previous released logic of the previous cloud-init public release 20.3. [Test Case] The following development and SRU process was followed: https://wiki.ubuntu.com/CloudinitUpdates The cloud-init team will be in charge of attaching the artifacts and console output of the appropriate run to the bug. cloud-init team members will not mark ‘verification-done’ until this has happened. * Automated Test Results <TODO: attach automated cloud-init-proposed test artifacts from tests for each release with lxd artifacts> <TODO: attach automated cloud-init-proposed test artifacts from tests for each release with kvm artifacts> <TODO: attach automated curtin vmtest with cloud-init proposed> <TODO: attach Solutions Testing team test results for each LTS> * Manual Test Results <TODO: attach manual cloud-init-proposed test artifacts from tests for each release on ec2 datasource> <TODO: attach manual cloud-init-proposed test artifacts from tests for each release on gce datasource> <TODO: attach manual cloud-init-proposed test artifacts from tests for each release on azure datasource> [Regression Potential] In order to mitigate the regression potential, the results of the aforementioned integration tests are attached to this bug. [Discussion] This should only affect public Azure VM launched which use Azure to --generate-ssh-keys either from the dashboard or from the `az cli` Any other cloud-platform is not affected by this change. == End SRU Template ==   * cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys     (#760) (LP: #1910835) == Original Description == cloud-init 20.4 or later will incorrectly add Azure publicKeys to .ssh/authorized_keys preventing ssh access for cloud-generated keys. To reproduce: launch an ubuntu VM from the portal.azure.com choosing to generate new ssh key. When the instance is launched you can see that the ssh-rsa content provided in the metadata publicKeys value contains CRLF characters (\r\n) thus splitting the content of the pubkey onto multiple lines when it is rendered into .ssh/authorized_keys. the solution is either for IMDS to stop adding the CRLF characters or cloud-init to strip them out. Here is the IMDS value provided to cloud-init cloud-init query --format '{{ds.meta_data.imds.compute.publicKeys}}' [{'keyData': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d\r\nk/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R\r\n9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW\r\nlkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq\r\n4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7\r\n6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu\r\niKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht\r\n6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9\r\nS2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure\r\n', 'path': '/home/ubuntu/.ssh/authorized_keys'}] cloud-init renders this directly to .ssh/authorized_keys without processing the string, resulting in an invalid keyline: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d k/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R^M 9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW^M lkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq^M 4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7^M 6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu^M iKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht^M 6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9^M S2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure this prevents ssh from actually reading the right key from azure: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys If we strip the CRLF (^M) characters and reparse with ssh-keygenm we see the proper key registered: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys 3072 SHA256:PQ9EKxTKONJKFC2N56UpL6+Oc/cujfA9HpsF5VW2QDI generated-by-azure (RSA) If cloud-init (or IMDS) were to strip those \r\n characters from each line ssh == Begin SRU Template == [Impact] The previous version of cloud-init used OpenSSL to process the SSH keys provided by the Azure platform. cloud-init 20.4 replaced that code with a more efficient implementation, but one which did not use OpenSSL: this meant that users passing certificates to instances, or users generating SSH keys in Azure's web UI (which inserts \r\n sequences into the public key content), were regressed: their certificates and misformed SSH keys were no longer handled, so they could fail to gain access to newly-launched instances. This release is only a single functional cherry-pick which solely affects Azure platform. It is a critical bug we wish to release as soon as possible   * Azure: cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys     (#760) (LP: #1910835) The functional changeset here introduces a raise KeyError exception which forces cloud-init to revert to previous released logic of the previous cloud-init public release 20.3. [Test Case] As this is a single commit backport, the cloud-init SRU exception need not apply. An upstream integration test has been written for this issue (https://github.com/canonical/cloud-init/blob/master/tests/integration_tests/bugs/test_lp1910835.py). A full run of the upstream test suite on Azure will therefore regression test the update generally and test this issue specifically: a log of a test run for each suite will be attached. [Regression Potential] The proposed change only modifies code paths used on Azure, specifically to revert to a previous behaviour: users unaffected by the bug should see no change (their keys will get to their instance via a different route), and users affected by the bug would have been unable to access their instances before (so cannot be relying on this behaviour in a way which we could break by fixing it). [Discussion] This should only affect public Azure VM launched which use Azure to --generate-ssh-keys either from the dashboard or from the `az cli` Any other cloud-platform is not affected by this change. == End SRU Template ==   * cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys     (#760) (LP: #1910835) == Original Description == cloud-init 20.4 or later will incorrectly add Azure publicKeys to .ssh/authorized_keys preventing ssh access for cloud-generated keys. To reproduce: launch an ubuntu VM from the portal.azure.com choosing to generate new ssh key. When the instance is launched you can see that the ssh-rsa content provided in the metadata publicKeys value contains CRLF characters (\r\n) thus splitting the content of the pubkey onto multiple lines when it is rendered into .ssh/authorized_keys. the solution is either for IMDS to stop adding the CRLF characters or cloud-init to strip them out. Here is the IMDS value provided to cloud-init cloud-init query --format '{{ds.meta_data.imds.compute.publicKeys}}' [{'keyData': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d\r\nk/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R\r\n9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW\r\nlkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq\r\n4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7\r\n6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu\r\niKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht\r\n6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9\r\nS2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure\r\n', 'path': '/home/ubuntu/.ssh/authorized_keys'}] cloud-init renders this directly to .ssh/authorized_keys without processing the string, resulting in an invalid keyline: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d k/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R^M 9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW^M lkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq^M 4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7^M 6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu^M iKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht^M 6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9^M S2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure this prevents ssh from actually reading the right key from azure: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys If we strip the CRLF (^M) characters and reparse with ssh-keygenm we see the proper key registered: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys 3072 SHA256:PQ9EKxTKONJKFC2N56UpL6+Oc/cujfA9HpsF5VW2QDI generated-by-azure (RSA) If cloud-init (or IMDS) were to strip those \r\n characters from each line ssh
2021-01-12 16:48:30 Dan Watkins description == Begin SRU Template == [Impact] The previous version of cloud-init used OpenSSL to process the SSH keys provided by the Azure platform. cloud-init 20.4 replaced that code with a more efficient implementation, but one which did not use OpenSSL: this meant that users passing certificates to instances, or users generating SSH keys in Azure's web UI (which inserts \r\n sequences into the public key content), were regressed: their certificates and misformed SSH keys were no longer handled, so they could fail to gain access to newly-launched instances. This release is only a single functional cherry-pick which solely affects Azure platform. It is a critical bug we wish to release as soon as possible   * Azure: cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys     (#760) (LP: #1910835) The functional changeset here introduces a raise KeyError exception which forces cloud-init to revert to previous released logic of the previous cloud-init public release 20.3. [Test Case] As this is a single commit backport, the cloud-init SRU exception need not apply. An upstream integration test has been written for this issue (https://github.com/canonical/cloud-init/blob/master/tests/integration_tests/bugs/test_lp1910835.py). A full run of the upstream test suite on Azure will therefore regression test the update generally and test this issue specifically: a log of a test run for each suite will be attached. [Regression Potential] The proposed change only modifies code paths used on Azure, specifically to revert to a previous behaviour: users unaffected by the bug should see no change (their keys will get to their instance via a different route), and users affected by the bug would have been unable to access their instances before (so cannot be relying on this behaviour in a way which we could break by fixing it). [Discussion] This should only affect public Azure VM launched which use Azure to --generate-ssh-keys either from the dashboard or from the `az cli` Any other cloud-platform is not affected by this change. == End SRU Template ==   * cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys     (#760) (LP: #1910835) == Original Description == cloud-init 20.4 or later will incorrectly add Azure publicKeys to .ssh/authorized_keys preventing ssh access for cloud-generated keys. To reproduce: launch an ubuntu VM from the portal.azure.com choosing to generate new ssh key. When the instance is launched you can see that the ssh-rsa content provided in the metadata publicKeys value contains CRLF characters (\r\n) thus splitting the content of the pubkey onto multiple lines when it is rendered into .ssh/authorized_keys. the solution is either for IMDS to stop adding the CRLF characters or cloud-init to strip them out. Here is the IMDS value provided to cloud-init cloud-init query --format '{{ds.meta_data.imds.compute.publicKeys}}' [{'keyData': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d\r\nk/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R\r\n9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW\r\nlkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq\r\n4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7\r\n6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu\r\niKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht\r\n6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9\r\nS2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure\r\n', 'path': '/home/ubuntu/.ssh/authorized_keys'}] cloud-init renders this directly to .ssh/authorized_keys without processing the string, resulting in an invalid keyline: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d k/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R^M 9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW^M lkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq^M 4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7^M 6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu^M iKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht^M 6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9^M S2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure this prevents ssh from actually reading the right key from azure: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys If we strip the CRLF (^M) characters and reparse with ssh-keygenm we see the proper key registered: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys 3072 SHA256:PQ9EKxTKONJKFC2N56UpL6+Oc/cujfA9HpsF5VW2QDI generated-by-azure (RSA) If cloud-init (or IMDS) were to strip those \r\n characters from each line ssh == Begin SRU Template == [Impact] The previous version of cloud-init used OpenSSL to process the SSH keys provided by the Azure platform. cloud-init 20.4 replaced that code with a more efficient implementation, but one which did not use OpenSSL: this meant that users passing certificates to instances, or users generating SSH keys in Azure's web UI (which inserts \r\n sequences into the public key content), were regressed: their certificates and malformed SSH keys were no longer handled, so they could fail to gain access to newly-launched instances. This release is only a single functional cherry-pick which solely affects Azure platform. It is a critical bug we wish to release as soon as possible   * Azure: cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys     (#760) (LP: #1910835) The functional changeset here introduces a raise KeyError exception which forces cloud-init to revert to previous released logic of the previous cloud-init public release 20.3. [Test Case] As this is a single commit backport, the cloud-init SRU exception need not apply. An upstream integration test has been written for this issue (https://github.com/canonical/cloud-init/blob/master/tests/integration_tests/bugs/test_lp1910835.py). A full run of the upstream test suite on Azure will therefore regression test the update generally and test this issue specifically: a log of a test run for each suite will be attached. [Regression Potential] The proposed change only modifies code paths used on Azure, specifically to revert to a previous behaviour: users unaffected by the bug should see no change (their keys will get to their instance via a different route), and users affected by the bug would have been unable to access their instances before (so cannot be relying on this behaviour in a way which we could break by fixing it). [Discussion] This should only affect public Azure VM launched which use Azure to --generate-ssh-keys either from the dashboard or from the `az cli` Any other cloud-platform is not affected by this change. == End SRU Template ==   * cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys     (#760) (LP: #1910835) == Original Description == cloud-init 20.4 or later will incorrectly add Azure publicKeys to .ssh/authorized_keys preventing ssh access for cloud-generated keys. To reproduce: launch an ubuntu VM from the portal.azure.com choosing to generate new ssh key. When the instance is launched you can see that the ssh-rsa content provided in the metadata publicKeys value contains CRLF characters (\r\n) thus splitting the content of the pubkey onto multiple lines when it is rendered into .ssh/authorized_keys. the solution is either for IMDS to stop adding the CRLF characters or cloud-init to strip them out. Here is the IMDS value provided to cloud-init cloud-init query --format '{{ds.meta_data.imds.compute.publicKeys}}' [{'keyData': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d\r\nk/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R\r\n9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW\r\nlkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq\r\n4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7\r\n6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu\r\niKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht\r\n6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9\r\nS2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure\r\n', 'path': '/home/ubuntu/.ssh/authorized_keys'}] cloud-init renders this directly to .ssh/authorized_keys without processing the string, resulting in an invalid keyline: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d k/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R^M 9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW^M lkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq^M 4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7^M 6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu^M iKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht^M 6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9^M S2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure this prevents ssh from actually reading the right key from azure: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys If we strip the CRLF (^M) characters and reparse with ssh-keygenm we see the proper key registered: $ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys 3072 SHA256:PQ9EKxTKONJKFC2N56UpL6+Oc/cujfA9HpsF5VW2QDI generated-by-azure (RSA) If cloud-init (or IMDS) were to strip those \r\n characters from each line ssh
2021-01-12 16:48:54 Dan Watkins attachment added xenial verification log https://bugs.launchpad.net/cloud-init/+bug/1910835/+attachment/5452345/+files/xenial.txt
2021-01-12 16:49:59 Dan Watkins attachment added bionic verification log https://bugs.launchpad.net/cloud-init/+bug/1910835/+attachment/5452346/+files/bionic.txt
2021-01-12 16:50:17 Dan Watkins attachment added focal verification log https://bugs.launchpad.net/cloud-init/+bug/1910835/+attachment/5452347/+files/focal.txt
2021-01-12 16:50:34 Dan Watkins attachment added groovy verification log https://bugs.launchpad.net/cloud-init/+bug/1910835/+attachment/5452348/+files/groovy.txt
2021-01-12 17:16:20 Chad Smith cloud-init (Ubuntu Hirsute): status New Fix Released
2021-01-12 17:27:54 Dan Watkins tags verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-needed-xenial verification-done verification-done-bionic verification-done-focal verification-done-groovy verification-done-xenial
2021-01-12 18:55:54 Launchpad Janitor cloud-init (Ubuntu Groovy): status Fix Committed Fix Released
2021-01-12 18:55:58 Brian Murray removed subscriber Ubuntu Stable Release Updates Team
2021-01-12 18:56:18 Launchpad Janitor cloud-init (Ubuntu Focal): status Fix Committed Fix Released
2021-01-12 18:56:55 Launchpad Janitor cloud-init (Ubuntu Bionic): status Fix Committed Fix Released
2021-01-12 18:58:40 Launchpad Janitor cloud-init (Ubuntu Xenial): status Fix Committed Fix Released
2021-01-12 20:54:26 Dan Watkins cloud-init: status Confirmed Fix Released
2023-05-12 11:19:23 James Falcon bug watch added https://github.com/canonical/cloud-init/issues/3825