Activity log for bug #1846535

Date Who What changed Old value New value Message
2019-10-03 17:19:42 Nikolay Vinogradov bug added bug
2019-10-03 17:42:49 David Coronel bug added subscriber David Coronel
2019-10-03 18:29:48 David Coronel bug added subscriber Canonical Field High
2019-10-03 18:40:15 Nikolay Vinogradov attachment added cloud-init-output.log https://bugs.launchpad.net/cloud-init/+bug/1846535/+attachment/5294072/+files/cloud-init-output.log
2019-10-03 18:41:29 Nikolay Vinogradov attachment added configuration with bonds https://bugs.launchpad.net/cloud-init/+bug/1846535/+attachment/5294073/+files/bonds.png
2019-10-03 18:42:14 Nikolay Vinogradov attachment added after removal of the bonds https://bugs.launchpad.net/cloud-init/+bug/1846535/+attachment/5294074/+files/no-bonds.png
2019-10-03 18:42:59 Jason Hobbs bug added subscriber Canonical Field Critical
2019-10-03 18:44:34 Jason Hobbs removed subscriber Canonical Field High
2019-10-03 18:55:54 Robie Basak tags regression-update
2019-10-03 20:25:30 Alexander Litvinov tags regression-update cpe-onsite regression-update
2019-10-03 22:44:00 Ryan Harper bug added subscriber Ryan Harper
2019-10-04 01:20:31 Yoshi Kadokawa bug added subscriber Yoshi Kadokawa
2019-10-04 03:14:50 Nobuto Murata bug added subscriber Nobuto Murata
2019-10-04 03:16:20 Nobuto Murata bug task added cloud-init (Ubuntu)
2019-10-04 06:33:47 Nicolas Pochet bug added subscriber Nicolas Pochet
2019-10-04 09:14:47 Nivedita Singhvi cloud-init (Ubuntu): importance Undecided Critical
2019-10-04 09:16:46 Nivedita Singhvi tags cpe-onsite regression-update cpe-onsite regression-update sts
2019-10-04 13:03:00 Launchpad Janitor cloud-init (Ubuntu): status New Confirmed
2019-10-04 13:29:24 Przemyslaw Hausman bug added subscriber Przemyslaw Hausman
2019-10-04 13:52:32 Launchpad Janitor merge proposal linked https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/373649
2019-10-04 14:13:53 Ivan Hitos bug added subscriber Ivan Hitos
2019-10-04 14:23:31 Dan Watkins cloud-init (Ubuntu): status Confirmed In Progress
2019-10-04 14:23:33 Dan Watkins cloud-init: status New In Progress
2019-10-04 14:23:35 Dan Watkins cloud-init: assignee Dan Watkins (daniel-thewatkins)
2019-10-04 14:23:38 Dan Watkins cloud-init: importance Undecided Critical
2019-10-04 14:23:40 Dan Watkins cloud-init (Ubuntu): assignee Dan Watkins (daniel-thewatkins)
2019-10-04 15:34:45 Server Team CI bot cloud-init: status In Progress Fix Committed
2019-10-04 15:39:21 Launchpad Janitor merge proposal linked https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/373655
2019-10-04 15:43:43 Launchpad Janitor merge proposal linked https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/373656
2019-10-04 15:48:20 Launchpad Janitor merge proposal linked https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/373657
2019-10-04 15:49:00 Dan Watkins nominated for series Ubuntu Eoan
2019-10-04 15:49:00 Dan Watkins bug task added cloud-init (Ubuntu Eoan)
2019-10-04 15:49:00 Dan Watkins nominated for series Ubuntu Xenial
2019-10-04 15:49:00 Dan Watkins bug task added cloud-init (Ubuntu Xenial)
2019-10-04 15:49:00 Dan Watkins nominated for series Ubuntu Disco
2019-10-04 15:49:00 Dan Watkins bug task added cloud-init (Ubuntu Disco)
2019-10-04 15:49:00 Dan Watkins nominated for series Ubuntu Bionic
2019-10-04 15:49:00 Dan Watkins bug task added cloud-init (Ubuntu Bionic)
2019-10-04 15:49:15 Dan Watkins cloud-init (Ubuntu Bionic): status New In Progress
2019-10-04 15:49:17 Dan Watkins cloud-init (Ubuntu Disco): status New In Progress
2019-10-04 15:49:22 Dan Watkins cloud-init (Ubuntu Xenial): status New In Progress
2019-10-04 15:49:27 Dan Watkins cloud-init (Ubuntu Bionic): assignee Dan Watkins (daniel-thewatkins)
2019-10-04 15:49:33 Dan Watkins cloud-init (Ubuntu Disco): assignee Dan Watkins (daniel-thewatkins)
2019-10-04 16:07:56 Launchpad Janitor merge proposal linked https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/373660
2019-10-04 16:26:03 Dan Watkins description Symptoms ======== After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state: $ juju ssh 0 ERROR retrieving SSH host keys for "0": keys not found $ juju machines Machine State DNS Inst id Series AZ Message 0 pending 172.20.10.125 block-3 bionic AZ3 Deployed 1 pending 172.20.10.124 block-2 bionic AZ2 Deployed 2 pending 172.20.10.126 block-1 bionic AZ1 Deployed 3 pending 172.20.10.127 object-2 bionic AZ1 Deployed 4 pending 172.20.10.128 object-1 bionic AZ2 Deployed 5 pending 172.20.10.129 object-3 bionic AZ3 Deployed It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails. We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't. [Impact] Any instances launched with bridges or bonds in their network configuration will fail to bring up networking. [Test Case] [Regression Potential] The change being SRU'd adds more conditions to an existing conditional. There is potential to regress the cases that the existing conditional was introduced to cover, so we will be testing those specifically. Other than that, there was some minor refactoring of the existing conditional statement (which did not change the logic it checks), which could cause issues for Oracle netfailover interfaces. We will also specifically test on Oracle. [Original Report] Symptoms ======== After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state: $ juju ssh 0 ERROR retrieving SSH host keys for "0": keys not found $ juju machines Machine State DNS Inst id Series AZ Message 0 pending 172.20.10.125 block-3 bionic AZ3 Deployed 1 pending 172.20.10.124 block-2 bionic AZ2 Deployed 2 pending 172.20.10.126 block-1 bionic AZ1 Deployed 3 pending 172.20.10.127 object-2 bionic AZ1 Deployed 4 pending 172.20.10.128 object-1 bionic AZ2 Deployed 5 pending 172.20.10.129 object-3 bionic AZ3 Deployed It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails. We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't.
2019-10-04 17:25:10 Chad Smith description [Impact] Any instances launched with bridges or bonds in their network configuration will fail to bring up networking. [Test Case] [Regression Potential] The change being SRU'd adds more conditions to an existing conditional. There is potential to regress the cases that the existing conditional was introduced to cover, so we will be testing those specifically. Other than that, there was some minor refactoring of the existing conditional statement (which did not change the logic it checks), which could cause issues for Oracle netfailover interfaces. We will also specifically test on Oracle. [Original Report] Symptoms ======== After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state: $ juju ssh 0 ERROR retrieving SSH host keys for "0": keys not found $ juju machines Machine State DNS Inst id Series AZ Message 0 pending 172.20.10.125 block-3 bionic AZ3 Deployed 1 pending 172.20.10.124 block-2 bionic AZ2 Deployed 2 pending 172.20.10.126 block-1 bionic AZ1 Deployed 3 pending 172.20.10.127 object-2 bionic AZ1 Deployed 4 pending 172.20.10.128 object-1 bionic AZ2 Deployed 5 pending 172.20.10.129 object-3 bionic AZ3 Deployed It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails. We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't. [Impact] Any instances launched with bridges or bonds in their network configuration will fail to bring up networking. [Test Case] # Juju bootstrap on maas of a machine sets up a network bridge that triggers a failure in cloud-init init stage. # This results in a maas machine deployment failure and the machine gets released Procedure: 1. juju bootrap mymaas --no-gui 2. After bootstrap failure and machine is powered off click on the failed/released node in the MAAS UI 3. Click Rescue Mode from the 'Take Action' drop down in the MAAS UI 4. Grab the IP from the interfaces tab 5. ssh ubuntu@<theRescueMachineIP> -- cloud-init status --long # Expect failure message 6. Click Exit Rescue Mode on the node in MAAS UI. 7. ssh to the maas server add the following lines to /etc/maas/preseeds/curtin_userdata to test official *-proposed packages: system_upgrade: {enabled: True} apt: sources: proposed.list: source: deb $MIRROR $RELEASE-proposed main universe # upstream -proposed 8. Repeat step 1 and expect bootstrap success # expect to see MAASDatasource from bootstrapped machine and no errors 9. juju ssh 0 -- cloud-init status --long [Regression Potential] The change being SRU'd adds more conditions to an existing conditional. There is potential to regress the cases that the existing conditional was introduced to cover, so we will be testing those specifically. Other than that, there was some minor refactoring of the existing conditional statement (which did not change the logic it checks), which could cause issues for Oracle netfailover interfaces. We will also specifically test on Oracle. [Original Report] Symptoms ======== After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state: $ juju ssh 0 ERROR retrieving SSH host keys for "0": keys not found $ juju machines Machine State DNS Inst id Series AZ Message 0 pending 172.20.10.125 block-3 bionic AZ3 Deployed 1 pending 172.20.10.124 block-2 bionic AZ2 Deployed 2 pending 172.20.10.126 block-1 bionic AZ1 Deployed 3 pending 172.20.10.127 object-2 bionic AZ1 Deployed 4 pending 172.20.10.128 object-1 bionic AZ2 Deployed 5 pending 172.20.10.129 object-3 bionic AZ3 Deployed It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails. We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't.
2019-10-04 17:36:03 Steve Langasek cloud-init (Ubuntu Disco): status In Progress Fix Committed
2019-10-04 17:36:06 Steve Langasek bug added subscriber Ubuntu Stable Release Updates Team
2019-10-04 17:36:10 Steve Langasek bug added subscriber SRU Verification
2019-10-04 17:36:17 Steve Langasek tags cpe-onsite regression-update sts cpe-onsite regression-update sts verification-needed verification-needed-disco
2019-10-04 17:36:58 Steve Langasek cloud-init (Ubuntu Bionic): status In Progress Fix Committed
2019-10-04 17:37:08 Steve Langasek tags cpe-onsite regression-update sts verification-needed verification-needed-disco cpe-onsite regression-update sts verification-needed verification-needed-bionic verification-needed-disco
2019-10-04 17:39:50 Chad Smith description [Impact] Any instances launched with bridges or bonds in their network configuration will fail to bring up networking. [Test Case] # Juju bootstrap on maas of a machine sets up a network bridge that triggers a failure in cloud-init init stage. # This results in a maas machine deployment failure and the machine gets released Procedure: 1. juju bootrap mymaas --no-gui 2. After bootstrap failure and machine is powered off click on the failed/released node in the MAAS UI 3. Click Rescue Mode from the 'Take Action' drop down in the MAAS UI 4. Grab the IP from the interfaces tab 5. ssh ubuntu@<theRescueMachineIP> -- cloud-init status --long # Expect failure message 6. Click Exit Rescue Mode on the node in MAAS UI. 7. ssh to the maas server add the following lines to /etc/maas/preseeds/curtin_userdata to test official *-proposed packages: system_upgrade: {enabled: True} apt: sources: proposed.list: source: deb $MIRROR $RELEASE-proposed main universe # upstream -proposed 8. Repeat step 1 and expect bootstrap success # expect to see MAASDatasource from bootstrapped machine and no errors 9. juju ssh 0 -- cloud-init status --long [Regression Potential] The change being SRU'd adds more conditions to an existing conditional. There is potential to regress the cases that the existing conditional was introduced to cover, so we will be testing those specifically. Other than that, there was some minor refactoring of the existing conditional statement (which did not change the logic it checks), which could cause issues for Oracle netfailover interfaces. We will also specifically test on Oracle. [Original Report] Symptoms ======== After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state: $ juju ssh 0 ERROR retrieving SSH host keys for "0": keys not found $ juju machines Machine State DNS Inst id Series AZ Message 0 pending 172.20.10.125 block-3 bionic AZ3 Deployed 1 pending 172.20.10.124 block-2 bionic AZ2 Deployed 2 pending 172.20.10.126 block-1 bionic AZ1 Deployed 3 pending 172.20.10.127 object-2 bionic AZ1 Deployed 4 pending 172.20.10.128 object-1 bionic AZ2 Deployed 5 pending 172.20.10.129 object-3 bionic AZ3 Deployed It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails. We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't. [Impact] Any instances launched with bridges or bonds in their network configuration will fail to bring up networking. [Test Case] # Juju bootstrap on maas of a machine sets up a network bridge that triggers a failure in cloud-init init stage. # This results in a maas machine deployment failure and the machine gets released Procedure: 1. juju bootrap mymaas --no-gui 2. After bootstrap failure and machine is powered off click on the failed/released node in the MAAS UI 3. Click Rescue Mode from the 'Take Action' drop down in the MAAS UI 4. Grab the IP from the interfaces tab 5. ssh ubuntu@<theRescueMachineIP> -- cloud-init status --long # Expect failure message 6. Click Exit Rescue Mode on the node in MAAS UI. 7. ssh to the maas server add the following lines to /etc/maas/preseeds/curtin_userdata to test official *-proposed packages: system_upgrade: {enabled: True} apt:   sources:     proposed.list:        source: deb $MIRROR $RELEASE-proposed main universe # upstream -proposed 8. Repeat step 1 and expect bootstrap success # expect to see MAASDatasource from bootstrapped machine and no errors 9. juju ssh 0 -- cloud-init status --long Also, this test case touches Azure cloud support for accelerated networking. We will be performing our manual SRU test case runs on Azure, OpenStack and Ec2 and attaching logs to confirm that this doesn't regress other clouds. * TODO Azure test template: https://github.com/cloud-init/ubuntu-sru/blob/master/sru-templates/manual/azure-sru * TODO Ec2 test template: https://github.com/cloud-init/ubuntu-sru/blob/master/sru-templates/manual/ec2-sru * TODO OpenStack test template: https://github.com/cloud-init/ubuntu-sru/blob/master/sru-templates/manual/openstack-sru [Regression Potential] The change being SRU'd adds more conditions to an existing conditional. There is potential to regress the cases that the existing conditional was introduced to cover, so we will be testing those specifically. Other than that, there was some minor refactoring of the existing conditional statement (which did not change the logic it checks), which could cause issues for Oracle netfailover interfaces. We will also specifically test on Oracle. [Original Report] Symptoms ======== After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state: $ juju ssh 0 ERROR retrieving SSH host keys for "0": keys not found $ juju machines Machine State DNS Inst id Series AZ Message 0 pending 172.20.10.125 block-3 bionic AZ3 Deployed 1 pending 172.20.10.124 block-2 bionic AZ2 Deployed 2 pending 172.20.10.126 block-1 bionic AZ1 Deployed 3 pending 172.20.10.127 object-2 bionic AZ1 Deployed 4 pending 172.20.10.128 object-1 bionic AZ2 Deployed 5 pending 172.20.10.129 object-3 bionic AZ3 Deployed It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails. We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't.
2019-10-04 17:46:32 Steve Langasek cloud-init (Ubuntu Xenial): status In Progress Fix Committed
2019-10-04 17:46:43 Steve Langasek tags cpe-onsite regression-update sts verification-needed verification-needed-bionic verification-needed-disco cpe-onsite regression-update sts verification-needed verification-needed-bionic verification-needed-disco verification-needed-xenial
2019-10-04 18:42:07 Launchpad Janitor cloud-init (Ubuntu Eoan): status In Progress Fix Released
2019-10-04 19:36:23 Chad Smith description [Impact] Any instances launched with bridges or bonds in their network configuration will fail to bring up networking. [Test Case] # Juju bootstrap on maas of a machine sets up a network bridge that triggers a failure in cloud-init init stage. # This results in a maas machine deployment failure and the machine gets released Procedure: 1. juju bootrap mymaas --no-gui 2. After bootstrap failure and machine is powered off click on the failed/released node in the MAAS UI 3. Click Rescue Mode from the 'Take Action' drop down in the MAAS UI 4. Grab the IP from the interfaces tab 5. ssh ubuntu@<theRescueMachineIP> -- cloud-init status --long # Expect failure message 6. Click Exit Rescue Mode on the node in MAAS UI. 7. ssh to the maas server add the following lines to /etc/maas/preseeds/curtin_userdata to test official *-proposed packages: system_upgrade: {enabled: True} apt:   sources:     proposed.list:        source: deb $MIRROR $RELEASE-proposed main universe # upstream -proposed 8. Repeat step 1 and expect bootstrap success # expect to see MAASDatasource from bootstrapped machine and no errors 9. juju ssh 0 -- cloud-init status --long Also, this test case touches Azure cloud support for accelerated networking. We will be performing our manual SRU test case runs on Azure, OpenStack and Ec2 and attaching logs to confirm that this doesn't regress other clouds. * TODO Azure test template: https://github.com/cloud-init/ubuntu-sru/blob/master/sru-templates/manual/azure-sru * TODO Ec2 test template: https://github.com/cloud-init/ubuntu-sru/blob/master/sru-templates/manual/ec2-sru * TODO OpenStack test template: https://github.com/cloud-init/ubuntu-sru/blob/master/sru-templates/manual/openstack-sru [Regression Potential] The change being SRU'd adds more conditions to an existing conditional. There is potential to regress the cases that the existing conditional was introduced to cover, so we will be testing those specifically. Other than that, there was some minor refactoring of the existing conditional statement (which did not change the logic it checks), which could cause issues for Oracle netfailover interfaces. We will also specifically test on Oracle. [Original Report] Symptoms ======== After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state: $ juju ssh 0 ERROR retrieving SSH host keys for "0": keys not found $ juju machines Machine State DNS Inst id Series AZ Message 0 pending 172.20.10.125 block-3 bionic AZ3 Deployed 1 pending 172.20.10.124 block-2 bionic AZ2 Deployed 2 pending 172.20.10.126 block-1 bionic AZ1 Deployed 3 pending 172.20.10.127 object-2 bionic AZ1 Deployed 4 pending 172.20.10.128 object-1 bionic AZ2 Deployed 5 pending 172.20.10.129 object-3 bionic AZ3 Deployed It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails. We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't. [Impact] Any instances launched with bridges or bonds in their network configuration will fail to bring up networking. [Test Case] # Juju bootstrap on maas of a machine sets up a network bridge that triggers a failure in cloud-init init stage. # This results in a maas machine deployment failure and the machine gets released Procedure: # Alternate steps on a maas machine with a bridge already created A1. confirm a bridge interface is configured for the target machine on interface eno1, name it broam, attach it to a subnet and select auto-assign for IP A2. click deploy -> bionic A3. Once manual deployment fails go to step 2 below # Alternative 2 juju bootstrap failure on maas B1: juju bootrap mymaas --no-gui B2: Once bootstrap fails go to step 2 2. After deployment failure and machine is powered off click on the failed/released node in the MAAS UI 3. Click Rescue Mode from the 'Take Action' drop down in the MAAS UI 4. Grab the IP from the interfaces tab 5. ssh ubuntu@<theRescueMachineIP> -- cloud-init status --long # Expect failure message 6. Click Exit Rescue Mode on the node in MAAS UI. 7. ssh to the maas server add the following lines to /etc/maas/preseeds/curtin_userdata to test official *-proposed packages: system_upgrade: {enabled: True} apt:   sources:     proposed.list:        source: deb $MIRROR $RELEASE-proposed main universe # upstream -proposed 8. Repeat step 1 and expect bootstrap success # expect to see MAASDatasource from bootstrapped machine and no errors 9. juju ssh 0 -- cloud-init status --long [Regression Potential] The change being SRU'd adds more conditions to an existing conditional. There is potential to regress the cases that the existing conditional was introduced to cover, so we will be testing those specifically. Other than that, there was some minor refactoring of the existing conditional statement (which did not change the logic it checks), which could cause issues for Oracle netfailover interfaces. We will also specifically test on Oracle. [Original Report] Symptoms ======== After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state: $ juju ssh 0 ERROR retrieving SSH host keys for "0": keys not found $ juju machines Machine State DNS Inst id Series AZ Message 0 pending 172.20.10.125 block-3 bionic AZ3 Deployed 1 pending 172.20.10.124 block-2 bionic AZ2 Deployed 2 pending 172.20.10.126 block-1 bionic AZ1 Deployed 3 pending 172.20.10.127 object-2 bionic AZ1 Deployed 4 pending 172.20.10.128 object-1 bionic AZ2 Deployed 5 pending 172.20.10.129 object-3 bionic AZ3 Deployed It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails. We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't.
2019-10-04 23:03:56 Chad Smith attachment added oracle-sru-19.2.36.ubuntu2.txt https://bugs.launchpad.net/cloud-init/+bug/1846535/+attachment/5294454/+files/oracle-sru-19.2.36.ubuntu2.txt
2019-10-04 23:04:25 Chad Smith attachment added gce-sru-19.2.36.ubuntu2.txt https://bugs.launchpad.net/cloud-init/+bug/1846535/+attachment/5294455/+files/gce-sru-19.2.36.ubuntu2.txt
2019-10-04 23:06:46 Chad Smith attachment added openstack-sru-19.2.36.ubuntu2.txt https://bugs.launchpad.net/bugs/1846535/+attachment/5294456/+files/openstack-sru-19.2.36.ubuntu2.txt
2019-10-04 23:07:10 Chad Smith attachment added ec2-sru-19.2.36.ubuntu2.txt https://bugs.launchpad.net/bugs/1846535/+attachment/5294457/+files/ec2-sru-19.2.36.ubuntu2.txt
2019-10-04 23:08:20 Chad Smith description [Impact] Any instances launched with bridges or bonds in their network configuration will fail to bring up networking. [Test Case] # Juju bootstrap on maas of a machine sets up a network bridge that triggers a failure in cloud-init init stage. # This results in a maas machine deployment failure and the machine gets released Procedure: # Alternate steps on a maas machine with a bridge already created A1. confirm a bridge interface is configured for the target machine on interface eno1, name it broam, attach it to a subnet and select auto-assign for IP A2. click deploy -> bionic A3. Once manual deployment fails go to step 2 below # Alternative 2 juju bootstrap failure on maas B1: juju bootrap mymaas --no-gui B2: Once bootstrap fails go to step 2 2. After deployment failure and machine is powered off click on the failed/released node in the MAAS UI 3. Click Rescue Mode from the 'Take Action' drop down in the MAAS UI 4. Grab the IP from the interfaces tab 5. ssh ubuntu@<theRescueMachineIP> -- cloud-init status --long # Expect failure message 6. Click Exit Rescue Mode on the node in MAAS UI. 7. ssh to the maas server add the following lines to /etc/maas/preseeds/curtin_userdata to test official *-proposed packages: system_upgrade: {enabled: True} apt:   sources:     proposed.list:        source: deb $MIRROR $RELEASE-proposed main universe # upstream -proposed 8. Repeat step 1 and expect bootstrap success # expect to see MAASDatasource from bootstrapped machine and no errors 9. juju ssh 0 -- cloud-init status --long [Regression Potential] The change being SRU'd adds more conditions to an existing conditional. There is potential to regress the cases that the existing conditional was introduced to cover, so we will be testing those specifically. Other than that, there was some minor refactoring of the existing conditional statement (which did not change the logic it checks), which could cause issues for Oracle netfailover interfaces. We will also specifically test on Oracle. [Original Report] Symptoms ======== After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state: $ juju ssh 0 ERROR retrieving SSH host keys for "0": keys not found $ juju machines Machine State DNS Inst id Series AZ Message 0 pending 172.20.10.125 block-3 bionic AZ3 Deployed 1 pending 172.20.10.124 block-2 bionic AZ2 Deployed 2 pending 172.20.10.126 block-1 bionic AZ1 Deployed 3 pending 172.20.10.127 object-2 bionic AZ1 Deployed 4 pending 172.20.10.128 object-1 bionic AZ2 Deployed 5 pending 172.20.10.129 object-3 bionic AZ3 Deployed It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails. We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't. [Impact] Any instances launched with bridges or bonds in their network configuration will fail to bring up networking. [Test Case] # Juju bootstrap on maas of a machine sets up a network bridge that triggers a failure in cloud-init init stage. # This results in a maas machine deployment failure and the machine gets released Procedure: # Alternate steps on a maas machine with a bridge already created A1. confirm a bridge interface is configured for the target machine on interface eno1, name it broam, attach it to a subnet and select auto-assign for IP A2. click deploy -> bionic A3. Once manual deployment fails go to step 2 below # Alternative 2 juju bootstrap failure on maas B1: juju bootrap mymaas --no-gui B2: Once bootstrap fails go to step 2 2. After deployment failure and machine is powered off click on the failed/released node in the MAAS UI 3. Click Rescue Mode from the 'Take Action' drop down in the MAAS UI 4. Grab the IP from the interfaces tab 5. ssh ubuntu@<theRescueMachineIP> -- cloud-init status --long # Expect failure message 6. Click Exit Rescue Mode on the node in MAAS UI. 7. ssh to the maas server add the following lines to /etc/maas/preseeds/curtin_userdata to test official *-proposed packages: system_upgrade: {enabled: True} apt:   sources:     proposed.list:        source: deb $MIRROR $RELEASE-proposed main universe # upstream -proposed 8. Repeat step 1 and expect bootstrap success # expect to see MAASDatasource from bootstrapped machine and no errors 9. juju ssh 0 -- cloud-init status --long Additional verification checks to avoid regression - DONE oracle - DONE ec2 - DONE openstack - DONE gce - TODO azure [Regression Potential] The change being SRU'd adds more conditions to an existing conditional. There is potential to regress the cases that the existing conditional was introduced to cover, so we will be testing those specifically. Other than that, there was some minor refactoring of the existing conditional statement (which did not change the logic it checks), which could cause issues for Oracle netfailover interfaces. We will also specifically test on Oracle. [Original Report] Symptoms ======== After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state: $ juju ssh 0 ERROR retrieving SSH host keys for "0": keys not found $ juju machines Machine State DNS Inst id Series AZ Message 0 pending 172.20.10.125 block-3 bionic AZ3 Deployed 1 pending 172.20.10.124 block-2 bionic AZ2 Deployed 2 pending 172.20.10.126 block-1 bionic AZ1 Deployed 3 pending 172.20.10.127 object-2 bionic AZ1 Deployed 4 pending 172.20.10.128 object-1 bionic AZ2 Deployed 5 pending 172.20.10.129 object-3 bionic AZ3 Deployed It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails. We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't.
2019-10-04 23:13:13 Chad Smith attachment added nocloud-lxd-sru-19.2.36.ubuntu2.txt https://bugs.launchpad.net/bugs/1846535/+attachment/5294458/+files/nocloud-lxd-sru-19.2.36.ubuntu2.txt
2019-10-04 23:13:53 Chad Smith attachment added nocloud-kvm-sru-19.2.36.ubuntu2.txt https://bugs.launchpad.net/bugs/1846535/+attachment/5294459/+files/nocloud-kvm-sru-19.2.36.ubuntu2.txt
2019-10-04 23:18:28 Chad Smith attachment added azure-sru-19.2.36.ubuntu2.txt https://bugs.launchpad.net/bugs/1846535/+attachment/5294460/+files/azure-sru-19.2.36.ubuntu2.txt
2019-10-04 23:19:06 Chad Smith description [Impact] Any instances launched with bridges or bonds in their network configuration will fail to bring up networking. [Test Case] # Juju bootstrap on maas of a machine sets up a network bridge that triggers a failure in cloud-init init stage. # This results in a maas machine deployment failure and the machine gets released Procedure: # Alternate steps on a maas machine with a bridge already created A1. confirm a bridge interface is configured for the target machine on interface eno1, name it broam, attach it to a subnet and select auto-assign for IP A2. click deploy -> bionic A3. Once manual deployment fails go to step 2 below # Alternative 2 juju bootstrap failure on maas B1: juju bootrap mymaas --no-gui B2: Once bootstrap fails go to step 2 2. After deployment failure and machine is powered off click on the failed/released node in the MAAS UI 3. Click Rescue Mode from the 'Take Action' drop down in the MAAS UI 4. Grab the IP from the interfaces tab 5. ssh ubuntu@<theRescueMachineIP> -- cloud-init status --long # Expect failure message 6. Click Exit Rescue Mode on the node in MAAS UI. 7. ssh to the maas server add the following lines to /etc/maas/preseeds/curtin_userdata to test official *-proposed packages: system_upgrade: {enabled: True} apt:   sources:     proposed.list:        source: deb $MIRROR $RELEASE-proposed main universe # upstream -proposed 8. Repeat step 1 and expect bootstrap success # expect to see MAASDatasource from bootstrapped machine and no errors 9. juju ssh 0 -- cloud-init status --long Additional verification checks to avoid regression - DONE oracle - DONE ec2 - DONE openstack - DONE gce - TODO azure [Regression Potential] The change being SRU'd adds more conditions to an existing conditional. There is potential to regress the cases that the existing conditional was introduced to cover, so we will be testing those specifically. Other than that, there was some minor refactoring of the existing conditional statement (which did not change the logic it checks), which could cause issues for Oracle netfailover interfaces. We will also specifically test on Oracle. [Original Report] Symptoms ======== After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state: $ juju ssh 0 ERROR retrieving SSH host keys for "0": keys not found $ juju machines Machine State DNS Inst id Series AZ Message 0 pending 172.20.10.125 block-3 bionic AZ3 Deployed 1 pending 172.20.10.124 block-2 bionic AZ2 Deployed 2 pending 172.20.10.126 block-1 bionic AZ1 Deployed 3 pending 172.20.10.127 object-2 bionic AZ1 Deployed 4 pending 172.20.10.128 object-1 bionic AZ2 Deployed 5 pending 172.20.10.129 object-3 bionic AZ3 Deployed It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails. We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't. [Impact] Any instances launched with bridges or bonds in their network configuration will fail to bring up networking. [Test Case] # Juju bootstrap on maas of a machine sets up a network bridge that triggers a failure in cloud-init init stage. # This results in a maas machine deployment failure and the machine gets released Procedure: # Alternate steps on a maas machine with a bridge already created A1. confirm a bridge interface is configured for the target machine on interface eno1, name it broam, attach it to a subnet and select auto-assign for IP A2. click deploy -> bionic A3. Once manual deployment fails go to step 2 below # Alternative 2 juju bootstrap failure on maas B1: juju bootrap mymaas --no-gui B2: Once bootstrap fails go to step 2 2. After deployment failure and machine is powered off click on the failed/released node in the MAAS UI 3. Click Rescue Mode from the 'Take Action' drop down in the MAAS UI 4. Grab the IP from the interfaces tab 5. ssh ubuntu@<theRescueMachineIP> -- cloud-init status --long # Expect failure message 6. Click Exit Rescue Mode on the node in MAAS UI. 7. ssh to the maas server add the following lines to /etc/maas/preseeds/curtin_userdata to test official *-proposed packages: system_upgrade: {enabled: True} apt:   sources:     proposed.list:        source: deb $MIRROR $RELEASE-proposed main universe # upstream -proposed 8. Repeat step 1 and expect bootstrap success # expect to see MAASDatasource from bootstrapped machine and no errors 9. juju ssh 0 -- cloud-init status --long Additional verification checks to avoid regression  - DONE oracle  - DONE ec2  - DONE openstack  - DONE gce  - DONE azure - DONE nocloud kvm - DONE nocloud lxd [Regression Potential] The change being SRU'd adds more conditions to an existing conditional. There is potential to regress the cases that the existing conditional was introduced to cover, so we will be testing those specifically. Other than that, there was some minor refactoring of the existing conditional statement (which did not change the logic it checks), which could cause issues for Oracle netfailover interfaces. We will also specifically test on Oracle. [Original Report] Symptoms ======== After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state: $ juju ssh 0 ERROR retrieving SSH host keys for "0": keys not found $ juju machines Machine State DNS Inst id Series AZ Message 0 pending 172.20.10.125 block-3 bionic AZ3 Deployed 1 pending 172.20.10.124 block-2 bionic AZ2 Deployed 2 pending 172.20.10.126 block-1 bionic AZ1 Deployed 3 pending 172.20.10.127 object-2 bionic AZ1 Deployed 4 pending 172.20.10.128 object-1 bionic AZ2 Deployed 5 pending 172.20.10.129 object-3 bionic AZ3 Deployed It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails. We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't.
2019-10-04 23:22:09 Chad Smith tags cpe-onsite regression-update sts verification-needed verification-needed-bionic verification-needed-disco verification-needed-xenial cpe-onsite regression-update sts verification-done verification-done-bionic verification-done-disco verification-done-xenial
2019-10-07 14:47:58 Dan Watkins cloud-init (Ubuntu Xenial): assignee Dan Watkins (daniel-thewatkins)
2019-10-07 20:20:51 Launchpad Janitor cloud-init (Ubuntu Disco): status Fix Committed Fix Released
2019-10-07 20:20:56 Brian Murray removed subscriber Ubuntu Stable Release Updates Team
2019-10-07 20:22:19 Launchpad Janitor cloud-init (Ubuntu Bionic): status Fix Committed Fix Released
2019-10-07 20:22:35 Launchpad Janitor cloud-init (Ubuntu Xenial): status Fix Committed Fix Released
2019-10-08 17:45:27 tom king bug added subscriber tom king
2019-10-09 12:40:15 Andreas Hasenack bug added subscriber Andreas Hasenack
2019-10-29 19:27:34 Joshua Powers removed subscriber Canonical Field Critical
2019-12-19 23:00:04 Chad Smith cloud-init: status Fix Committed Fix Released