It's looking like the issue may be that when next boot is set to pxe, it does not get cleared. So, during the next, next boot is still set to pxe and so on.
Those were the scenario that I tested:
1. Deploy system from MAAS
2. While system boots and before it gets to petitboot menu, do:
a. Reset IPMI back to factory default from ASM web console.
b. Power off system using ipmitool.
c. Again with ipmitool, set next boot device to pxe and power system back on.
3. Observe system pxe booting, doing curtin install and rebooting.
4. When system hits petitboot, it does not attempt to PXE boot nor boot from disk as previously observed.
Here is the execution data:
================================================================================================
1. Deployed IBM S822LC 8335-GTA node from MAAS UI.
2. Used ipmitool to power off and power on after ipmi is restored to factory settings. This wipes out anything maas wrote through ipmi-chassis-config.
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00
01 03 00
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00
01 05 00 00 00 00 00
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin chassis bootdev pxe
Set Boot Device to pxe
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00
01 03 00
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00
01 05 80 04 00 00 00
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin chassis power on
Chassis Power Control: Up/On
3. I let the deployment proceed and monitored IPMI raw settings during the curtin installation and prior to system being rebooted.
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00
01 05 80 04 00 00 00
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00
01 03 00
4. After system hangs in petitboot, again queried raw boot settings using ipmitool:
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00
01 05 80 04 00 00 00
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00
01 03 00
================================================================================================
So with maas essentially ruled out, I wanted to try a variation of this test. Basically, I retried step 1 through 4 (with 1c modified to skip the step to set next boot to PXE). Without next boot device set to PXE, I would wait for petitboot menu and manually select the PXE boot device (it's execute entry), then hit ENTER:
Here is data:
================================================================================================
1. Again deployed the node from MAAS.
2. While system was still booting but before it reached petitboot. I restored ipmi to factory setting from the web console. I then used ipmitool to power off and power on (taking maas out of the equation as in earlier test).
ubuntu@maas-trusty-back-may22:/usr/lib/python2.7/dist-packages/provisioningserver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin chassis power off
Chassis Power Control: Down/Off
ubuntu@maas-trusty-back-may22:/usr/lib/python2.7/dist-packages/provisioningserver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00
01 05 00 00 00 00 00
ubuntu@maas-trusty-back-may22:/usr/lib/python2.7/dist-packages/provisioningserver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00
01 03 00
ubuntu@maas-trusty-back-may22:/usr/lib/python2.7/dist-packages/provisioningserver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin chassis power on
Chassis Power Control: Up/On
ubuntu@maas-trusty-back-may22:/usr/lib/python2.7/dist-packages/provisioningserver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00
01 05 00 00 00 00 00
ubuntu@maas-trusty-back-may22:/usr/lib/python2.7/dist-packages/provisioningserver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00
01 03 00
3. Halted automatic boot in petitboot menu to manually select the PXE NIC's execute entry. Monitored raw data through curtin installation.
ubuntu@maas-trusty-back-may22:/usr/lib/python2.7/dist-packages/provisioningserver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00
01 05 80 00 00 00 00
ubuntu@maas-trusty-back-may22:/usr/lib/python2.7/dist-packages/provisioningserver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00
01 03 00
4. After curtin, system booted normally from disk and I could see the raw boot settings having changed 01 05 00 00 00 00 00:
ubuntu@maas-trusty-back-may22:/usr/lib/python2.7/dist-packages/provisioningserver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00
01 05 00 00 00 00 00
ubuntu@maas-trusty-back-may22:/usr/lib/python2.7/dist-packages/provisioningserver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00
01 03 00
================================================================================================
It's looking like the issue may be that when next boot is set to pxe, it does not get cleared. So, during the next, next boot is still set to pxe and so on.
Those were the scenario that I tested:
1. Deploy system from MAAS
2. While system boots and before it gets to petitboot menu, do:
a. Reset IPMI back to factory default from ASM web console.
b. Power off system using ipmitool.
c. Again with ipmitool, set next boot device to pxe and power system back on.
3. Observe system pxe booting, doing curtin install and rebooting.
4. When system hits petitboot, it does not attempt to PXE boot nor boot from disk as previously observed.
Here is the execution data: ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= ===== config. ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= =====
=======
1. Deployed IBM S822LC 8335-GTA node from MAAS UI.
2. Used ipmitool to power off and power on after ipmi is restored to factory settings. This wipes out anything maas wrote through ipmi-chassis-
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00
01 03 00
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00
01 05 00 00 00 00 00
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin chassis bootdev pxe
Set Boot Device to pxe
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00
01 03 00
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00
01 05 80 04 00 00 00
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin chassis power on
Chassis Power Control: Up/On
3. I let the deployment proceed and monitored IPMI raw settings during the curtin installation and prior to system being rebooted.
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00
01 05 80 04 00 00 00
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00
01 03 00
4. After system hangs in petitboot, again queried raw boot settings using ipmitool:
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00
01 05 80 04 00 00 00
$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00
01 03 00
=======
So with maas essentially ruled out, I wanted to try a variation of this test. Basically, I retried step 1 through 4 (with 1c modified to skip the step to set next boot to PXE). Without next boot device set to PXE, I would wait for petitboot menu and manually select the PXE boot device (it's execute entry), then hit ENTER:
Here is data: ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= ===== maas-trusty- back-may22: /usr/lib/ python2. 7/dist- packages/ provisioningser ver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin chassis power off maas-trusty- back-may22: /usr/lib/ python2. 7/dist- packages/ provisioningser ver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00 maas-trusty- back-may22: /usr/lib/ python2. 7/dist- packages/ provisioningser ver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00 maas-trusty- back-may22: /usr/lib/ python2. 7/dist- packages/ provisioningser ver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin chassis power on maas-trusty- back-may22: /usr/lib/ python2. 7/dist- packages/ provisioningser ver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00 maas-trusty- back-may22: /usr/lib/ python2. 7/dist- packages/ provisioningser ver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00 maas-trusty- back-may22: /usr/lib/ python2. 7/dist- packages/ provisioningser ver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00 maas-trusty- back-may22: /usr/lib/ python2. 7/dist- packages/ provisioningser ver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00 maas-trusty- back-may22: /usr/lib/ python2. 7/dist- packages/ provisioningser ver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x05 0x00 0x00 maas-trusty- back-may22: /usr/lib/ python2. 7/dist- packages/ provisioningser ver$ ipmitool -I lanplus -H 192.168.224.118 -U ADMIN -P admin raw 0x00 0x09 0x03 0x00 0x00 ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= ======= =====
=======
1. Again deployed the node from MAAS.
2. While system was still booting but before it reached petitboot. I restored ipmi to factory setting from the web console. I then used ipmitool to power off and power on (taking maas out of the equation as in earlier test).
ubuntu@
Chassis Power Control: Down/Off
ubuntu@
01 05 00 00 00 00 00
ubuntu@
01 03 00
ubuntu@
Chassis Power Control: Up/On
ubuntu@
01 05 00 00 00 00 00
ubuntu@
01 03 00
3. Halted automatic boot in petitboot menu to manually select the PXE NIC's execute entry. Monitored raw data through curtin installation.
ubuntu@
01 05 80 00 00 00 00
ubuntu@
01 03 00
4. After curtin, system booted normally from disk and I could see the raw boot settings having changed 01 05 00 00 00 00 00:
ubuntu@
01 05 00 00 00 00 00
ubuntu@
01 03 00
=======