get_pending_boot_mode: 'NoneType' object has no attribute 'upper'

[frigo]

Following
https://storyboard.openstack.org/#!/story/2008717
I am deactivating VID and relaunching the clean-up.

Cleaning fail with:
last_error: 'Failed to prepare node 00000000-0000-0000-0000-000000000001 for cleaning:
  ''NoneType'' object has no attribute ''upper'''

```
Mar 15 09:59:00 director ironic-conductor[2021]: 2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils [req-a93fe396-fb3b-43c3-b7ee-77ef20e1eb3b bifrost_user - - - -] Failed to prepare node 00000000-0000-0000-0000-000000000001 for cleaning: 'NoneType' object has no attribute 'upper': AttributeError: 'NoneType' object has no attribute 'upper'
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils Traceback (most recent call last):
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils File "/opt/stack/bifrost/lib64/python3.6/site-packages/ironic/conductor/cleaning.py", line 86, in do_node_clean
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils prepare_result = task.driver.deploy.prepare_cleaning(task)
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils File "/opt/stack/bifrost/lib64/python3.6/site-packages/ironic_lib/metrics.py", line 59, in wrapped
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils result = f(*args, **kwargs)
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils File "/opt/stack/bifrost/lib64/python3.6/site-packages/ironic/drivers/modules/agent_base.py", line 771, in prepare_cleaning
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils task, manage_boot=self.should_manage_boot(task))
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils File "/opt/stack/bifrost/lib64/python3.6/site-packages/ironic/drivers/modules/deploy_utils.py", line 656, in prepare_inband_cleaning
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils task.driver.boot.prepare_ramdisk(task, ramdisk_opts)
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils File "/opt/stack/bifrost/lib64/python3.6/site-packages/ironic_lib/metrics.py", line 59, in wrapped
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils result = f(*args, **kwargs)
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils File "/opt/stack/bifrost/lib64/python3.6/site-packages/ironic/drivers/modules/ilo/boot.py", line 436, in prepare_ramdisk
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils prepare_node_for_deploy(task)
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils File "/opt/stack/bifrost/lib64/python3.6/site-packages/ironic/drivers/modules/ilo/boot.py", line 316, in prepare_node_for_deploy
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils ilo_common.update_boot_mode(task)
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils File "/opt/stack/bifrost/lib64/python3.6/site-packages/ironic/drivers/modules/ilo/common.py", line 609, in update_boot_mode
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils set_boot_mode(node, boot_mode)
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils File "/opt/stack/bifrost/lib64/python3.6/site-packages/ironic/drivers/modules/ilo/common.py", line 544, in set_boot_mode
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils p_boot_mode = ilo_object.get_pending_boot_mode()
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils File "/opt/stack/bifrost/lib64/python3.6/site-packages/proliantutils/ilo/client.py", line 493, in get_pending_boot_mode
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils return self._call_method('get_pending_boot_mode')
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils File "/opt/stack/bifrost/lib64/python3.6/site-packages/proliantutils/ilo/client.py", line 350, in _call_method
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils return method(*args, **kwargs)
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils File "/opt/stack/bifrost/lib64/python3.6/site-packages/proliantutils/ilo/ris.py", line 1098, in get_pending_boot_mode
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils return boot_mode.upper()
2021-03-15 09:59:00.740 2021 ERROR ironic.conductor.utils AttributeError: 'NoneType' object has no attribute 'upper'
2021-03-1
```

this is using
proliantutils 2.10.0
ironic 16.2.1.dev6

[frigo (rigault-francois)]

Following the deactivation of VID, it seems the server needed to be rebooted once, then the clean-up starts fine.

[Shivanand Tendulker (stendulker)]

Is this issue observed on Gen9 or Gen10 servers? Can you please provide more details about exact steps you had followed to get this error, including how you enabled and disabled the VID.

[frigo (rigault-francois)]

sure! Basically I added some Gen9 blades part of C7000 enclosures managed by HPE Oneview, into a standalone Ironic installation.
Following a clean-up of the node, I discovered that Ironic failed during clean-up, and I opened https://storyboard.openstack.org/#!/story/2008717 to track this.
The reason for the failure is Ironic trying to wipe read only file systems such as the VID.

To work around Ironic clean-up, I am deactivating, through HPE Oneview, the VID setting. The "exact step" I followed to deactivate VID was to run an Ansible playbook to deactivate the server profile, using parameter:
```
          bios:
            manageBios: true
            overriddenSettings:
              - id: VirtualInstallDisk
                value: Disabled
            ...
```

At this point I tried to provide the node, but it failed and I opened this ticket.
However, missing from this ticket: I think at this step the server was in bad shape and "lost" somehow, its configuration. It needed a few reboots and the server profile to be forced (re applied?) again from Oneview.
Following the re-application of the server profile, the blade works just fine.
I think I have only a couple of blades configured with VID enabled.