crash trying to install Focal 20.04.4 on a ppc64le

[Patricia Domingues]

while testing new Focal.4 on a pcc64le(POWER8) ` ( 20220218.1/) current/ - 2022-02-19 02:23` I was not able to move from `Configure Ubuntu archive mirror` after that it fails:

```
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-100-generic ppc64le)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

  System information as of Mon Feb 21 22:55:59 UTC 2022

  System load: 0.05 Users logged in: 0
  Usage of /home: unknown IPv4 address for enP34p1s0f0: 10.245.70.45
  Memory usage: 8% IPv4 address for enP34p1s0f1: 10.245.71.183
  Swap usage: 0% IPv4 address for enP34p1s0f2: 10.245.70.41
  Processes: 1225 IPv4 address for enP34p1s0f3: 10.245.70.46

0 updates can be applied immediately.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

connecting...
generating crash report
report saved to /var/crash/1645484181.109517336.ui.crash
Traceback (most recent call last):
  File "/snap/subiquity/3094/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/snap/subiquity/3094/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/__main__.py", line 5, in <module>
    sys.exit(main())
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/cmd/tui.py", line 149, in main
    subiquity_interface.run()
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/client/client.py", line 404, in run
    super().run()
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquitycore/tui.py", line 374, in run
    super().run()
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquitycore/core.py", line 135, in run
    raise exc
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/client/client.py", line 480, in move_screen
    await super().move_screen(increment, coro)
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquitycore/tui.py", line 237, in move_screen
    view = await self.wait_with_progress(
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquitycore/tui.py", line 206, in wait_with_progress
    return await self._wait_with_indication(awaitable, self.show_progress)
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquitycore/tui.py", line 163, in _wait_with_indication
    result = await awaitable
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquitycore/tui.py", line 226, in _move_screen
    return await self.make_view_for_controller(new)
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/client/client.py", line 501, in make_view_for_controller
    view = await super().make_view_for_controller(new)
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquitycore/tui.py", line 132, in make_view_for_controller
    view = await maybe_view
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/client/controllers/filesystem.py", line 64, in make_ui
    return self.make_guided_ui(status)
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/client/controllers/filesystem.py", line 76, in make_guided_ui
    self.app.show_error_report(status.error_report)
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/client/client.py", line 571, in show_error_report
    self.add_global_overlay(ErrorReportStretchy(self, error_ref))
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/ui/views/error.py", line 178, in __init__
    (w, self.pile.options('pack')) for w in self._pile_elements()]
  File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/ui/views/error.py", line 215, in _pile_elements
    assert self.report
AssertionError
```

[Michael Hudson-Doyle (mwhudson)]

Are there crash reports in /var/crash?

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1961640

[Patricia Domingues (patriciasd)]

sure, attaching it.
Let me know if I can provide any other info or test

[Dan Bungert (dbungert)]

Thanks for the crash logs Patricia.

   File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/server/controllers/filesystem.py", line 427, in _probe
     await asyncio.wait_for(self._probe_once_task.task, 15.0)
   File "/snap/subiquity/3094/usr/lib/python3.8/asyncio/tasks.py", line 494, in wait_for
     return fut.result()
   File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquitycore/context.py", line 148, in decorated_async
     return await meth(self, **kw)
   File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/server/controllers/filesystem.py", line 412, in _probe_once
     self.model.load_probe_data(storage)
   File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/models/filesystem.py", line 1287, in load_probe_data
     self.reset()
   File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/models/filesystem.py", line 965, in reset
     self._orig_config = storage_config.extract_storage_config(
   File "/snap/subiquity/3094/lib/python3.8/site-packages/curtin/storage_config.py", line 1346, in extract_storage_config
     tree = get_config_tree(cfg.get('id'), final_config)
   File "/snap/subiquity/3094/lib/python3.8/site-packages/curtin/storage_config.py", line 306, in get_config_tree
     for dep in find_item_dependencies(item, sconfig):
   File "/snap/subiquity/3094/lib/python3.8/site-packages/curtin/storage_config.py", line 276, in find_item_dependencies
     _validate_dep_type(item_id, dep_key, dep, config)
   File "/snap/subiquity/3094/lib/python3.8/site-packages/curtin/storage_config.py", line 223, in _validate_dep_type
     raise ValueError(
 ValueError: Invalid dep_id (disk-dm-1) not in storage config

[Patricia Domingues (patriciasd)]

same happening with latest image (20220221) - current/ 2022-02-22 08:28
```
 2022-02-22 17:56:15,785 ERROR root:39 finish: subiquity/Filesystem/_probe/probe_once: FAIL: Invalid dep_id (disk-dm-1) not in storage config
 2022-02-22 17:56:15,785 ERROR block-discover:433 block probing failed restricted=False
 Traceback (most recent call last):
   File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/server/controllers/filesystem.py", line 427, in _probe
     await asyncio.wait_for(self._probe_once_task.task, 15.0)
   File "/snap/subiquity/3094/usr/lib/python3.8/asyncio/tasks.py", line 494, in wait_for
     return fut.result()
   File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquitycore/context.py", line 148, in decorated_async
     return await meth(self, **kw)
   File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/server/controllers/filesystem.py", line 412, in _probe_once
     self.model.load_probe_data(storage)
   File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/models/filesystem.py", line 1287, in load_probe_data
     self.reset()
   File "/snap/subiquity/3094/lib/python3.8/site-packages/subiquity/models/filesystem.py", line 965, in reset
     self._orig_config = storage_config.extract_storage_config(
   File "/snap/subiquity/3094/lib/python3.8/site-packages/curtin/storage_config.py", line 1346, in extract_storage_config
     tree = get_config_tree(cfg.get('id'), final_config)
   File "/snap/subiquity/3094/lib/python3.8/site-packages/curtin/storage_config.py", line 306, in get_config_tree
     for dep in find_item_dependencies(item, sconfig):
   File "/snap/subiquity/3094/lib/python3.8/site-packages/curtin/storage_config.py", line 276, in find_item_dependencies
     _validate_dep_type(item_id, dep_key, dep, config)
   File "/snap/subiquity/3094/lib/python3.8/site-packages/curtin/storage_config.py", line 223, in _validate_dep_type
     raise ValueError(
 ValueError: Invalid dep_id (disk-dm-1) not in storage config
```

[Patricia Domingues (patriciasd)]

yesterday I saw this issue only on Power8 systems and I was able to install image 20220218.1/ on two different Power9 systems, but today I cannot install it with `20220221` image. attaching the logs for this system (my own reference: baltar)

[Dan Bungert (dbungert)]

Are you able to retest with a kernel option on the command line? If you pass subiquity-channel=edge that should pick up our proposed fix. Or if you can get to a command line in the install environment you can try `sudo snap refresh subiquity --edge`.

[Alexandre Erwin Ittner (aittner)]

@dbungert: tried with subiquity 22.02.1+git5.bd4da8b0 (3108); installer crashed several times, allowing me to advance the install until a random point every time. Then it crashed for real and prevented any progress.

I'm attaching everything I got from /var/log after a few crashes.

Now I noticed that 22.02.1+git7.b67c0b7d is now available. I'm currently testing again.

[Alexandre Erwin Ittner (aittner)]

Oh, just posted this in the wrong bug! It was intended to #1961628 ... copying there.

[Frank Heimes (fheimes)]

I tested with subiquity from edge:
root@ubuntu-server:/# snap list subiquity
Name Version Rev Tracking Publisher Notes
subiquity 22.02.2 3120 latest/edge canonical* classic
and it seems to work now.

I was able to complete a complete default installation, incl. post-install reboot and login:
ubuntu@gulpin:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal
ubuntu@gulpin:~$ uname -r
5.4.0-100-generic
ubuntu@gulpin:~$ uptime
 07:50:35 up 2 min, 1 user, load average: 0.45, 0.21, 0.08

I shamelessly used a P8 system from patriciasd (gulpin) that was off anyway.

I can't try atm on P9, since there is no bare-metal available right now.

But generally I think subiquity 3120 fixed this.

[Patricia Domingues (patriciasd)]

Thanks Dan, I was able to install on a Power9 with `subiquity 22.02.2 3120 latest/edge canonical* classic`: `Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-100-generic ppc64le)` image `20220223`

[Dan Bungert (dbungert)]

Thanks Patricia and Frank.

The fix to allow the point release to move forward was to not have Subiquity use os-prober temporarily. We have addressed that and are now using a newer version of os-prober which should not have the device mapper problems seen in this LP. When you get a moment, can you re-verify?

[Patricia Domingues (patriciasd)]

Hi Dan, I got to run 3 tests:

1. tried with subiquity edge and it got stuck, didn't finish the `installing system` / no error/ no crash.

2. tried with stable subiquity, no updates and it was fine:
 ```Starting Subiquity revision 3120
Ubuntu-Server 20.04.4 LTS "Focal Fossa" - Release ppc64el (20220301)patricia@baltar:/var/log/installer$```

3. repeated 1 and got the same:

1 and 3 got stuck on ` executing late commands `:
```
  Installing system [ Help ]
================================================================================
  ┌──────────────────────────────────────────────────────────────────────────┐
  │ configuring iscsi service ^│
  │ configuring raid (mdadm) service │
  │ installing kernel │
  │ setting up swap │
  │ apply networking config │
  │ writing etc/fstab │
  │ configuring multipath │
  │ updating packages on target system │
  │ configuring pollinate user-agent on target │
  │ updating initramfs configuration │
  │ configuring target system bootloader │
  │ installing grub to target devices │
  │ finalizing installation │
  │ running 'curtin hook' │
  │ curtin command hook ││
  │ executing late commands v│
  └──────────────────────────────────────────────────────────────────────────┘

                               [ View full log ]

┌──────────────────────────── Full installer output ───────────────────────────┐
│start: cmd-install/stage-hook: finalizing installation ^│
│start: cmd-install/stage-hook/builtin: running 'curtin hook' │
│start: cmd-install/stage-hook/builtin/cmd-hook: curtin command hook │
│Finalizing /target │
│finish: cmd-install/stage-hook/builtin/cmd-hook: SUCCESS: curtin command hook │
│finish: cmd-install/stage-hook/builtin: SUCCESS: running 'curtin hook' │
│builtin took 0.665 seconds │
│stage_hook took 0.666 seconds │
│finish: cmd-install/stage-hook: SUCCESS: finalizing installation │
│start: cmd-install/stage-late: executing late commands │
│stage_late took 0.000 seconds │
│finish: cmd-install/stage-late: SUCCESS: executing late commands │
│curtin: Installation finished. ...

[Patricia Domingues (patriciasd)]

Hey Dan, same is happening on a POWER8 node, but now the subiquity from edge is newer `subiquity 22.02.2+git48.6927d3be 3153`: got stuck on the same point:

x running 'curtin hook' x
x curtin command hook xx
x executing late commands vx

[Patricia Domingues (patriciasd)]

Hi Dan, I was checking this one again and not able to install on a Power9 with subiquity from edge (22.02.2+git149.689a7fd5 3231) let me know if you need any other tests

[Michael Hudson-Doyle (mwhudson)]

Hi, if you can still reproduce this, can you attach the crash report?

[Patricia Domingues (patriciasd)]

Hi. not able to reproduce with latest subiquity from edge (3263):
`22.02.2+git176.1e833a54 3263 `

`:~$ cat /var/log/installer/media-info
Ubuntu-Server 20.04.4 LTS "Focal Fossa" - Release ppc64el (20220329)patricia@baltar:~$
`

[Michael Hudson-Doyle (mwhudson)]

Great, thanks for checking. Lets mark this bug fix released and file new bugs for any new problems.