curtin fails to find itself in subiquity snap, when running on serial-subiquity@.service

[Dimitri John Ledkov]

serial-subiquity@.service runs very differently to the tty1 subiquity.

In seiral-subiquity@ case, it doesn't block starting on snapd as it doesn't use /usr/bin/snap run.

Also, it doesn't execute subiquity.subiquity-service script, meaning that PATH setting there is not exported, causing troubles for curtin to reexec itself, or to find the right python in $PATH.

Fix that in both livecd-rootfs & subiquity.

https://github.com/CanonicalLtd/subiquity/pull/428
https://code.launchpad.net/~xnox/livecd-rootfs/+git/livecd-rootfs/+merge/364888

[Dan Watkins (oddbloke)]

It's not clear to me why this is a bug; users shouldn't be driving the curtin in subiquity's snap themselves (right?), and the only evidence here is that that usecase breaks. Could you expand on what user-facing issue this is causing?

subiquity doesn't expose the curtin executable as an "app", so adding /snap/bin to PATH won't help. (Looking at the code, it will try using `which` before it tries constructing paths itself.)

The curtin binary being installed in /bin seems like a reasonable thing to support.

[Dan Watkins (oddbloke)]

(Marked as Incomplete until we have some more info on what's motivating this bug report.)

[Dimitri John Ledkov (xnox)]

Subiquity calls curtin, which then fails to find curtin_exe, which then fails the installation. With all subiquity disco images currently failing to install.

It is correct that curtin is not exposed to end users. The code snippets I show is the envrionment that curtin is executed in, and fails to find itself, when executed by subiquity installpath.py module.

[Dimitri John Ledkov (xnox)]

Ditto with 727 subiquity snap.

[Ryan Harper (raharper)]

Curtin is designed to be called by it's shell wrapper (bin/curtin) in the code.

I would suggest getting that into PATH and invoking 'curtin'

[Ryan Harper (raharper)]

What's new in subiquity that we now see this?

[Michael Hudson-Doyle (mwhudson)]

Oh, it might have helped if I saw this bug before commenting on the MP. It explains the "what's new so that this is broken" stuff, which is nothing: this way of invoking curtin has never worked.

The reason this isn't a problem in the install environment is this line:

export PATH=$SNAP/bin:$SNAP/usr/bin:$PATH

in bin/subiquity-service in subiquity which I added as part of my "use curtin from the snap" changes. Making it easier to invoke curtin from the shell like this would be nice.

[Dimitri John Ledkov (xnox)]

Ok, but with even the respun images, i still see the same failure on ppc64le. Let me attach a few logs.

[Dimitri John Ledkov (xnox)]

Looks like there are ordering issues wrt. the default console to use hvc0 vs tty0, As I see lots of errors like this one:
Mar 19 14:44:32 ubuntu-server agetty[1923]: hvc0: can't exec /snap/bin/subiquity: No such file or directory

I see squashfs errors:
$ journalctl --file ./system.journal | grep SQUASHFS
Mar 19 14:44:14 ubuntu-server kernel: SQUASHFS error: squashfs_read_data failed to read block 0x8180e1e300333a37
Mar 19 14:44:14 ubuntu-server kernel: SQUASHFS error: Unable to read metadata cache entry [8180e1e300333a37]
Mar 19 14:44:16 ubuntu-server kernel: SQUASHFS error: squashfs_read_data failed to read block 0x8180e1e38a089b3a
Mar 19 14:44:16 ubuntu-server kernel: SQUASHFS error: Unable to read metadata cache entry [8180e1e38a089b3a]
Mar 19 14:45:03 ubuntu-server kernel: SQUASHFS error: Unable to read metadata cache entry [8180e1e38a089b3a]

At least in journal, curtin does fail:
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: Traceback (most recent call last):
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: File "/snap/subiquity/666/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: "__main__", mod_spec)
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: File "/snap/subiquity/666/usr/lib/python3.6/runpy.py", line 85, in _run_code
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: exec(code, run_globals)
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: File "/snap/subiquity/666/lib/python3.6/site-packages/curtin/__main__.py", line 4, in <module>
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: sys.exit(main())
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: File "/snap/subiquity/666/lib/python3.6/site-packages/curtin/commands/main.py", line 177, in main
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: raise OSError("Unable to find helpers or 'curtin' exe to add to path")
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: OSError: Unable to find helpers or 'curtin' exe to add to path

Funily enough the above is in the journal, and not in the subiquity log.

Subiquity installer log has:
2019-03-19 14:46:58,841 subiquity.controllers.snaplist:103 loading list of snaps failed
Traceback (most recent call last):
  File "/snap/subiquity/666/lib/python3.6/site-packages/subiquity/controllers/snaplist.py", line 101, in _fetched_list
    response.raise_for_status()
  File "/snap/subiquity/666/usr/lib/python3/dist-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+unix://%2Frun%2Fsnapd.socket/v2/find?section=server

There is also this nice piece of traceback

2019-03-19 14:47:00,750 subiquitycore.core:430 Exception in controller.run():
Traceback (most recent call last):
  File "/snap/subiquity/666/usr/lib/python3/dist-packages/urwid/main_loop.py", line 384, in _run
    self.event_loop.run()
  File "/snap/subiquity/666/usr/lib/python3/dist-packages/urwid/main_loop.py", line 788, in run
    self._loop()
  File "/snap/subiquity/666/usr/lib/python3/dist-packages/urwid/main_loop.py", line 825, in _loo...

[Dimitri John Ledkov (xnox)]

Maybe easier to read the curtin failure with this pastebinit:

https://paste.ubuntu.com/p/XDjhfHQxTK/

[Dimitri John Ledkov (xnox)]

Squashfs errors: https://paste.ubuntu.com/p/FhKBw2Hwp8/

Agetty failing: https://paste.ubuntu.com/p/HvGHqwcjK3/

snapstore fail?!: https://paste.ubuntu.com/p/TGMnc5v8Zg/

possibly i may need to file extra bugs about this.

[Dimitri John Ledkov (xnox)]

Ok! Progress!

If i give the VM network, and provide the right proxy, and restart snapd to use the proxy.... install works!

[Dimitri John Ledkov (xnox)]

New fail:

│ + echo grub-install /dev/sda │
│ grub-install /dev/sda │
│ + grub-install /dev/sda │
│ Installing for powerpc-ieee1275 platform. │
│ grub-install: error: the chosen partition is not a PReP │
│partition. │
│ + exit │
│ failed to install grub! │
│

[Ryan Harper (raharper)]

For PReP, you need:

https://git.launchpad.net/curtin/tree/examples/tests/basic.yaml#n85

  - id: pnum_disk_p2
        type: partition
        number: 2
        size: 8MB
        device: pnum_disk
        flag: prep
        wipe: zero
        name: prep

[Michael Hudson-Doyle (mwhudson)]

This is fixed now I think? I guess the livecd-rootfs change needs to go back to bionic?

[Jim K (jimk2048)]

I received the error below, on a physical device. The device is an APU2D4 with a 120 GB mSATA SSD and has no video, so I was using the serial port as a putty terminal.

vendor site on the device: https://www.pcengines.ch/apu2d4.htm

I was following this article to use the serial port during install:

http://www.sundby.com/index.php/install-ubuntu-16-04-with-usb-stick-and-serial-console/

I was using the ubuntu-18.04.2-live-server-amd64.iso image loaded on the USB with RUFUS.

An error occurred during installation

┌──────────────────────────── Full installer output ───────────────────────────┐
│Traceback (most recent call last): │
│ File "/snap/subiquity/664/usr/lib/python3.6/runpy.py", line 193, in │
│_run_module_as_main │
│ "__main__", mod_spec) │
│ File "/snap/subiquity/664/usr/lib/python3.6/runpy.py", line 85, in _run_code│
│ exec(code, run_globals) │
│ File "/snap/subiquity/664/lib/python3.6/site-packages/curtin/__main__.py", │
│line 4, in <module> │
│ sys.exit(main()) │
│ File │
│"/snap/subiquity/664/lib/python3.6/site-packages/curtin/commands/main.py", │
│line 177, in main │
│ raise OSError("Unable to find helpers or 'curtin' exe to add to path") │
│OSError: Unable to find helpers or 'curtin' exe to add to path │
│ │
└──────────────────────────────────────────────────────────────────────────────┘
                                 [ Close ]

[Jake Billo (ev98)]

I also receive the same "Unable to find helpers or 'curtin' exe to add to path" error as Jim K reported in https://bugs.launchpad.net/curtin/+bug/1820604/comments/18, after defining a partition layout in the installer. I am installing on a Lanner FW-8758 using a Rufus-generated USB stick and have tried both "ubuntu-18.04.2-live-server-amd64.iso" and "ubuntu-18.04.2-server-amd64.iso".

I activated the serial console output using similar instructions to https://github.com/ynkjm/ubuntu-serial-install - mainly adding "console=tty0 console=ttyS0,115200n8" where necessary.

Unfortunately dropping to shell after the failure occurs won't let me log in to investigate further, or at least I don't know the appropriate credentials (root and ubuntu users with a blank password, or the same username/password result in an authentication failure.)

When installing using VGA mode and without the grub parameters, this error didn't occur on the same hardware and without having adjusted the partitions on the SSD.

It'd be great to get an updated install ISO, or at least a workaround that I could use to pre-prepare a USB stick in case I have to drop back to using the console port.

[Viktor Tadijanovic (vtadijanovic)]

I am having a same issue as reported by Jake and Jim. Trying to install on Lanner hardware using USB stick with "ubuntu-18.04.2-live-server-amd64.iso" image. I used serial console output by enabling console=ttyS0,115200n8 in Grub config. The installer fails once install attempts to format a hard drive. I get the following error:

 Full installer output ───────────────────────────┐
│Traceback (most recent call last): │
│ File "/snap/subiquity/664/usr/lib/python3.6/runpy.py", line 193, in │
│_run_module_as_main │
│ "__main__", mod_spec) │
│ File "/snap/subiquity/664/usr/lib/python3.6/runpy.py", line 85, in _run_code│
│ exec(code, run_globals) │
│ File "/snap/subiquity/664/lib/python3.6/site-packages/curtin/__main__.py", │
│line 4, in <module> │
│ sys.exit(main()) │
│ File │
│"/snap/subiquity/664/lib/python3.6/site-packages/curtin/commands/main.py", │
│line 177, in main │
│ raise OSError("Unable to find helpers or 'curtin' exe to add to path") │
│OSError: Unable to find helpers or 'curtin' exe to add to path │
│

[Michael Hudson-Doyle (mwhudson)]

I'm pretty sure this was fixed for disco and will be fixed for the next bionic point release.

[Steve Langasek (vorlon)]

Michael, the livecd-rootfs task for bionic is still open, which implies there's still work to be done to fix this for 18.04.3.

[Michael Hudson-Doyle (mwhudson)]

Ah but it was backported in https://launchpad.net/ubuntu/+source/livecd-rootfs/2.525.23 back in May. Bad bug discipline it seems.

[Dimitri John Ledkov (xnox)]

However, in practice this is like fix-committed. As it will be fix released on 1st of August or later, when the 18.04.3 media is shipped.

[Steve Langasek (vorlon)]

No, the purpose of this bug task is to tell us whether work needs to be done on this package. Leaving this task open wasted quite a lot of time.