curtin fails to find itself in subiquity snap, when running on serial-subiquity@.service

Bug #1820604 reported by Dimitri John Ledkov on 2019-03-18
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
curtin
Undecided
Unassigned
subiquity
Critical
Unassigned
livecd-rootfs (Ubuntu)
Undecided
Unassigned
Bionic
Undecided
Unassigned

Bug Description

serial-subiquity@.service runs very differently to the tty1 subiquity.

In seiral-subiquity@ case, it doesn't block starting on snapd as it doesn't use /usr/bin/snap run.

Also, it doesn't execute subiquity.subiquity-service script, meaning that PATH setting there is not exported, causing troubles for curtin to reexec itself, or to find the right python in $PATH.

Fix that in both livecd-rootfs & subiquity.

https://github.com/CanonicalLtd/subiquity/pull/428
https://code.launchpad.net/~xnox/livecd-rootfs/+git/livecd-rootfs/+merge/364888

Related branches

Dan Watkins (daniel-thewatkins) wrote :

It's not clear to me why this is a bug; users shouldn't be driving the curtin in subiquity's snap themselves (right?), and the only evidence here is that that usecase breaks. Could you expand on what user-facing issue this is causing?

subiquity doesn't expose the curtin executable as an "app", so adding /snap/bin to PATH won't help. (Looking at the code, it will try using `which` before it tries constructing paths itself.)

The curtin binary being installed in /bin seems like a reasonable thing to support.

Changed in curtin:
status: New → Confirmed
status: Confirmed → Incomplete
Dan Watkins (daniel-thewatkins) wrote :

(Marked as Incomplete until we have some more info on what's motivating this bug report.)

Dimitri John Ledkov (xnox) wrote :

Subiquity calls curtin, which then fails to find curtin_exe, which then fails the installation. With all subiquity disco images currently failing to install.

It is correct that curtin is not exposed to end users. The code snippets I show is the envrionment that curtin is executed in, and fails to find itself, when executed by subiquity installpath.py module.

Changed in curtin:
status: Incomplete → Confirmed
Changed in subiquity:
importance: Undecided → Critical
status: New → Confirmed
Dimitri John Ledkov (xnox) wrote :

Ditto with 727 subiquity snap.

Ryan Harper (raharper) wrote :

Curtin is designed to be called by it's shell wrapper (bin/curtin) in the code.

I would suggest getting that into PATH and invoking 'curtin'

Ryan Harper (raharper) wrote :

What's new in subiquity that we now see this?

Changed in curtin:
status: Confirmed → Incomplete
Michael Hudson-Doyle (mwhudson) wrote :

Oh, it might have helped if I saw this bug before commenting on the MP. It explains the "what's new so that this is broken" stuff, which is nothing: this way of invoking curtin has never worked.

The reason this isn't a problem in the install environment is this line:

export PATH=$SNAP/bin:$SNAP/usr/bin:$PATH

in bin/subiquity-service in subiquity which I added as part of my "use curtin from the snap" changes. Making it easier to invoke curtin from the shell like this would be nice.

Dimitri John Ledkov (xnox) wrote :

Ok, but with even the respun images, i still see the same failure on ppc64le. Let me attach a few logs.

Dimitri John Ledkov (xnox) wrote :
Dimitri John Ledkov (xnox) wrote :
Dimitri John Ledkov (xnox) wrote :
Download full text (4.9 KiB)

Looks like there are ordering issues wrt. the default console to use hvc0 vs tty0, As I see lots of errors like this one:
Mar 19 14:44:32 ubuntu-server agetty[1923]: hvc0: can't exec /snap/bin/subiquity: No such file or directory

I see squashfs errors:
$ journalctl --file ./system.journal | grep SQUASHFS
Mar 19 14:44:14 ubuntu-server kernel: SQUASHFS error: squashfs_read_data failed to read block 0x8180e1e300333a37
Mar 19 14:44:14 ubuntu-server kernel: SQUASHFS error: Unable to read metadata cache entry [8180e1e300333a37]
Mar 19 14:44:16 ubuntu-server kernel: SQUASHFS error: squashfs_read_data failed to read block 0x8180e1e38a089b3a
Mar 19 14:44:16 ubuntu-server kernel: SQUASHFS error: Unable to read metadata cache entry [8180e1e38a089b3a]
Mar 19 14:45:03 ubuntu-server kernel: SQUASHFS error: Unable to read metadata cache entry [8180e1e38a089b3a]

At least in journal, curtin does fail:
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: Traceback (most recent call last):
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: File "/snap/subiquity/666/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: "__main__", mod_spec)
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: File "/snap/subiquity/666/usr/lib/python3.6/runpy.py", line 85, in _run_code
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: exec(code, run_globals)
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: File "/snap/subiquity/666/lib/python3.6/site-packages/curtin/__main__.py", line 4, in <module>
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: sys.exit(main())
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: File "/snap/subiquity/666/lib/python3.6/site-packages/curtin/commands/main.py", line 177, in main
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: raise OSError("Unable to find helpers or 'curtin' exe to add to path")
Mar 19 14:46:38 ubuntu-server curtin_log.3058[3496]: OSError: Unable to find helpers or 'curtin' exe to add to path

Funily enough the above is in the journal, and not in the subiquity log.

Subiquity installer log has:
2019-03-19 14:46:58,841 subiquity.controllers.snaplist:103 loading list of snaps failed
Traceback (most recent call last):
  File "/snap/subiquity/666/lib/python3.6/site-packages/subiquity/controllers/snaplist.py", line 101, in _fetched_list
    response.raise_for_status()
  File "/snap/subiquity/666/usr/lib/python3/dist-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+unix://%2Frun%2Fsnapd.socket/v2/find?section=server

There is also this nice piece of traceback

2019-03-19 14:47:00,750 subiquitycore.core:430 Exception in controller.run():
Traceback (most recent call last):
  File "/snap/subiquity/666/usr/lib/python3/dist-packages/urwid/main_loop.py", line 384, in _run
    self.event_loop.run()
  File "/snap/subiquity/666/usr/lib/python3/dist-packages/urwid/main_loop.py", line 788, in run
    self._loop()
  File "/snap/subiquity/666/usr/lib/python3/dist-packages/urwid/main_loop.py", line 825, in _loo...

Read more...

Dimitri John Ledkov (xnox) wrote :

Maybe easier to read the curtin failure with this pastebinit:

https://paste.ubuntu.com/p/XDjhfHQxTK/

Dimitri John Ledkov (xnox) wrote :

Squashfs errors: https://paste.ubuntu.com/p/FhKBw2Hwp8/

Agetty failing: https://paste.ubuntu.com/p/HvGHqwcjK3/

snapstore fail?!: https://paste.ubuntu.com/p/TGMnc5v8Zg/

possibly i may need to file extra bugs about this.

Dimitri John Ledkov (xnox) wrote :

Ok! Progress!

If i give the VM network, and provide the right proxy, and restart snapd to use the proxy.... install works!

Dimitri John Ledkov (xnox) wrote :

New fail:

│ + echo grub-install /dev/sda │
│ grub-install /dev/sda │
│ + grub-install /dev/sda │
│ Installing for powerpc-ieee1275 platform. │
│ grub-install: error: the chosen partition is not a PReP │
│partition. │
│ + exit │
│ failed to install grub! │

Ryan Harper (raharper) wrote :

For PReP, you need:

https://git.launchpad.net/curtin/tree/examples/tests/basic.yaml#n85

  - id: pnum_disk_p2
        type: partition
        number: 2
        size: 8MB
        device: pnum_disk
        flag: prep
        wipe: zero
        name: prep

Changed in curtin:
status: Incomplete → Invalid
description: updated
summary: - curtin fails to find itself in subiquity snap
+ curtin fails to find itself in subiquity snap, when running on serial-
+ subiquity@.service
Changed in livecd-rootfs (Ubuntu):
status: New → Confirmed

This is fixed now I think? I guess the livecd-rootfs change needs to go back to bionic?

Changed in livecd-rootfs (Ubuntu):
status: Confirmed → Fix Released
Changed in subiquity:
status: Confirmed → Fix Released
Changed in livecd-rootfs (Ubuntu Bionic):
status: New → Triaged
Jim K (jimk2048) wrote :

I received the error below, on a physical device. The device is an APU2D4 with a 120 GB mSATA SSD and has no video, so I was using the serial port as a putty terminal.

vendor site on the device: https://www.pcengines.ch/apu2d4.htm

I was following this article to use the serial port during install:

http://www.sundby.com/index.php/install-ubuntu-16-04-with-usb-stick-and-serial-console/

I was using the ubuntu-18.04.2-live-server-amd64.iso image loaded on the USB with RUFUS.

An error occurred during installation

┌──────────────────────────── Full installer output ───────────────────────────┐
│Traceback (most recent call last): │
│ File "/snap/subiquity/664/usr/lib/python3.6/runpy.py", line 193, in │
│_run_module_as_main │
│ "__main__", mod_spec) │
│ File "/snap/subiquity/664/usr/lib/python3.6/runpy.py", line 85, in _run_code│
│ exec(code, run_globals) │
│ File "/snap/subiquity/664/lib/python3.6/site-packages/curtin/__main__.py", │
│line 4, in <module> │
│ sys.exit(main()) │
│ File │
│"/snap/subiquity/664/lib/python3.6/site-packages/curtin/commands/main.py", │
│line 177, in main │
│ raise OSError("Unable to find helpers or 'curtin' exe to add to path") │
│OSError: Unable to find helpers or 'curtin' exe to add to path │
│ │
└──────────────────────────────────────────────────────────────────────────────┘
                                 [ Close ]

Jake Billo (ev98) wrote :

I also receive the same "Unable to find helpers or 'curtin' exe to add to path" error as Jim K reported in https://bugs.launchpad.net/curtin/+bug/1820604/comments/18, after defining a partition layout in the installer. I am installing on a Lanner FW-8758 using a Rufus-generated USB stick and have tried both "ubuntu-18.04.2-live-server-amd64.iso" and "ubuntu-18.04.2-server-amd64.iso".

I activated the serial console output using similar instructions to https://github.com/ynkjm/ubuntu-serial-install - mainly adding "console=tty0 console=ttyS0,115200n8" where necessary.

Unfortunately dropping to shell after the failure occurs won't let me log in to investigate further, or at least I don't know the appropriate credentials (root and ubuntu users with a blank password, or the same username/password result in an authentication failure.)

When installing using VGA mode and without the grub parameters, this error didn't occur on the same hardware and without having adjusted the partitions on the SSD.

It'd be great to get an updated install ISO, or at least a workaround that I could use to pre-prepare a USB stick in case I have to drop back to using the console port.

tags: added: id-5d1adb08f3b51440c727e268

I am having a same issue as reported by Jake and Jim. Trying to install on Lanner hardware using USB stick with "ubuntu-18.04.2-live-server-amd64.iso" image. I used serial console output by enabling console=ttyS0,115200n8 in Grub config. The installer fails once install attempts to format a hard drive. I get the following error:

 Full installer output ───────────────────────────┐
│Traceback (most recent call last): │
│ File "/snap/subiquity/664/usr/lib/python3.6/runpy.py", line 193, in │
│_run_module_as_main │
│ "__main__", mod_spec) │
│ File "/snap/subiquity/664/usr/lib/python3.6/runpy.py", line 85, in _run_code│
│ exec(code, run_globals) │
│ File "/snap/subiquity/664/lib/python3.6/site-packages/curtin/__main__.py", │
│line 4, in <module> │
│ sys.exit(main()) │
│ File │
│"/snap/subiquity/664/lib/python3.6/site-packages/curtin/commands/main.py", │
│line 177, in main │
│ raise OSError("Unable to find helpers or 'curtin' exe to add to path") │
│OSError: Unable to find helpers or 'curtin' exe to add to path │

I'm pretty sure this was fixed for disco and will be fixed for the next bionic point release.

Steve Langasek (vorlon) wrote :

Michael, the livecd-rootfs task for bionic is still open, which implies there's still work to be done to fix this for 18.04.3.

Ah but it was backported in https://launchpad.net/ubuntu/+source/livecd-rootfs/2.525.23 back in May. Bad bug discipline it seems.

Changed in livecd-rootfs (Ubuntu Bionic):
status: Triaged → Fix Released
Dimitri John Ledkov (xnox) wrote :

However, in practice this is like fix-committed. As it will be fix released on 1st of August or later, when the 18.04.3 media is shipped.

Changed in livecd-rootfs (Ubuntu Bionic):
status: Fix Released → Fix Committed
milestone: none → ubuntu-18.04.3
Steve Langasek (vorlon) wrote :

No, the purpose of this bug task is to tell us whether work needs to be done on this package. Leaving this task open wasted quite a lot of time.

Changed in livecd-rootfs (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers