Error when starting copy to disk due to processes accessing /var/log

[Alfonso Sanchez-Beato]

Installation fails when subiquity starts copying to disk due to some process accessing /var/log. The full crash log is

https://paste.ubuntu.com/p/Jw4ndyQFzk/

The error happens only if re-installing on top of an already partitioned disk. I am using version 21.04.2+git1.2719c57c.

[Michael Hudson-Doyle (mwhudson)]

Oh I think what has happened here is that the sd card you are installing to has a partition with a filesystem with label 'writable' (maybe it has ubuntu core on it, or maybe it was used to hold the installer ISO when installing another system).

This causes casper to create directories on this file system and then bind-mount as /var/log and /var/crash. Then when curtin tries to delete this partition, the kernel understandably gets upset.

I'm not sure what we should do about this :(

[Michael Hudson-Doyle (mwhudson)]

Fortunately there is an easy workaround: format the SD card you want to install to beforehand.

[Ryan Harper (raharper)]

Interesting.

This looks like a gap in clear-holders; we've not encountered automounted devices before. I think having clear-holders check for mounts of devices would be reasonable.

[Michael Hudson-Doyle (mwhudson)]

I think all curtin can do here is fail more gracefully, if anything -- there's not going to be a sensible way to unmount a partition that has /var/log on it.

Maybe casper should only automount a writable filesystem from the same media as the livefs was found? That probably breaks some other use cases, but might be less surprising all in all.

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Note that this is not an SD card, but a SmartNIC with an emmc that is not easily accessible. The server images we were flashing previously do have a label 'writable' for the rootfs, as they were constructed in a bit of a special way. This implies that we can easily find this problem in the field if somebody tries to user the installer on a SmartNIC that had one of these images already installed.

To workaround this problem, what I did was to add break=premount in the kernel command line and dd with zeros the emmc from there, which unfortunately is not that straightforward...

[Ryan Harper (raharper)]

> I think all curtin can do here is fail more gracefully, if anything --
> there's not going to be a sensible way to unmount a partition that has
> /var/log on it.

Agreed. Curtin could certainly attempt to umount it and if that succeeded
we could proceed otherwise we'd at least fail early showing that we
couldn't
umount the device that was part of the config.

On Mon, May 3, 2021 at 5:55 PM Michael Hudson-Doyle <
<email address hidden>> wrote:

> I think all curtin can do here is fail more gracefully, if anything --
> there's not going to be a sensible way to unmount a partition that has
> /var/log on it.
>
> Maybe casper should only automount a writable filesystem from the same
> media as the livefs was found? That probably breaks some other use
> cases, but might be less surprising all in all.
>
> --
> You received this bug notification because you are subscribed to
> subiquity.
> Matching subscriptions: subiquity-bugs
> https://bugs.launchpad.net/bugs/1926137
>
> Title:
> Error when starting copy to disk due to processes accessing /var/log
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/subiquity/+bug/1926137/+subscriptions
>

[Michael Hudson-Doyle (mwhudson)]

I think we should change casper to only (automatically) use a fs with label writable if it's on the same device as the livefs. That may break some use cases but it's definitely the common one.

[Launchpad Janitor (janitor)]

This bug was fixed in the package casper - 1.465

---------------
casper (1.465) impish; urgency=medium

  * Fix typos in last upload that prevented auto log functionality from
    working at all.

 -- Michael Hudson-Doyle <email address hidden> Thu, 22 Jul 2021 12:25:06 +1200

[Michael Hudson-Doyle (mwhudson)]

This should be fixed in the latest impish dailies -- Alfonso, do you still have a way to test this?

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

On Fri, Jul 23, 2021 at 2:35 AM Michael Hudson-Doyle <
<email address hidden>> wrote:

> This should be fixed in the latest impish dailies -- Alfonso, do you
> still have a way to test this?
>

Yes, I can reproduce it. I can try the fix if it is available in some
subiquity snap channel.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1926137
>
> Title:
> Error when starting copy to disk due to processes accessing /var/log
>
> Status in subiquity:
> New
> Status in casper package in Ubuntu:
> Fix Released
>
> Bug description:
> Installation fails when subiquity starts copying to disk due to some
> process accessing /var/log. The full crash log is
>
> https://paste.ubuntu.com/p/Jw4ndyQFzk/
>
> The error happens only if re-installing on top of an already
> partitioned disk. I am using version 21.04.2+git1.2719c57c.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/subiquity/+bug/1926137/+subscriptions
>
>

[Michael Hudson-Doyle (mwhudson)]

The fix is in how the iso is constructed unfortunately, so not trivial to
test on focal. If you can link me an iso you've been testing, I should be
able to hack the new behavior into it (early next week at this point)

On Fri, 23 Jul 2021, 18:25 Alfonso Sanchez-Beato, <
<email address hidden>> wrote:

> On Fri, Jul 23, 2021 at 2:35 AM Michael Hudson-Doyle <
> <email address hidden>> wrote:
>
> > This should be fixed in the latest impish dailies -- Alfonso, do you
> > still have a way to test this?
> >
>
> Yes, I can reproduce it. I can try the fix if it is available in some
> subiquity snap channel.
>
>
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1926137
> >
> > Title:
> > Error when starting copy to disk due to processes accessing /var/log
> >
> > Status in subiquity:
> > New
> > Status in casper package in Ubuntu:
> > Fix Released
> >
> > Bug description:
> > Installation fails when subiquity starts copying to disk due to some
> > process accessing /var/log. The full crash log is
> >
> > https://paste.ubuntu.com/p/Jw4ndyQFzk/
> >
> > The error happens only if re-installing on top of an already
> > partitioned disk. I am using version 21.04.2+git1.2719c57c.
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/subiquity/+bug/1926137/+subscriptions
> >
> >
>
> --
> You received this bug notification because you are subscribed to
> subiquity.
> https://bugs.launchpad.net/bugs/1926137
>
> Title:
> Error when starting copy to disk due to processes accessing /var/log
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/subiquity/+bug/1926137/+subscriptions
>
>

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

I have tried the fix, but unfortunately it failed. See attached logs.