Different partition layout after recovery with keep home partition

[Franz Hsieh]

When using ubuntu-recovery to recover the system,
The partition layout is different when keeping home partition.

consider disk-recipe:

 100 150 200 free
 $iflabel{ gpt }
 $reusemethod{ }
 method{ efi }
 format{ } .

128 512 256 ext2
 $defaultignore{ }
 method{ format }
 format{ }
 use_filesystem{ }
 filesystem{ ext2 }
 mountpoint{ /boot } .

32000 10000 32000 $default_filesystem
 $lvmok{ }
 method{ format }
 format{ }
 use_filesystem{ }
 reserved_for_root{ 1 }
 $default_filesystem{ }
 mountpoint{ / } .

8000 512 8000 linux-swap
 $lvmok{ }
 method{ swap }
 format{ } .

500 10000 -1 $default_filesystem
 $lvmok{ }
 $reusemethod{ }
 method{ format }
 format{ }
 use_filesystem{ }
 $default_filesystem{ }
 label{ User_Data }
 mountpoint{ /home } .

In this case, we want to build this layout in a 320G hard disk and keep the home partition.
If we recover system WITHOUT keeping home partition, the partition layout looks like:

Model: ATA ST320LT012-9WS14 (scsi)
Disk /dev/sda: 320072933376B
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags
        17408B 20479B 3072B Free Space
 1 20480B 99999743B 99979264B fat32 EFI System Partition boot
 2 99999744B 4100001791B 4000002048B fat32 Recovery Partition
 3 4100001792B 36100001791B 32000000000B ext4
 4 36100001792B 44100001791B 8000000000B linux-swap(v1)
 5 44100001792B 320072916479B 275972914688B ext4 Ubuntu Home Partition

But if we recovery system WITH keeping home partition, the partition layout looks like:

Model: ATA ST320LT012-9WS14 (scsi)
Disk /dev/sda: 320072933376B
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags
        17408B 20479B 3072B Free Space
 1 20480B 99999743B 99979264B fat32 EFI System Partition boot
 2 99999744B 4100001791B 4000002048B fat32 Recovery Partition
 4 4100001792B 42073001983B 37973000192B ext4
 6 42073001984B 44100001791B 2026999808B linux-swap(v1)
 5 44100001792B 320072916479B 275972914688B ext4 Ubuntu Home Partition

The behavior is abnormal.

According to my test, partman thinks the necessary space is 100+32000+8000 while the home partition is kept. And I discovered that partman knows EFI partition has $reusemethod option and the partition exists but it still adds the partition size to total. So the total need (40100m) is not equal to total free (40000M) and then partman use alternative recipe instead. This is the root cause of why partition layout is different.

------------------------------------------------------------------------------------------------------
recipe.sh line: 187~ 209

foreach_partition () {
 local - doing IFS pcount last partition
 doing=$1
 pcount=$(echo "$scheme" | wc -l)
 last=no

 IFS="$NL"
 for partition in $scheme; do
  restore_ifs
  [ $pcount -gt 1 ] || last=yes
  set -- $partition
  eval "$doing"
  pcount=$(($pcount - 1))
 done
}

min_size () {
 local size
 size=0
 foreach_partition '
  size=$(($size + $1))'
 echo $size
}

These codes are used for calculate total recipe size, and I can't find any expression deals with $reusemethod.

[Brian Murray (brian-murray)]

Which release of Ubuntu were you testing this (and is important to fix) on?

[Franz Hsieh (franz-hsieh)]

@Brian

The release of Ubuntu is 12.04.2 and the version of ubiquity is 2.10.24

[Dimitri John Ledkov (xnox)]

What is ubuntu-recovery and how can I run it? Quickly searching ubuntu saucy archive, didn't find it.
I think the patch below, might help. But it is untested.

[Franz Hsieh (franz-hsieh)]

@Dmitrijs

I tested your patch and it cause another problem.
In the stage 2 (create target partition and install software), ubiquity prompted an error dialog of "No root file system is defined".
The ubiquity debug log and partman log are attached.

[Franz Hsieh (franz-hsieh)]

partman log

[Dimitri John Ledkov (xnox)]

For such bug it would be interesting to see the generated preseed file / expert recipe. Does ubuntu-recovery store those for debugging somewhere? I'll try running ubuntu-recovery / reproduce above "no root file system" error.

[Franz Hsieh (franz-hsieh)]

attach disk-recipe and all preseed config files.

[Colin Watson (cjwatson)]

Could you please try this patch instead? min_size is a bit too far down the stack to be editing $scheme - doing it in choose_recipe should work better.

Incidentally, $reusemethod is never going to be reliable for a partition containing an ordinary filesystem that doesn't have its own distinct method. It only works properly for specialised partitions such as method{ efi } and method{ swap }. For method{ format }, it's certainly possible that it will work sometimes, but I can't promise that its behaviour is defined. (Of course, ubuntu-recovery may be doing something magic to avoid that problem.) To make this reliable, I think you'd need something like a $reuselabel feature to select the partition to reuse based on the filesystem label, or maybe some other criterion; but that's probably a separate bug.

[Franz Hsieh (franz-hsieh)]

@Colin

I hace verified the patch , and the partition layout now looks the same.

[Launchpad Janitor (janitor)]

This bug was fixed in the package partman-auto - 107ubuntu3

---------------
partman-auto (107ubuntu3) saucy; urgency=low

  * Filter out reused partitions when calculating the size used by a recipe
    (LP: #1197766).
 -- Colin Watson <email address hidden> Mon, 30 Sep 2013 15:48:32 +0100

[Stéphane Graber (stgraber)]

Hello Franz, or anyone else affected,

Accepted partman-auto into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/partman-auto/101ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Franz Hsieh (franz-hsieh)]

Hi Stéphane Graber,

I saw your words but I found the partman-auto in ubiquity-2.10.26 is still 101ubuntu2, not 101ubuntu2.2. Has 101ubuntu2 integrated to ubiquity?

[Dimitri John Ledkov (xnox)]

On 29 November 2013 03:35, Franz Hsieh <email address hidden> wrote:
> Hi Stéphane Graber,
>
> I saw your words but I found the partman-auto in ubiquity-2.10.26 is
> still 101ubuntu2, not 101ubuntu2.2. Has 101ubuntu2 integrated to
> ubiquity?
>

Indeed, ubiquity embeds some partman packages. I am opening a task to
rebuild ubiquity.

Regards,

Dmitrijs.

[Colin Watson (cjwatson)]

Hello Franz, or anyone else affected,

Accepted ubiquity into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/ubiquity/2.10.28 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Franz Hsieh (franz-hsieh)]

Hi Colin,

I have verified ubiquity 2.10.28 on my platform and the issue cannot be reproduced any more.

[Colin Watson (cjwatson)]

Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package partman-auto - 101ubuntu2.2

---------------
partman-auto (101ubuntu2.2) precise; urgency=low

  * Filter out reused partitions when calculating the size used by a recipe
    (LP: #1197766).

partman-auto (101ubuntu2.1) precise; urgency=low

  * Bump EFI partition size range to 512-1024MB, in line with Debian; 100MB
    is too small on a disk with 4KiB logical sectors (LP: #1065281).
 -- Colin Watson <email address hidden> Mon, 30 Sep 2013 15:50:44 +0100

[Colin Watson (cjwatson)]

The verification of the Stable Release Update for partman-auto has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubiquity - 2.10.28

---------------
ubiquity (2.10.28) precise; urgency=low

  * Automatic update of included source packages: partman-
    basicfilesystems 71ubuntu3.4. (LP: #978032)

ubiquity (2.10.27) precise; urgency=low

  * Automatic update of included source packages: netcfg 1.68ubuntu14.1
    (LP: #901700), partman-auto 101ubuntu2.2 (LP: #1197766, #1065281),
    partman-base 153ubuntu6 (LP: #1065281), partman-basicfilesystems
    71ubuntu3.3 (LP: #978032, #1215458), partman-btrfs 8ubuntu1.1 (LP:
    #978032), partman-efi 25ubuntu1.2 (LP: #1065281), partman-ext3
    67ubuntu1.1 (LP: #978032).
 -- Dmitrijs Ledkovs <email address hidden> Tue, 03 Dec 2013 18:04:05 +0000