Impish live session takes ages to boot on BIOS systems

[José Marinho]

First of all, I change the description of this bug because, thanks to Chris Guiver comments, I could check that the live session effectively works but it takes too long to complete. That's why I change the description of the bug from live session does not boot to live session takes ages to boot. I hope this is the best approach to this.

I think the problem is the same as described here: https://discourse.ubuntubudgie.org/t/20-10-grub-error-can-t-find-command-grub-platform/4292. I can see prior to grub menu, briefly, the same error: Error can't find grub_platform. After the solution described below, this error is not showed and the system is able to boot.

I try making the live usb using startup disk creator and with gnome-disks --> Restore disk image and get the same results.

The live-usb has a gpt partition table instead of mbr like 20.04 live-usb has. That implies, I think, that the first one does not boot on BIOS systems and the second does.

I try the same live-usb on an EFI laptop and it boots perfectly (perhaps it takes long time, but more less than in this case.

If I try the solution described here: https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1905491/comments/8 then it works.

ProblemType: Bug
DistroRelease: Ubuntu 21.04
Package: casper 1.461
ProcVersionSignature: Ubuntu 5.11.0-13.14-generic 5.11.7
Uname: Linux 5.11.0-13-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu61
Architecture: amd64
CasperMD5CheckResult: pass
CasperVersion: 1.461
CurrentDesktop: ubuntu:GNOME
Date: Fri Apr 2 09:55:24 2021
LiveMediaBuild: Ubuntu 21.04 "Hirsute Hippo" - Beta amd64 (20210331.1)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=gl_ES.UTF-8
 SHELL=/bin/bash
SourcePackage: casper
UpgradeStatus: No upgrade log present (probably fresh install)

[Chris Guiver (guiverc)]

Thank you for taking the time to report this problem, and help making Ubuntu better.

QUESTION: What OS & release are you writing the thumb-drive with? (ie. details of the GNOME-DISKS and StartUp Disk Creator, as I wonder if it's how you write your ISO to media.

I am writing various ISOs (mainly Lubuntu, but other flavors too and on occasion Ubuntu) to thumb-drive and booting it on various BIOS only devices (hp dc7700, dc7900, dell 745, 755, 780 etc).

I personally use `mkusb` to write my media, but regularly also just use `dd`.., but I'm usually writing media on a hirsute box (hence the request for details on what OS/release you use to write your ISO)

I note your `lshw` shows an "i5-3570" CPU which is more modern than any BIOS only box I use (my boxes are c2d (t6570, u9400, e6600, e6850, e8300..), c2q-q9400 etc) so maybe it's a quirk on your hardware, compared to the older hp, dell, lenovo.. I test with.

[José Marinho (jmarinho)]

Hi.

The software versions I have installed are:

cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

apt policy gnome-disk-utility
gnome-disk-utility:
  Instalado: 3.36.1-1ubuntu1
  Candidato: 3.36.1-1ubuntu1
  Táboa de versións:
 *** 3.36.1-1ubuntu1 500
        500 http://es.archive.ubuntu.com/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status

 apt policy usb-creator-gtk
usb-creator-gtk:
  Instalado: 0.3.7
  Candidato: 0.3.7
  Táboa de versións:
 *** 0.3.7 500
        500 http://es.archive.ubuntu.com/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status

Furthermore, it can be related to the tool I use for write the ISO to usb, or perhaps like you say, to a quirk on my hardware. But in many years I was using ubuntu and others linux distributions I never or exceptionally had this kind of issues.
The fact is that some changes was introduced in the way that ISO images are created because if I use an Ubuntu 20.04 ISO and write it to a usb drive it defaults to a mbr partition table and now it defaults (using the same tool and doing it in the same manner) to a gpt partition table and that causes the issue on my hardware.
I just want to let you know. It is on your decision if this is right for you and the distribution or not. You (the whole Ubuntu staff not only you personally) are the developer and know better than me if the pros of doing in that way worth it or not against the contras (It have issues on some hardware). I only put this information at your disposal.
I, at least, have a workaround and can make use of it.
Any other information that could be useful for debugging this issue, I send without any problem.

[Chris Guiver (guiverc)]

To José Marinho (jmarinho)

I'm not a developer, but involved in QA-testing primarily with Lubuntu.

Today's hirsute (Lubuntu) daily was written to media using

`sudo dd bs=4M oflag=sync status=progress of=/dev/sdb if=/de2900/lubuntu_64/hirsute-desktop-amd64.iso`

and it was test booted on a number of BIOS devices (see http://iso.qa.ubuntu.com/qatracker/milestones/419/builds/228949/testcases/1303/results)

The last device (thinkpad sl510; c2d-t6570) was rebooted & using the installed bionic system, the Ubuntu Desktop hirsute daily was written to a thumb-drive using usb-creator-gtk (0.3.2ubuntu16.04.2)

The thinkpad sl510 was then rebooted & the thumb-drive that was just written was successfully booted on sl510 (http://iso.qa.ubuntu.com/qatracker/milestones/419/builds/228934/testcases/1303/results).

I cannot re-create this issue on
- lenovo thinkpad sl510 (c2d-t6570, 2gb ram, i915)
- motion computing j3400 (c2d-u9400, 4gb, intel mobile 4 series)
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
- dell [optiplex] 755 (c2d-e6850, 5gb, amd/ati radeon rv516/x1300/x1550)
- dell [optiplex] 755 (c2d-e8300, 8gb, amd/ati radeon rv610/radeon hd2400 pro/xt)
- dell [optiplex] 780 (c2q-q9400, 8gb, amd/ati cedar radeon hd 5000/6000/7350/8350)

being BIOS c2d/c2q devices (all tested today)

[José Marinho (jmarinho)]

To Chris

I have to admit that you are right and the system finally boots but it take ages (it takes less time if I change partition table to mbr).

I see that in the last testcase you point that the device is extremely slow and perhaps that is the bug.

I don't know the specs of the device you tested, but my pc is pretty decent and it is not normal that slowness.

And like I said, that wasn't the case on previous Ubuntu versions and that wasn't the case too on other distributions (Fedora, Opensuse, Debian, Devuan, etc)

What is clear is that this bug must be closed and I don't know if it is better to file a new bug or that behavior isn't consider as that

[José Marinho (jmarinho)]

Finally, I change the description of the bug for better match the nature of the problem.

[José Marinho (jmarinho)]

I was doing some new text and I'm going to put some information about what I have done.

First I create another media with startup disc creator. I think that the tool is not relevant here and I test with dd, gnome-disk and this tool and in all I get the same results.

Once created I booted it, but I edit the grub for removing "quiet splash" parameters. After I hit F10 and continue the boot sequence the computer gets stuck with the following message in the screen:

"Booting a command list"

This last for almost 10 minutes and then begin the normal boot sequence.

I'm going to append two more files to help to debug the issue: the output of dmesg and the output of /var/log/syslog.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1922342

[Chris Guiver (guiverc)]

I just rebooted and timed (phone stopwatch) the boot of
- motion computing j3400 (c2d-u9400, 4gb, intel mobile 4 series)

using Ubuntu 21.04 desktop daily (see comment #4; http://iso.qa.ubuntu.com/qatracker/milestones/419/builds/228934/testcases/1303/results)

which is the device that @jmarinho noted I referred to as booting "extremely slow"

I started the stopwatch when a GRUB message appeared on screen

** the first sign of plymouth (ubuntu logo) appeared @ 8mins 45secs

** the wallpaper first appeared @ 9 mins 58 secs

** the "Try Ubuntu/Install Ubuntu" question appears at 10 mins 18 secs

I haven't installed hirsute on that box, so I don't know if it's an issue only with the live/casper system, or related to the 5.11 kernel (I've not used that box all that frequently this hirsute cycle, but I do use it on occasion and don't recall it being that slow a ~month ago or at least 5.8 days)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ice-Tea (ice-tea)]

If you look at the freshly written image with Gnome disks , 1st ubuntu partition , click the cog wheel and choose edit partition you'll notice that the build team have not enabled the Legacy BIOS bootable option?

If you enable it and click change the drive will now boot , there is still a really long 20 second pause with a black screen and blinking cursor but it eventually swaps over to the Ubuntu boot logo and starts.

[José Marinho (jmarinho)]

Well, I just test it with the final ISO downloaded from ubuntu.com and the problem persists, but with your workaround it boots more quickly.
Perhaps it's a minor issue but I think that it should be avoided because it gives a bad image to whoever tries Ubuntu for the first time. On a short term support release it's not as important as on a LTS is. Since Ubuntu is often considered a friendly option to test a GNU/Linux-based system, these kinds of details should be avoided as much as possible IMHO.

[José Marinho (jmarinho)]

As you like.
I don't know if it's necessary to open a new bug for that, but no problem.
I already marked that the bug you opened as it affects me.
It seems that you have to fill the bug against a package. I don't know which. I fill that bug against casper, but I think that is not correct.

Regards

[Ice-Tea (ice-tea)]

@José Marinho

I've removed the other bug report as i see that you've been trying to get this resolved for ages with no luck so i'm guessing they just don't care so it was pointless starting a new one.

I've given up to be honest as there is some really odd stuff going on with regards to the live image with legacy computers as after i installed hirsute the USB stick was no longer bootable so the Gnome disks work around does not resolve it , i had to go back to Gnome disks and uncheck the legacy bios option and then re-enable it to get the USB stick to boot again?

The latest versions of Ubuntu Live writes install log files to the USB stick so something it's doing is corrupting the stick on legacy computers but no problem on EFI boards.

It's annoying to have to use a different distro on legacy computers if they are no longer going to support them?

[Chris Guiver (guiverc)]

@lucap

I dispute that 'they' don't care.

Canonical employees are paid to work only a set number of hours a week, and have responsibilities/tasks they need to complete in their hours, so have very limited time to fix extra issues.

This bug was reported once during the hirsute cycle in QA-testing (by http://launchpad.net/~jmarinho).

I would also likely have reported it too, only at the time the bug report was NOT that the system took ages to boot, but instead that "HIrsute live session does not work on BIOS systems" so my QA tests did not fit the then description of this report (it booted).

Two boxes (of the six) I used were specifically chosen as they were troublesome in the groovy cycle as they were slow. The result of lack of reports (in QA-testing) was this was not a high priority bug.

A number of issues with BIOS devices were discovered during recent cycles in QA-testing (esp. groovy if I recall correctly), and I saw a lot of energy spent on resolving those issues (ISOs were spun for me to download & boot on the two (plus one other) boxes I described as troublesome earlier just so more could be learnt). Late in the cycle though there is limited time, thus bugs with low rating being ignored is to be expected.

The time to get issues detected is usually earlier in the cycle (alpha stage), and via the QA testing site (so testing is tracked, appears in weekly reports regularly/repeatedly etc).

80% of my QA-testing is on BIOS only boxes, but I tend to QA-test lighter flavors as GNOME isn't much 'fun' on c2d & like machines

[José Marinho (jmarinho)]

Well, what they do with this bug report is beyond what you and me can do so it's their decision. I think that the official iso of Ubuntu (at least if we talk about a Long Term Support Release) should't have this problem.
Perhaps as they are preparing a new installer that is intended to be used on next LTS, they don't pay attention to this.
But that's simply my guess. They will do what they consider better for their purposes. You and me too.
I have several options. Firstly, stay on Ubuntu 20.04 and wait if that is solved on 22.04. Secondly, we have at least 2 workarounds to avoid the issue and finally, we always could choose another distro on Debian/Ubuntu family or on a different taste.

Regards

[Ice-Tea (ice-tea)]

@Chris Guiver

>>>I dispute that 'they' don't care.
>>>thus bugs with low rating being ignored is to be expected.

Bad choice of words , i should have used i guess it's not considered important.

[Ice-Tea (ice-tea)]

How far into the ‘Impish Indri’ Development should we start reporting that the live image doesn't boot on legacy computers and do we report it here or start a new bug?

[Ice-Tea (ice-tea)]

https://cdimage.ubuntu.com/daily-live/current/

@José Marinho

Don't know if you have tried the latest 21.10 development build but it's still exactly the same as before as you still have to enable legacy bios in gnome disks to get it to boot and there's still a long freeze before it starts.

[José Marinho (jmarinho)]

Hi @Lucap

I don't kown either at what time is best for reporting that issue and if it is better continuing here or creating a new bug report.
Perhaps the last option is better because the bug will be reported against next release and this one is reported against Hirsute Hippo and it's no longer under development.
I heard that Ubuntu devs are working on a new installer (https://www.omgubuntu.co.uk/2021/02/ubuntu-is-working-on-a-brand-new-installer) so perhaps is more appropiate to wait some time and see if the new installer is included on next Ubuntu release. If this is the case, try it and, if the bug persists, fill a bug against it.
I did not try any daily-live yet, but I suppose that the problem persists as you said.
So, for what was said above, if no one opposes on next days I'll close this bug report because I think it is useless now and later, if in next release the bug persists, someone could create a new one

[Chris Guiver (guiverc)]

I'll provide my 2c.

Hirsute (21.04) ISO is already released & development completed. Only security fixes are now being performed on the now stable 21.04 system, which does not include this issue (as this issue relates to the ISO itself, and re-spins are only done for LTS releases, ie. next ISO will likely be 20.04.3 or a respin of focal, then impish/21.10 though that's likely not a complete list)

If this issue matters, I'd test using the impish (21.10) ISO and report on the QA testing site, even filing a new bug (referencing back to this one yes, but start a new bug). Even if the report is marked duplicate of this, it'll increase heat & gain more detail/attention. This issue was reported only once on the QA-tracker for hirsute (http://iso.qa.ubuntu.com/qatracker/reports/bugs/1922342) which indicates it wasn't a very important issue. Yes I may have included myself too, had the bug report been about speed, but it was "not booting" at the time, where my QA-test booted so I didn't link to this.

As for the new installer; the boot process occurs before the installer is started. Yes there is a `maybe-ubiquity` kernel parameter but the system is booted prior to ubiquity running.. so I'd suggest not waiting for the new installer, but test with `impish` now giving developers the most time possible to address if they're able.

If you look at testing done so far in 'impish' you'll note well more than 100 QA-tests have been performed (http://iso.qa.ubuntu.com/qatracker/reports/testers) so it's not too early.

[Ice-Tea (ice-tea)]

Well i don't hold much confidence if it's not considered important...

I wouldn't be the best person to start a new bug as i don't know what package this is supposed to be put against , how best to articulate it or how best to provide any extra information needed as i'm just a point and click user.

[Chris Guiver (guiverc)]

This issue still exists in impish.

Refer https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1928939

Current Lubuntu Focal daily vs Impish daily
plymouth: Focal: ~46 sec Impish: 7 mins 20 secs
scan media start: Focal: ~50 sec
scan media pass: Focal: 3 min 08 sec
black screen & mouse-pointer: Focal: 3 min 32 sec Impish: ~8 mins 00 secs
full LXQt running: Focal: 3 min 45 sec Impish: 8 mins 33 secs

If the boot needs to be this slow - some messages on screen are needed !

The much faster FOCAL boot shows the scan start & progress for more than a couple of minutes meaning details are shown on screen, yet on IMPISH it's just black screen, and is taking far longer.

[José Marinho (jmarinho)]

I have confirmed that the bug still exists in Impish and mark your bug report as it affects me.
@Chris Guiver, as you are more involved in that project and know better than me the quirks of this processes, tell me if I must close this bug report and if it is convenient to do something else on your bug report apart from mark the bug as it affects me.
You suggests to the developers that, if the boot should be slow it could be convenient to insert a message warning that. I think that the fix suggested by @Lucap (to enable the boot flag on the first partition) does not have any side effect and reduce the boot time ostensibly.
Besides that, your filled your bug report with reference to Lubuntu iso. I tested Lubuntu and regular Ubuntu and is the same thing on both. For that reason I submit the Ubuntu test result. If I do something wrong feel free to correct me. The tests results are here: http://iso.qa.ubuntu.com/qatracker/milestones/424/builds/230926/testcases/1303/results

[Ice-Tea (ice-tea)]

https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1922342/comments/12

Yeah, if you do as post 12 it cuts the black screen boot time of Hirsute and Impish down to about 20 seconds , not tried it with Lubuntu...

[Michael Hudson-Doyle (mwhudson)]

If this is it do with the partition table / flags then it needs to be fixed in debian-cd. You can see the options passed to xorriso here: https://bazaar.launchpad.net/~ubuntu-cdimage/debian-cd/ubuntu/view/head:/tools/boot/impish/boot-amd64

As you can see, it's a bit complicated. The name of the --mbr-force-bootable sounds like it should be adding a partition with a bootable flag...

[Ice-Tea (ice-tea)]

Yeah it is complicated , when you look at a freshly written USB stick with gnome disks there is a box you can tick that says Legacy BIOS bootable , I'm not sure if gnome disks is doing as described below with the "bios_grub" flag as the link talks about a hardrive so i'm not sure if it would apply to a USB stick?

https://help.ubuntu.com/community/DiskSpace

>BIOS-Boot or EFI partition (required on GPT disks)

>The BIOS-boot partition contains GRUB 2's core. It is necessary if you install Ubuntu on a GPT >disk, and if the firmware (BIOS) is set up in Legacy (not EFI) mode. It must be located at the >start of a GPT disk, and have a "bios_grub" flag.

[Michael Hudson-Doyle (mwhudson)]

On Fri, 21 May 2021 at 02:00, Lucap <email address hidden> wrote:

> Yeah it is complicated , when you look at a freshly written USB stick
> with gnome disks there is a box you can tick that says Legacy BIOS
> bootable , I'm not sure if gnome disks is doing as described below with
> the "bios_grub" flag as the link talks about a hardrive so i'm not sure
> if it would apply to a USB stick?
>

That is perhaps setting the bootable flag on the single partition entry in
the "protective" MBR, which is AIUI technically a violation of the GPT
spec. But if it helps here maybe we should hack it anyway. I'm going to
subscribe Thomas Schmitt, xorriso upstream, who knows enormously more than
I do about all this stuff.

[Ice-Tea (ice-tea)]

Thanks :)

[Thomas Schmitt (scdbackup)]

Hi,

Michael Hudson-Doyle wrote:
> I'm going to
> subscribe Thomas Schmitt, xorriso upstream, who knows enormously more than
> I do about all this stuff.

Well, firmware can always outsmart any preparation in an ISO.

My personal expertise is restricted to ISO 9660 and ends when firmware found
the bootloader in the ISO image.
When the machine can complain about "grub" then the firmware obviously found
such an entry point and ran some GRUB equipment.

Naive guess:
Can the problem here be related to the switch from ISOLINUX to GRUB for
legacy BIOS booting ?
(Ubuntu 19.04 has ISOLINUX, 20.10 has GRUB in the MBR.)

The fact that it works better after conversion from GPT to MBR partition
table could be explained that GRUB wants some more of its components when
having been booted from a medium with valid GPT. I write "wants", because
it finally is able to boot without it, possibly after a long timeout.

The complained "grub_platform" is documented to be a variable which is
promised to be set by GRUB "normal" mode.
  https://www.gnu.org/software/grub/manual/grub/html_node/grub_005fplatform.html
So i wonder whether the BIOS MBR of GRUB and its subsequent boot stages
do not set grub_platform because they are not in "normal" mode ?

Maybe grub-devel mailing list can give hints about MBR boot code, normal
mode, and the grub_platform variable.

--------------------------------------------------------------------------

Whatever, here is a summary of recent efforts to make all x86 firmwares
recognize one of the offered entry points:

The youngest attempt to get it right for all existing machines was
  https://bbs.archlinux.org/viewtopic.php?id=264096
where in the end a slightly improved variation of the Fedora layout by
Matthew J. Garrett seems to have done the trick. The layout by mjg was
used by Ubuntu up to october 2020 and is still used by Debian and Fedora.

Last year Ubuntu for a short time moved to a neat MBR partition table layout,
but then had to switch to GPT, because some EFIs did not recognize the
MBR-only ISO:
  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1886148
But that neat GPT did not work for some old HP machines. So a small deviation
from the GPT specs was added in form of an extra MBR partition of type 0x00
which carries the "bootable" flag:
  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1899308

The archlinux thread above started about an ISO with that layout. But it
was discovered that Lenovo Thinkpad T420 did not work with it in legacy
BIOS mode (CSM). It did well with EFI.
The solution was to go back to mjg layout with the improvement that the
EFI partition ins not inside the ISO 9660 partition any more (i.e. it is
not a data file in the ISO 9660 filesystem).

The result is still not as neat as Ubuntu's current partition layout.
I suspect that nearly 10 years of mjg layout in Fedora, Ubuntu, and Debian
spoiled the firmware programmers who made sure that this weird jackalope
is recognized by their EFI.
To my theory the mistake sneaked in that some firmwares decide that there
is no EFI equipment if there is no GPT (valid or not) whereas some decide
there is no BIOS equipment if there is a valid GPT.
Additionally there is the old...

[Chris Guiver (guiverc)]

I'm 'floored' (WOW) by Thomas Schmitt (scdbackup)'s comment 32.

The long delay still exists in impish (on some devices only).

Ubuntu impish - https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1929029
Lubuntu impish - https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1928939

Delay occurs regardless of ISO being written using mkusb, dd, or gnome-disks..

It would be nice if something could be shown on screen, the fact that the OP of this bug report originally wrote that the issue wasn't that it took ages, but failed to boot - highlights some don't wait long enough...

I wait 30-60mins in QA-testing & just comment the time in the iso.qa.ubu report (when I consider excessive), which is longer than I suspect end-users would expect/accept..

[Thomas Schmitt (scdbackup)]

Hi,

Chris Guiver wrote:
> Delay occurs regardless of ISO being written using mkusb, dd, or gnome-
> disks..

Hmm. IIRC mkusb offers to unpack the ISO into a filesystem of the USB stick.
If the problem survives that conversion, then it is far inside the GRUB
equipment.

It should be possible to make experimental changes with the .cfg files of
such an USB stick.
(I can propose xorriso runs which repack an ISO to bear new .cfg content
and the same boot entry points as the original ISO. But directly editing
in a FAT filesystem would be much more straightforward.)

Being curious i looked into ubuntu-20.10-desktop-amd64.iso for occurences
of grub_platform and found in /boot/grub/grub.cfg

    grub_platform
    if [ "$grub_platform" = "efi" ]; then
    menuentry 'Boot from next volume' {
            exit
    }

It could be that the complaint of GRUB is about the command grub_platform,
not about the use of $grub_platform.

I fail to find the implementation of the command in a freshly pulled GRUB git.
(Other known commands have a call to grub_register_command() with their
name as argument. grub_platform has not.)

A first experiment would be to remove above snippet from grub.cfg and
to check whether its presence caused the long delay.

--------------------------------------------------------------------------
Red herring warning:

The documentation about "normal" mode and grub_platform might be not helpful
for understanding our special problem. Possibly its text describes not the
work of the command but only the situation when it is _not_ needed.

Maybe GRUB is not running in "normal" mode, or maybe grub_platform updates the
variable, which is set at initialization time in grub-core/normal/main.c
from macro GRUB_PLATFORM:

    GRUB_MOD_INIT(normal)
    {
      ...
      grub_env_set ("grub_platform", GRUB_PLATFORM);
      ...

That macro seems to be set by the compile time configuration of GRUB in
./configure which is supposed to stem from configure.ac

    GRUB_PLATFORM="${platform}"
    ...
    AC_SUBST(GRUB_PLATFORM)

The variable platform is set in the same file, either from option
--with-platform or from automatic detection

    # Guess the platform if not specified.
    if test "x$with_platform" = x; then
      case "$target_cpu"-"$target_vendor" in
        ...
        x86_64-apple) platform=efi ;;
        x86_64-*) platform=pc ;;
        ...

So i'd assume that if /boot/grub/grub.cfg expects the possibility to see
"efi", then GRUB would have to be compiled with --with-platform=efi .
Meanwhile the GRUB in Ubuntu ISOs serves both, EFI and legacy BIOS.
I wonder how this is supposed to play with the compile time setting.

There is a compile time variable GRUB_PLATFORMS:
  https://wiki.gentoo.org/wiki/GRUB2_Quick_Start#Installing_GRUB2_software
  "To install GRUB2, first set the GRUB_PLATFORMS variable with one or more
   appropriate values in the system's make.conf. If unset, GRUB2 will guess
   which platform to use on the system. It guesses pc (which is the MBR
   style of installation) for x86/amd64 architectures."
But the relation of the single value GRUB_PLATFORM and the value list in
GRUB_PLATFORMS is not clear to me.

The GRUB manual talks ...

[José Marinho (jmarinho)]

Well, I'm only came here again firstly to say thanks to Michael and Thomas for taking into account this problem. Specially, knowing that this bug probably is quite exceptional and affects at only few cases. Today I was trying on some old computers on my job and it booted fairly well on all of this so, I had bad luck with this issue.
Secondly, I don't know what is the best approach for fixing this problem. If it would be easy to insert a message when the boot process gets stuck or what. I think it is normal that, if the USB does not reach the desktop environment after more than 5 or 6 minutes and if you listen the cpu fan working at full speed, it is normal to think that the usb it's not going to boot.
I was timing how long it takes to reach plymouth and the full desktop environment with a Lubuntu daily build iso and this was the results (the start point was grub in all cases):
With the iso without any change:
- Plymouth --> 7 minutes
- Desktop environment --> 7 minutes 50 seconds
Marking the first partition as bootable legacy BIOS:
- Plymouth --> 54 seconds.
- Desktop environment --> 1 minute 45 seconds
Without marking any partition as bootable but changing the partition table from GPT to MBR:
- Plymouth --> 25 seconds.
- Desktop environment --> 1 minute 20 seconds.
With this results is clear that the last is the best so, a naive question for my part is: would it be too inconvenient to release 2 isos: a main iso for most computers and a legacy iso with an MBR partition table for use if somebody has problems with the main iso?
And finaly, other possible solution might be to include on the release notes or somewhere more suitable that, on some systems, this problem could be present and the possible workarounds.
And nothing more on my part. It's time for the people on charge of this decide what is better. If something more is needed from me let me know.
Thanks and have a nice day.

[Thomas Schmitt (scdbackup)]

Hi,

José Marinho wrote:
> Marking the first partition as bootable legacy BIOS:
> - Plymouth --> 54 seconds.
> - Desktop environment --> 1 minute 45 seconds

Without converting to GPT ?
That would be really strange and surely no solution for the gerneral public.

(It is not allowed by GPT specs to set the "bootable" flag on the
"protective MBR partition" of type 0xEE, which serves as indication that
GPT is valid and also shall protect the data range of the GPT partitions.
Tests with grub-mkrescue ISOs on EFI systems failed. This led to the
introduction of the second partition of type 0x00 with boot flag.)

> Without marking any partition as bootable but changing the partition table
> from GPT to MBR:
> - Plymouth --> 25 seconds.
> - Desktop environment --> 1 minute 20 seconds.

So the BIOS or GRUB on that BIOS get deceived by a boot flag at the second
(dummy) partition, but the BIOS does not demand the boot flag on MBR
partition table.

> would it be too inconvenient to release 2 isos: a main iso
> for most computers and a legacy iso with an MBR partition table for use if
> somebody has problems with the main iso?

Debian does something similar for its "netinst" ISOs.
The BIOS-only ISO with no GPT and no EFI partition is deceivingly named
"mac", because the machines which need it are old x86 Apple machines.
Those machines are said to take offense from the presence of the EFI
partition or of the invald GPT in mjg's partition layout.

Normal BIOS+EFI ISO:
  https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.9.0-amd64-netinst.iso
The BIOS-only "mac" ISO is at:
  https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-mac-10.9.0-amd64-netinst.iso
Each has about 330 MB.
The version number may change in a few weeks or days. Look by a web browser at
  https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/
for
  debian-*.*.*-amd64-netinst.iso
  debian-mac-*.*.*-amd64-netinst.iso

Have a nice day :)

Thomas

[Ice-Tea (ice-tea)]

Amazing how we can all get different results doing the same thing. :)

For me a freshly written ubuntu ISO to a USB regardless if i use Ubuntu USB creator or DD refuses to boot my old legacy gear that is the last generation phoenix bios just before EFI was released.

It shows an error just before the grub screen starts "cannot find grub_platform" then it shows the grub menu and then boots to a black screen with a blinking cursor and just sits there forever and never boots even if i leave it for 15 or 20mins before i give up.

If i look at the USB sticks first partition with gnome disks , edit partition and tick the box that says Legacy BIOS bootable then the USB stick no longer shows "cannot find grub_platform" and goes straight to the grub menu , then it boots to a black screen with a blinking cursor for about 20 seconds and then it boots as normal just like any previous Ubuntu release?

[Thomas Schmitt (scdbackup)]

Hi,

Lucap wrote:
> If i look at the USB sticks first partition with gnome disks , edit
> partition and tick the box that says Legacy BIOS bootable then the USB
> stick no longer shows "cannot find grub_platform" and goes straight to
> the grub menu , then it boots to a black screen with a blinking cursor
> for about 20 seconds and then it boots as normal just like any previous
> Ubuntu release?

Is this a question or a report of an observed fact ?

If it is an observation, then i would be strongly interested in seeing the
results of two runs of

  STICK=/dev/sdc
  xorriso -indev stdio:"$STICK" -report_system_area plain

with STICK holding the device address which you used with dd.

Each run will emit about 50 lines of text.
You may catch them in a /tmp file by

  xorriso -indev stdio:"$STICK" -report_system_area plain 2>&1 | \
    tee -i -a /tmp/report_system_area.txt

Please make one run after freshly writing the unmodified ISO to the stick
before you use gnome disks. Afterwards make a second run to show the result
of gnome disk's activities.

Have a nice day :)

Thomas

[Thomas Schmitt (scdbackup)]

Hi,

just in case somebody wants to make experiments with changed grub.cfg,
here are instructions how to create a modified ISO from an Ubuntu amd64
original (but not from a "powerpc" ISO):

Set paths to the original ISO image and for the emerging ISO image:

  ORIG=ubuntu-21.04-desktop-amd64.iso
  NEW=ubuntu-21.04-test.iso

Extract grub.cfg from the original ISO to the current working directory:

  xorriso -indev "$ORIG" -osirrox on -extract /boot/grub/grub.cfg ./grub.cfg

Make the extracted file writable:

  chmod u+w ./grub.cfg

Now edit the file.
My proposal is to remove the 5 lines about grub_platform.

Make sure that a file with path "$NEW" does not yet exist.
Then create the new modified ISO:

  xorriso -indev "$ORIG" \
          -outdev "$NEW" \
          -map ./grub.cfg /boot/grub/grub.cfg \
          -boot_image any replay

This run of xorriso will issue some righteous warnings
  xorriso : WARNING : -volid text problematic as automatic mount point name
  xorriso : WARNING : -volid text does not comply to ISO 9660 / ECMA 119 rules
because of the blanks and lower case characters in
  Volume id : 'Ubuntu 21.04 amd64'
It will analyze "$ORIG" for boot equipment and boot entry points in order to
create the same in "$NEW"
  xorriso : NOTE : Replayed 22 boot related commands
  xorriso : NOTE : Copying to System Area: 32768 bytes from file '--interval:imported_iso:0s-15s:zero_mbrpt,zero_gpt:ubuntu-21.04-desktop-amd64.iso'

Then it begins to write to "$NEW" an ISO image of roughly the same size as
the size of "$ORIG". Enjoy the progress messages. :))

When done, put "$NEW" onto the USB stick and try what happens.
(Maybe mount "$NEW" and check whether your changes really did get into
 effect in /boot/grub/grub.cfg, before the lengthy dd to USB stick.)

Have a nice day :)

Thomas

[José Marinho (jmarinho)]

Hi Thomas,

I made the experiment you proposed modifying grub.cfg file. In this case I use a regular Ubuntu (Main edition with Gnome) Impish iso file.
After Grub I get a blank screen with the cursor blinking and after only 32 seconds I reach Ubuntu logo and after 1 minute and 22 seconds the window where you select language settings and if you want to try Ubuntu first or begin installation process now without trying Ubuntu.
So it seems that you find the cause of the problem.
Hope that helps.

Have a nice weekend.

[Thomas Schmitt (scdbackup)]

Hi,

José Marinho wrote:
> I made the experiment you proposed modifying grub.cfg file.
> [...]
> So it seems that you find the cause of the problem.

By luck and 14 years of xorriso development. :))

Can i talk you into a second test ?
This time only disable the part in grub.cfg which evaluates the variable:

    if [ "$grub_platform" = "efi" ]; then
    menuentry 'Boot from next volume' {
            exit
    }

but leave the command in effect:

    grub_platform

The resulting behavior will be of interest when asking GRUB experts for an
explanation.

-------------------------------------------------------------------------

We still could need to know what wonder exactly gnome disks does for
Lucap.
Somehow the partition table of the ISO influences the ability of
command and/or variable evaluation to do its job in reasonable time.

The proposed xorriso runs will hopefully show what exactly lets
grub_platform or the evaluation of $grub_platform block boot progress.

-------------------------------------------------------------------------

When both open questions are answered, it will be time that Ubuntu ISO
production takes this case to one of the mailing lists about misbehaving
GRUB. Let's have a look into the archives.

Not much replies:
  https://lists.gnu.org/mailman/listinfo/bug-grub

Probably not the right audience:
  https://lists.gnu.org/mailman/listinfo/help-grub

Quite oriented to "[PATCH]"-work. But at least they discuss what is posted:
  https://lists.gnu.org/mailman/listinfo/grub-devel/
I am subscribed to grub-devel, because of grub-mkrescue. So i could serve
as curious bystander who wonders where command grub_platform is
implemented.

Have a nice day :)

Thomas

[Ice-Tea (ice-tea)]

@José Marinho

I don't mean to put this on you but as the Gnome disks options also worked for you i might be better if you run the report_system_area scan between the normal ISO and the Gnome disk edited version so it gets done correctly as i'm getting out of my depth here as a point and click user , for example.

xorriso 1.5.2 : RockRidge filesystem manipulator, libburnia project.

libburn : SORRY : Neither stdio-path nor its directory exist
xorriso : FAILURE : Cannot acquire drive 'stdio:$/dev/sdc'
xorriso : aborting : -abort_on 'FAILURE' encountered 'FAILURE'

My USB stick is /dev/sdc and i also tried sdc1 but it's still the same error.

[Ice-Tea (ice-tea)]

My bad , I've got it working with /dev/sdc instead of $/dev/sdc.

[Ice-Tea (ice-tea)]

@Thomas Schmitt

run1.txt = Standard ISO unmodified
run2.txt = Gnome disks - first partition legacy bios enable.

I'll post the difference between the logs , if you want the full logs posted let me know and i will paste them up.

diff run1.txt run2.txt
21,22c21
< MBR partition : 1 0x00 0xee 1 5508391
< MBR partition : 2 0x80 0x00 0 1
---
> MBR partition : 1 0x00 0xee 1 15649199
26c25
< GPT lba range : 64 5508328 5508391
---
> GPT lba range : 64 15649136 15649199
31c30
< GPT partition flags: 1 0x1000000000000001
---
> GPT partition flags: 1 0x0000000000000005

[Thomas Schmitt (scdbackup)]

Hi,

Lucap wrote:
> < MBR partition : 1 0x00 0xee 1 5508391
> < MBR partition : 2 0x80 0x00 0 1
> ---
> > MBR partition : 1 0x00 0xee 1 15649199

So gnome disks expanded the range of the protective MBR partition to
nearly 8 GB, which i assume is the size of the USB stick.
I do not believe that this change is significant.

It also removed the data in the dummy partition 2 which carries the
bootable flag to please some old HP machines with old BIOS.
This removal of could be a significant change.

> < GPT lba range : 64 5508328 5508391
> ---
> > GPT lba range : 64 15649136 15649199

Accordingly the range of the GPT now offers space for new partitions.

> < GPT partition flags: 1 0x1000000000000001
> ---
> > GPT partition flags: 1 0x0000000000000005

Original flags mean: bit60=read-only, bit0=System partition.
gnome disks flags mean: bit2=Legacy BIOS bootable, bit0=System partition.

Having set bit2 could be a significant change.

-------------------------------------------------------------------------

Let's try to find out which of the two suspects does the trick:

The flag field of partition 1 in ubuntu-21.04-desktop-amd64.iso is at
byte address 1024+48 = 1072 and 8 bytes long.
Please verify that xorriso says about your original ISO

   GPT entry array : 2 248 separated

where the value 2 tells the start block of the partition slot array
meaning byte offset 2 * 512 = 1024.
(The meaning of the reported lines is roughly explained by
   xorriso -report_system_area help
)

Assuming that you see the expected value 2, you can begin byte surgery:

Make a copy of your original ISO:

  ORIG=ubuntu-21.04-desktop-amd64.iso
  NEW=ubuntu-21.04-test.iso
  cp "$ORIG" "$NEW"

Extract the flag bytes from the working USB stick:

  STICK=/dev/sdc
  FLAGSFILE=gnome_disk_flags.img
  dd if="$STICK" bs=1 skip=1072 count=8 of="$FLAGSFILE"

This should yield a "$FLAGSFILE" with 8 bytes.

Implant these 8 bytes into the "$NEW" copy of the original ISO:

  dd if="$FLAGSFILE" conv=notrunc bs=1 seek=1072 count=8 of="$NEW"

Verify by xorriso -report_system_area of "$NEW" that the flags of
partition 1 are now like in "$STICK":

  xorriso -indev stdio:"$NEW" -report_system_area plain | \
    grep 'GPT partition flags: 1'

should say:

  GPT partition flags: 1 0x0000000000000005

Copy "$NEW" onto the stick and check whether it boots in reasonable time.

------------------------------------------------------------------------

If it works nicely despite the fact that "$NEW" has MBR partition 2 with
MBR boot flag, then it might be possible to fix this issue by a change in
xorriso (namely setting the GPT partition flags to a different value).

For that we would need to verify that the read-only flag does not hamper
success.

Copy byte 1072+7 = 1079 from "$ORIG" to byte 7 of "$FLAGSFILE":

  dd if="$ORIG" bs=1 skip=1079 count=1 of="$FLAGSFILE" conv=notrunc seek=7

Implant into "$NEW":

  dd if="$FLAGSFILE" conv=notrunc bs=1 seek=1072 count=8 of="$NEW"

Verify correctness by:

  xorriso -indev stdio:"$NEW" -report_system_area plain | \
    grep 'G...

[Thomas Schmitt (scdbackup)]

Hi,

regrettably the display of launchpad's website eats blanks.
This mutilates my proposed grep 'GPT partition flags: 1' by condensing
three blanks before "1" to a single blank.

So i change the inspection proposal to

  xorriso -indev stdio:"$NEW" -report_system_area plain | \
    grep 'GPT partition flags:'

which should yield something like

  GPT partition flags: 1 0x1000000000000005
  GPT partition flags: 2 0x0000000000000000
  GPT partition flags: 3 0x1000000000000001

Interesting for our problem is only the first of these lines.

Have a nice day :)

Thomas

[Ice-Tea (ice-tea)]

You might need to hold my hand through this as my technical skills are limited.

Before i attempt what you ask i've just built a new ISO with grub_platform removed and it now boots up in the same manner as the Gnome disk option , after the grub menu it gives a black screen for about 20 to 30 seconds and then boots as normal so removing grub_platform works.

I don't know if this is of any significance but the new ISO is written to USB differently as gnome disks no longer says it's a GPT layout and says the first partition is Linux (Bootable)(0x83)

So i'm not sure if it's the removal of grub_platform or the way the new ISO has been written as Linux bootable?

[Ice-Tea (ice-tea)]

I've just built a new ISO with xorriso but this time i never removed grub_platform and left it all untouched and now the USB stick boots without any grub_platform error and after 20 to 30 seconds of a black screen it boots as normal so it's the way the new ISO is being written to the USB stick as Linux (Bootable)(0x83) instead of GPT.

José Marinho can confirm this if you don't mind building another ISO but this time don't remove
grub_platform from grub.cfg just leave it all original and build a new ISO and see if it boots?

Thanks

[Ice-Tea (ice-tea)]

>>Thomas Schmitt , Let's try to find out which of the two suspects does the trick:

Sorry Thomas i tried to read through the instructions but i just simply don't have any type of command or development skills and i find it confusing. :(

I'm just a point and click Ubuntu user...

[José Marinho (jmarinho)]

@ Lucap

I've just test the iso modified keeping grub_platform command and removing the part that evaluates the variable and I get the same results as when I removed both: 32 seconds to togo and 1 minute 22 seconds to language selector.
I wasn't noticed what was changing on partitions layout with all this tweaks but in this last experiment I see with Gnome disks that the usb drive with the modified iso has a MBR partition table instead the GPT partition table of the original ISO. In fact, before all of this experiments on the first post of this report I link a post where give instructions for changing the partition table from GPT to MBR.
According to fdisk the partition table resulting from the modified ISO is:

Orden (m para obtener ayuda): p
Disco /dev/sdd: 3,79 GiB, 4048551936 bytes, 7907328 sectores
Disk model: Transcend 4GB
Unidades: sectores de 1 * 512 = 512 bytes
Tamaño de sector (lógico/físico): 512 bytes / 512 bytes
Tamaño de E/S (mínimo/óptimo): 512 bytes / 512 bytes
Tipo de etiqueta de disco: dos
Identificador del disco: 0x00000000

Dispositivo Inicio Comienzo Final Sectores Tamaño Id Tipo
/dev/sdd1 * 64 5496567 5496504 2,6G 83 Linux
/dev/sdd2 5496568 5506607 10040 4,9M ef EFI (FAT-12/16/32)
/dev/sdd3 5509120 7907327 2398208 1,1G 83 Linux

The last one is a writable partition and was preceding of a empty space of 1.3 MB according with gnome-disks
Furthermore, the first partition is marked as bootable
So I confirm your results.
I'm going to trying the last instructions of Thomas. I don't have too much technical skills either but I'm going to try my luck.

[Ice-Tea (ice-tea)]

Thanks José for confirming that and trying Thomas instructions on GPT.

[Thomas Schmitt (scdbackup)]

Hi,

Lucap wrote:
> gnome disks no longer says it's a GPT layout and
> says the first partition is Linux (Bootable)(0x83)

José Marinho wrote:
> in this last experiment I see with Gnome disks that the usb drive with
> the modified iso has a MBR partition table instead the GPT partition table
> of the original ISO.

Aargh. My fault. I forgot that the original GPT layout of modern Ubuntu
ISOs needs a newer xorriso for automatic boot equipment replay.
I now repeated my proposal in comment #39 with xorriso-1.5.2 and indeed
you are right. The -boot_image replay command did not correctly recognize
the new Ubuntu ISO layout.

So we have to retract the diagnosis that grub_platform in grub.cfg is
to blame, until one of you can repack an ISO to GPT.

In my tests with current upstream release xorriso-1.5.4 it worked fine.
But the packaging for xorriso-1.5.4 is sitting in Debian's salsa git and
waits for the end of Debian 11 release freeze.

There are two ways to work around:

--------------------------------------------------------------------------
1st way:

Build the current GNU xorriso from source. Do not install it, but rather
use it from where it was built. This keeps it from confusing Ubuntu's
package management.

You need the compiler equipment of meta-package "build-essential".

Download the tarball to a new directory:

  cd "$HOME"
  mkdir xorriso_dir
  cd xorriso_dir
  wget https://ftp.gnu.org/gnu/xorriso/xorriso-1.5.4.pl02.tar.gz

(Mistrusting people also download xorriso-1.5.4.pl02.tar.gz.sig and let
gpg --verify that both match.)

Build it according to its README file:

  tar xzf xorriso-1.5.4.pl02.tar.gz
  cd xorriso-1.5.4
  ./configure --prefix=/usr
  make

Expect a storm of messages from ./configure and "make".
Success is when "make" ends and a run of

  ./xorriso/xorriso

can say
  GNU xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.

  usage : xorriso-1.5.4 [commands]
          More is told by command -help

Use this resulting xorriso binary by its absolute path

  "$HOME"/xorriso_dir/xorriso-1.5.4/xorriso/xorriso \
          -indev "$ORIG" \
          -outdev "$NEW" \
          -map ./grub.cfg /boot/grub/grub.cfg \
          -boot_image any replay

You may later safely remove the tree "$HOME"/xorriso_dir to get rid of
the GNU xorriso construction site.

--------------------------------------------------------------------------
2nd way:

xorriso-1.5.2 is well able to repack the ISO to GPT. (After all it did
pack up the original ISO.)
It just needs the correct list of boot related commands, as analyzed by
xorriso-1.5.4 and told by its command -report_system_area "cmd".

The bad news is that this command list is long and that i need to see
the exact "$ORIG" ISO in order to let xorriso-1.5.4 tell the exact
arguments.

With "$ORIG" = ubuntu-21.04-desktop-amd64.iso which has
MD5 736a9acaf195063600a6e1876d48a263 it would be:

  xorriso \
    -indev "$ORIG" \
    -outdev "$NEW" \
    -map ./grub.cfg /boot/grub/grub.cfg \
    \
    -volid 'Ubuntu 21.04 amd64' \
    -volume_date uuid '2021042011161600' \
    -boot_image grub grub2_mbr=--interval:imported_iso:0s-15s:zero_mbrpt,zero_gpt:'ubuntu-21.04-desktop-amd64.iso' \
    -...

[José Marinho (jmarinho)]

Well, I try the first method (compiling xorriso) and removing the if condition from grub.cfg keeping grub_platform command. The usb drive now has a GPT partition table but unfortunately, the issue persists so, it seems that the grub_platform thing is not the cause.
If any other test is needed don't hesitate on ask me, but for today I have enough, so perhaps I'll take a while on doing it.
Have a nice day

[Thomas Schmitt (scdbackup)]

Hi,

José Marinho:
> Well, I try the first method (compiling xorriso) and removing the if
> condition from grub.cfg keeping grub_platform command. The usb drive now has
> a GPT partition table but unfortunately, the issue persists so, it seems
> that the grub_platform thing is not the cause.

Remove the grub_platform command and try again.

Initially we thought that removing all five lines had helped. But that
test has to be repeated, now that we have to expect that this success was
due to the partition table changes.

If the repeated test with no command grub_platform shows that the problem is
gone, then the test result with just no "if"-lines would tell us that indeed
the command is to blame for the long delay.

If removing all five lines does not help, then the error message about
grub_platform is just an unrelated complaint.

-----------------------------------------------------------------------------

> If any other test is needed don't hesitate on ask me, but for today I have
> enough, so perhaps I'll take a while on doing it.

Please try what i proposed to Lucap in comment #45
  https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1922342/comments/45
(beware of swallowed blanks, see #46)

Somehow the quite marginal changes of gnome disk prevent the problem.
The proposed dd runs manipulate the GPT, veterinarian style.

If all we need to do is to set the flags of GPT partition 1 to
  0x1000000000000005
then i could quite directly provide a remedy in GNU xorriso-1.5.5.

Further there would be a veterinary solution for the time being.
The biggest obstacle would be to get byte value 5 by shell means.
In bash there is a special form of quoting:

  echo $'\005' | dd bs=1 count=1 conv=notrunc seek=1072 of="$NEW"

In dash the built-in echo command interprets backslashes on its own:

  echo '\005' | dd bs=1 count=1 conv=notrunc seek=1072 of="$NEW"

Have a nice day :)

Thomas

[José Marinho (jmarinho)]

Hi Thomas,

I try the solution from comment #39 erasing the five lines referring to grub_platform and it did not work.

Now I'm trying to do what you proposed on comment #45 being aware of comment #46 and, like it happened this morning when I first tried it, the output of -report_system_area is different to what you said. At morning, before your post realizing that the Ubuntu version of xorriso can't create an ISO with a GPT partition table, I was trying that and got the same output. I continued burning the iso to the usb drive but the boot delayed again. Then you mentioned the xorriso affair and I stopped the test.

I return to the test now and I have the same output of the report_system_area command. In this case I stopped here because I think that is anything wrong and it is going to be unsuccessful again.

I sumarize what I did (my working directory is $HOME/Impish_tests). I use absolute paths for avoiding possible errors.

1- Set the ORIG variable to $HOME/Impish_tests/ubuntu-21.04-desktop-amd64.iso

ORIG=/home/jose/Impish_tests/ubuntu-21.04-desktop-amd64.iso

2- Set the NEW variable to $HOME/Impish_tests/ubuntu-21.04-test.iso

NEW=/home/jose/Impish_tests/ubuntu-21.04-test.iso

3- Make a copy of the original iso (cp "$ORIG" "$NEW")

4- Set the STICK variable to /dev/sdd. My system assign this letter to the usb drive. I assume that the usb drive shoud have an original ubuntu 21.04 ISO burned.

STICK=/dev/sdd

5- Set the FLAGSFILE variable to the file $HOME/Impish_tests/gnome_disk_flags.img

FLAGSFILE=gnome_disk_flags.img

6- Extract 8 bytes of the working USB stick after the 1072 one and create the file referenced by $FLAGSFILE (this command shoud be issued with sudo).

sudo dd if="$STICK" bs=1 skip=1072 count=8 of="$FLAGSFILE"
8+0 records in
8+0 records out
8 bytes copied, 0,00128856 s, 6,2 kB/s

7- Implant this 8 bytes after the first 1072 of the $NEW ISO file.

dd if="$FLAGSFILE" conv=notrunc bs=1 seek=1072 count=8 of="$NEW"
8+0 records in
8+0 records out
8 bytes copied, 0,000311537 s, 25,7 kB/

8- Verify that these 8 bytes into $NEW are now like in "$STICK". Here is the problem. The output is the following:

~/Impish_tests$ xorriso_dir/xorriso-1.5.4/xorriso/xorriso -indev stdio:"$NEW" -report_system_area plain | grep 'GPT partition flags:'
GNU xorriso 1.5.4.pl02 : RockRidge filesystem manipulator, libburnia project.

xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE : 908 nodes read in 1 seconds
libisofs: NOTE : Found hidden El-Torito image for EFI.
libisofs: NOTE : EFI image start and size: 1373661 * 2048 , 10040 * 512
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -indev 'stdio:/home/jose/Impish_tests/ubuntu-21.04-test.iso'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record : El Torito , MBR protective-msdos-label grub2-mbr cyl-align-off GPT
Media summary: 1 session, 1376337 data blocks, 2688m data, 19.1g free
Volume id : 'Ubuntu 21.04 amd64'
GPT partition flags: 1 0x1000000000000001 --> But you said it must be GPT partition flags: 1 0x0000000000000005
GPT partition flags: 2 0x0000000000000000
GPT...

[José Marinho (jmarinho)]

Oh wait! I'm just to see this:

In bash there is a special form of quoting:

  echo $'\005' | dd bs=1 count=1 conv=notrunc seek=1072 of="$NEW"

And now the output:

.........

Media summary: 1 session, 1376337 data blocks, 2688m data, 19.1g free
Volume id : 'Ubuntu 21.04 amd64'
GPT partition flags: 1 0x1000000000000005
GPT partition flags: 2 0x0000000000000000
GPT partition flags: 3 0x1000000000000001

[José Marinho (jmarinho)]

Bad luck!

It does not work.

I mount the resulting ISO ($NEW) as a loop device and inspect the partition layout with fdisk. Here is the output:

jose@jose-pc:~/Impish_tests$ sudo fdisk /dev/loop34

Bienvenido a fdisk (util-linux 2.34).
Los cambios solo permanecerán en la memoria, hasta que decida escribirlos.
Tenga cuidado antes de utilizar la orden de escritura.

La tabla GPT primaria está dañada, pero la de respaldo parece que está bien, así que esa será la que se utilice.

Orden (m para obtener ayuda): p

Disco /dev/loop34: 2,64 GiB, 2818738176 bytes, 5505348 sectores
Unidades: sectores de 1 * 512 = 512 bytes
Tamaño de sector (lógico/físico): 512 bytes / 512 bytes
Tamaño de E/S (mínimo/óptimo): 512 bytes / 512 bytes
Tipo de etiqueta de disco: gpt
Identificador del disco: AF8737A9-1E23-4373-B87A-C8B16199D461

Dispositivo Comienzo Final Sectores Tamaño Tipo
/dev/loop34p1 64 5494643 5494580 2,6G Datos básicos de Microsoft
/dev/loop34p2 5494644 5504683 10040 4,9M Sistema EFI
/dev/loop34p3 5504684 5505283 600 300K Datos básicos de Microsoft

Orden (m para obtener ayuda):

But the layout is different from the usb stick burned with that ISO file

jose@jose-pc:~/Impish_tests$ sudo fdisk /dev/sdd

Bienvenido a fdisk (util-linux 2.34).
Los cambios solo permanecerán en la memoria, hasta que decida escribirlos.
Tenga cuidado antes de utilizar la orden de escritura.

GPT PMBR size mismatch (5505347 != 7907327) will be corrected by write.

Orden (m para obtener ayuda): p

Disco /dev/sdd: 3,79 GiB, 4048551936 bytes, 7907328 sectores
Disk model: Transcend 4GB
Unidades: sectores de 1 * 512 = 512 bytes
Tamaño de sector (lógico/físico): 512 bytes / 512 bytes
Tamaño de E/S (mínimo/óptimo): 512 bytes / 512 bytes
Tipo de etiqueta de disco: dos
Identificador del disco: 0x00000000

Dispositivo Inicio Comienzo Final Sectores Tamaño Id Tipo
/dev/sdd1 1 7907327 7907327 3,8G ee GPT
/dev/sdd2 * 0 0 1 512B 0 Vacía

Las entradas de la tabla de particiones no están en el orden del disco.

Orden (m para obtener ayuda):

[Thomas Schmitt (scdbackup)]

Hi,

José Marinho wrote:
> 4- [...] I assume that the usb drive shoud have an original ubuntu
> 21.04 ISO burned.

The "$STICK" in the setting of Lucap at comment #44 was additionally
treaded with gnome disks: "edit partition and tick the box that says
Legacy BIOS bootable".
This yielded GPT flags 0x0...05.

> 8- Verify that these 8 bytes into $NEW are now like in "$STICK".
> GPT partition flags: 1 0x1000000000000001 --> But you said it must be
> GPT partition flags: 1 0x0000000000000005

0x10...01 is the original setting of Ubuntu ISOs.

> Did I something wrong or that is OK but this solution will not work?

Your dd transplantation did not change the flags of GPT partition 1,
because the "$STICK" did not get the gnome disks treatment before
the flags bytes were copied out of it.

-------------------------------------------------------------------------

> Oh wait! I'm just to see this:
> echo $'\005' | dd bs=1 count=1 conv=notrunc seek=1072 of="$NEW"
> [yields]
> GPT partition flags: 1 0x1000000000000005

Now you are back on the proposed test track.

> It does not work.
> I mount the resulting ISO ($NEW) as a loop device and inspect the
> partition layout with fdisk.

(fdisk does not need mounting. Actually the device should not be mounted
if you change partitions, in order not to confuse the system drivers.
Whatever if loop34 is connected to "$NEW" then the result should be the
ame as with fdisk examining the plain file "$NEW".)

But fdisk is picky with the GPT checksum, which we spoil by inserting the
new flags value. Mine reports
  The primary GPT table is corrupt, but the backup appears OK,
  so that will be used.
and my very restricted spanish skills tell me that yours raises protest,
too:
> La tabla GPT primaria está dañada, pero la de respaldo parece que está
> bien, así que esa será la que se utilice.

All firmwares which i know do not react on checksum mismatches.
But well, we are in unchartered territory.

So fdisk shows you the unchanged content of the GPT backup at the end of
the ISO image:

> Tipo de etiqueta de disco: gpt
> [...]
> /dev/loop34p1 64 5494643 5494580 2,6G Datos básicos de Microsoft
> /dev/loop34p2 5494644 5504683 10040 4,9M Sistema EFI
> /dev/loop34p3 5504684 5505283 600 300K Datos básicos de Microsoft

This looks like what i expect after echo | dd to the original ISO.

> But the layout is different from the usb stick burned with that ISO file
> jose@jose-pc:~/Impish_tests$ sudo fdisk /dev/sdd
> [...]
> Tipo de etiqueta de disco: dos
> [...]
> /dev/sdd1 1 7907327 7907327 3,8G ee GPT
> /dev/sdd2 * 0 0 1 512B 0 Vacía

But this cannot be the original ISO treated by the echo | dd pipeline.
The size of /dev/sdd1 already reflects the size of the stick, which the
original ISO cannot have known in advance. So some software must have
manipulated the table before or after copying to the stick.

---------------------------------------------------------------------------

Please check again by

  xorriso -indev stdio:"$NEW" -report_system_area plain

that "$NEW" still contains
  ...
  MBR partition table: N Status Type Start Blocks
...

[José Marinho (jmarinho)]

Hi Thomas.

I corrected the steps I did wrong and I'm going to try to explain the best I can for letting you know.

<<Your dd transplantation did not change the flags of GPT partition 1,
because the "$STICK" did not get the gnome disks treatment before
the flags bytes were copied out of it>>

I did it and check with xorriso and at least it showed the desired output:

GPT partition flags: 1 0x0000000000000005

But, here begins the problems. When I dd the modified ISO (test-iso1) into a USB stick the partition layout according with fdisk is again different at the ISO one. I can assure that the method I used for copying the ISO into the stick was dd. The partition layout according with fdisk was:

GPT PMBR size mismatch (5505347 != 7907327) will be corrected by write.

Orden (m para obtener ayuda): p

Disco /dev/sdd: 3,79 GiB, 4048551936 bytes, 7907328 sectores
Disk model: Transcend 4GB
Unidades: sectores de 1 * 512 = 512 bytes
Tamaño de sector (lógico/físico): 512 bytes / 512 bytes
Tamaño de E/S (mínimo/óptimo): 512 bytes / 512 bytes
Tipo de etiqueta de disco: dos
Identificador del disco: 0x00000000

Dispositivo Inicio Comienzo Final Sectores Tamaño Id Tipo
/dev/sdd1 1 7907327 7907327 3,8G ee GPT
/dev/sdd2 * 0 0 1 512B 0 Vacía

But, if I run xorriso -report_system_area I get:

GNU xorriso 1.5.4.pl02 : RockRidge filesystem manipulator, libburnia project.

xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE : 908 nodes read in 1 seconds
libisofs: NOTE : Found hidden El-Torito image for EFI.
libisofs: NOTE : EFI image start and size: 1373661 * 2048 , 10040 * 512
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -indev 'stdio:/dev/sdd'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record : El Torito , MBR protective-msdos-label grub2-mbr cyl-align-off GPT
Media summary: 1 session, 1376337 data blocks, 2688m data, 1173m free
Volume id : 'Ubuntu 21.04 amd64'
GPT partition flags: 1 0x0000000000000005
GPT partition flags: 2 0x0000000000000000
GPT partition flags: 3 0x1000000000000001

Fdisk reports a GPT PMBR size mismatch and another strange fact is this: Tipo de etiqueta de disco: dos (Disk label type: dos). But the first partition is marked as GPT.

I have to say too that the ISO (test-iso1) layout was the following according to fdisk:

jose@jose-pc:~$ sudo fdisk /home/jose/Impish_tests/ubuntu-21.04-test.iso

Bienvenido a fdisk (util-linux 2.34).
Los cambios solo permanecerán en la memoria, hasta que decida escribirlos.
Tenga cuidado antes de utilizar la orden de escritura.

La tabla GPT primaria está dañada, pero la de respaldo parece que está bien, así que esa será la que se utilice.

Orden (m para obtener ayuda): p

Disco /home/jose/Impish_tests/ubuntu-21.04-test.iso: 2,64 GiB, 2818738176 bytes, 5505348 sectores
Unidades: sectores de 1 * 512 = 512 bytes
Tamaño de sector (lógico/físico): 512 bytes / 512 bytes
Tamaño de E/S (mínimo/óptimo): 512 bytes / 512 bytes
Tipo de etiqueta de disco: gpt
Identificador del disco: AF8737A9-1E23-4373-B87A-C8B16199D461

Disposi...

[Thomas Schmitt (scdbackup)]

Hi,

José Marinho wrote:
> GPT PMBR size mismatch (5505347 != 7907327) will be corrected by write.
> ...
> /dev/sdd1 1 7907327 7907327 3,8G ee GPT
> /dev/sdd2 * 0 0 1 512B 0 Vacía

Looks like fdisk reports to you what would be the result if you let it
write what it thinks would be correct.

> another strange fact is this:
> Tipo de etiqueta de disco: dos (Disk label type: dos).

(It would help if you do
   export LANG=C
before running fdisk, in the hope that it will then talk english.
Revoke this later by
   unset LANG
)

Probably fdisk does not accept the partition table as GPT because of the
second MBR partition of type 0x00 which carries the boot flag.

> Disco /home/jose/Impish_tests/ubuntu-21.04-test.iso: 2,64 GiB, 2818738176
> Tipo de etiqueta de disco: gpt

For some reason it believes in GPT as long as the ISO image is not on the
stick. Not very consistent.

We should not give fdisk too much attention here. It has its own ideas
which it does not communicate transparently.

> the first ISO burned on the stick disklabel is dos but the first partition
> is type GPT.

It is of MBR partition type 0xee which the EFI specs call "GPT Protective".
(EFI is where GPT is specified.)
This partition is supposed to cover the whole block range of the storage
device. It is also supposed to be the only MBR partition to indicate the
presence of a valid GPT.

The latter expectation is not fulfilled by the Ubuntu ISO because of the
boot flag and non-zero bytes in MBR partition slot 2.
But the EFI specs also say:
  A [MBR] Partition Record that contains an OSType value of zero or a
  SizeInLBA value of zero may be ignored.
The MBR partition 2 of the Ubuntu ISO has OSType zero (0x00). So EFI
firmwares normally ignore it and accept the MBR as "protective" indicating
the presence of a valid GPT.

--------------------------------------------------------------------------

So the manipulations about setting the "Legacy BIOS bootable" flag in the
GPT EFI partition did not help.

(We can revisit this attempt when xorriso-1.5.5 can set this flag while
producing a GPT with correct checksums. I began to implement this
yesterday but will also have to offer the opportunity to suppress the
"read-only" flag, which gnome disks did not set.)

For now our experiments exhausted the first alternatives of
  https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1922342/comments/45

This leaves us with the last alternative:
  "If the transplated flags field does not fix the problem, then it must be
   about the MBR partition 2 with its boot flag.
   In this case it would not be possible to create an ISO that pleases all.
   So if "$NEW" does not boot swiftly after the first transplantation, try
   whether it helps to overwrite MBR partition slot 2 by zeros"

So try this now:

  cp "$ORIG" "$NEW"
  dd if=/dev/zero bs=1 count=16 of="$NEW" conv=notrunc seek=462

Afterwards

  xorriso -indev "$NEW" -report_system_area plain

should then report only a single MBR partition

  MBR partition table: N Status Type Start Blocks
  MBR partition : 1 0x00 0xee 1 5505347

Now the "protective MBR" is flaw...

[José Marinho (jmarinho)]

Well, that's strange: the first time I booted from the usb stick it booted fine, reaching logo on 25 seconds but the second time and the following it failed again (probably will boot after a long wait but I did not check)

I'm going to try to reproduce this behavior and send you some logs. The most curious thing is that this happened again with other tentative. I think that it was the gnome-disks flag one (the veterinarian way as you named it... XD). The first time worked fine but the following failed. Give some time and tomorrow or the day after I'll send you more details.

[Thomas Schmitt (scdbackup)]

Hi,

i uploaded a new development snapshot of GNU xorriso which can create a
GPT similar to the result of Lucap's gnome disks run.

  cd "$HOME"/xorriso_dir
  wget http://www.gnu.org/software/xorriso/xorriso-1.5.5.tar.gz
  tar xzf xorriso-1.5.5.tar.gz
  cd xorriso-1.5.5
  ./configure --prefix=/usr
  make

Then

  xorriso/xorriso -version

should say

  GNU xorriso 1.5.5 : RockRidge filesystem manipulator, libburnia project.
  ...
  Version timestamp : 2021.05.25.195904
  ...

This program is able to repack "$ORIG" to gnome disks' idea of a GPT:

  "$HOME"/xorriso_dir/xorriso-1.5.5/xorriso/xorriso \
    -indev "$ORIG" \
    -outdev "$NEW" \
    -changes_pending yes \
    -boot_image any replay \
    -boot_image any mbr_force_bootable=off \
    -boot_image any gpt_iso_bootable=on \
    -boot_image any gpt_iso_not_ro=on

Inspection of "$NEW" by xorriso should yield
  ...
  MBR partition table: N Status Type Start Blocks
  MBR partition : 1 0x00 0xee 1 5503775
  GPT : N Info
  ...
  GPT partition flags: 1 0x0000000000000005
  ...

The partition tables of "$NEW" will differ from "$ORIG" by:
- No dummy MBR partition 2, because of mbr_force_bootable=off.
- bit 2 = "Legacy bootable" of the ISO 9660 partition is set, because of
  gpt_iso_bootable=on
- bit 60 = "Read-only" of that partition is not set, because of
  gpt_iso_not_ro=on

(The latter two -boot_image options are new in 1.5.5.)

Have a nice day :)

Thomas

[antoha-mi (antoha-mi)]

Why not create a BIOS boot partition and install grub on it?

[José Marinho (jmarinho)]

Hi Thomas,

As I said yesterday, this changes only work at first boot but in the following boots the behaviour is the same again.

This happened today with the new version of xorriso and yesterday with "dd if=/dev/zero bs=1 count=16 of="$NEW" conv=notrunc seek=462" command. On first boot, after dd the ISO file on the stick it boots fine reaching logo on 25 seconds but after the first boot we have the regression again.

I'm going to to attach two files (boot1.txt and boot2.txt). The first one are the output of two xorriso commands BEFORE trying to boot from the stick, just after dd the iso file on it. The second one (boot2.txt) is the same but AFTER the first successful boot. The changes on the output could explain why boots fine only one time.

If it's necessary I append the same command outputs with the iso created using the dd if=/dev/zero method. These files I append now are from the iso created using the xorriso method just like you said today morning.

[José Marinho (jmarinho)]

It seems I only can attach a file per comment. Here you have the second one file.

[Ice-Tea (ice-tea)]

@ José Marinho

>As I said yesterday, this changes only work at first boot but in the >following boots the behaviour is the same again.

If you look at post #16 i had the issue with Hirsute where the USB stick would no longer boot and i had to go to Gnome Disks and untick legacy bios and then enable it again to get the stick to boot but it only happened once and i've not had that issue again with the Impish ISO but i'm guessing what ever gnome disks is doing is a bit flakey or unorthodox is some way?

@ Thomas Schmitt

https://gitlab.gnome.org/GNOME/gnome-disk-utility

Would the code be worth you looking through for any insight for the legacy bios option?

[Thomas Schmitt (scdbackup)]

Hi,

José Marinho wrote:
> changes only work at first boot [...]
> This happened today with the new version of xorriso and yesterday with
> "dd if=/dev/zero bs=1 count=16 of="$NEW" conv=notrunc seek=462" command.
> [...]
> the output of two xorriso commands BEFORE trying to boot from
> the stick, just after dd the iso file on it.
> https://launchpadlibrarian.net/540588248/boot1.txt

So xorriso is now able to at ISO production time do what gnome disks did
for Lucap to make the stick bootable.

The fact that "dd ... seek=462" did the trick too, indicates that the
problem is associated with the MBR partition 2.

(So probably the command
  -boot_image any mbr_force_bootable=off
after
  -boot_image any replay
would be sufficient, whereas
  -boot_image any gpt_iso_bootable=on
  -boot_image any gpt_iso_not_ro=on
were not really needed with the xorriso-1.5.5 run.
If you are curious, you could check this. Just to be sure.)

> The second one (boot2.txt)
> [https://launchpadlibrarian.net/540588529/boot2.txt]
> is the same but AFTER the first successful boot.

Surprisingly the MBR dummy partition is back again.
(The "GPT partition flags:" of GPT partition are the same as in boot1.txt,
 namely 0x0...5, which indicates that not the "$ORIG" ISO sneaked in with
 flags value its 0x1...1.)

What does happen if you zeroize it again ?

  dd if=/dev/zero bs=1 count=16 of=/dev/sdd conv=notrunc seek=462

Does it boot and then that partition gets installed again ?

Does the stick stay bootable (i.e. without MBR partition 2) if you
end the boot attempt at the first GRUB menu which shows up ?

------------------------------------------------------------------------

I have no idea how the MBR partition slot 2 gets back onto the stick.
There is no backup of it which could be restored by any partition
editor.

Somehow the bootloader or the booted system install that partition.
This partition is not an official GRUB feature, although it was originally
invented as workaround of a boot problem with grub-mkrescue ISOs on
old HP laptops. (One has to add xorrisofs option --mbr-force-bootable
to the grub-mkrescue run in order to get that second partition.)

Ubuntu intentionally brings the second MBR partition into the ISO.
But i wonder which stage of a booting Ubuntu system copies the original
MBR partition table over the table of the stick from where it booted.

I am doubtful that a modern partition editor would be willing to create
such an MBR partition on a device with proper "protective" MBR partition
table and thus with valid GPT.
So when it happens, it seems to be some raw binary operation, similar to
our dd run which removes the partition.

I will try to reproduce this strange behavior, but would still be clueless
how to identify the particular culprit in the booting system.
Any idea is welcome.

------------------------------------------------------------------------

Whatever exactly is happening here, the findings already indicate that
there will be no Ubuntu ISO possible which fits all firmwares.
(Without the problematic MBR partiton 2 there cannot be an MBR boot flag
and without that flag, some old HP firmwares don't boot.)

A possible solution would be a standard...

[Thomas Schmitt (scdbackup)]

Hi,

Lucap wrote:
> https://gitlab.gnome.org/GNOME/gnome-disk-utility
> Would the code be worth you looking through for any insight for the
> legacy bios option?

Seems not necessary. Your diff in comment #44 shows the few changes which
gnome disks made for you.

José Marinho's boot1.txt shows that xorriso-1.5.5 with its new -boot_image
options was able to create the same partition table, except the size of
partition 0xee and the device size field in the GPT header block.
This device size depends on the actual USB stick on which the ISO is copied.
So it is futile to try predicting it when the ISO gets produced.

Have a nice day :)

Thomas

[Ice-Tea (ice-tea)]

@ Thomas Schmitt

Thanks for explaining , it was just a thought as i noticed in the diff text there was some numbers on there own such as the one at the top so i wasn't sure if there was something else going on.

>21,22c21

[Thomas Schmitt (scdbackup)]

Hi,

the Ubuntu system really manipulates the installation stick's partition
table if i choose to try out Ubuntu.

It looks like i was involved as adviser when the addition of MBR
partition 2 was introduced to "casper's persistent partition creation".
See
  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1899308
beginning at comment #60 up to
  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1899308/comments/63
(Sorry for my dim memory. It's just half a year ago.)

Can we assume that Steve Langasek is already subscribed to this bug report ?
Else we should notify him.

I repeat my opinion that under these circumstances no single ISO can serve
all firmwares.

-------------------------------------------------------------------------
Details:

I tested the result of xorriso-1.5.5 repacking gnome-disks-style, i.e.
with only one MBR partition and no MBR boot flag:

  MBR partition table: N Status Type Start Blocks
  MBR partition : 1 0x00 0xee 1 5503775
  GPT : N Info
  ...
  GPT lba range : 64 5503712 5503775
  ...
  GPT partition flags: 1 0x0000000000000005
  GPT start and size : 1 64 5493608

The stick stays that way if i reset the computer when it shows the first
white-on-black GRUB menu. (Stick afterwards inspected by the Debian on
the SSD of that machine.)

If i instead choose "Ubuntu" and the try-out option in the subsequent menu,
i get to the drawing of a stubbly hippo and cannot find any way to start
a shell terminal. (Is it Gnome or is it Ubuntu, which is to blame ?
The user impression for me is abysmal. I was unix-ly socialized in 1985.)
At least i find the logout button to the upper right.

Afterwards, xorriso on Debian reports about the stick

  MBR partition table: N Status Type Start Blocks
  MBR partition : 1 0x00 0xee 1 245759999
  MBR partition : 2 0x80 0x00 0 1
  GPT : N Info
  ...
  GPT lba range : 64 245759936 245759999
  ...
  GPT partition flags: 1 0x0000000000000005
  GPT start and size : 1 64 5493608

The difference affects the MBR partition table, where the protective
partition was expanded to the size of the USB stick (nominally "128 GB").
Further the MBR partition of type 0x00 with boot flag was added.
Also affected is the GPT header block, were the partitionable size was
adapted to the real stick size.
The backup GPT was moved to the end of the stick's block range.

Contrary to my expectation this looks like the work of a partition editor,
which can deal with GPT. On the other hand it - or some other program -
is willing to add MBR partition 2 to a GPT.

At this point i began to remember that Steve Langasek, sudodus, and i
discussed the same effect back last year: The HP machine, which needed
the MBR boot flag, booted the new ISO once because casper's partition
editing then removed MBR partition 2.
Now casper probably does after the partition editor run:

  echo -n '\0200\00\01\00\00\00\01\00\00\00\00\00\01\00\00\00' | \
  dd of="$DEVICE" bs=1 seek=462 conv=notrunc

Have a nice day :)

Thomas

[Thomas Schmitt (scdbackup)]

Hi,

it comes to me that while no single ISO can serve all firmwares, it would
be possible that one casper can serve the original ISO and an ISO without
MBR partition 2 alike.

The trick would be to let casper memorize the 16 bytes 462 to 477 before
it runs the partition editor, and to let it write the memorized bytes
back to offset 462 afterwards.
So if a user did to the ISO image file or to the ISO on stick:

  dd if=/dev/zero bs=1 count=16 of="$ISO_OR_STICK" conv=notrunc seek=462

then this would survive casper's adjustment of the partition table.

Have a nice day :)

Thomas

[José Marinho (jmarinho)]

> What does happen if you zeroize it again ?
> dd if=/dev/zero bs=1 count=16 of=/dev/sdd conv=notrunc seek=462

I can't believe it... it boots fine again but once, twice, three times... and there is no MBR partition anywhere.

And I repeated the steps again, just in case

- Burn on stick the iso that was builded with xorriso-1.5.5.
- Boot it for the first time --> Success.
- Boot it again --> Fails
- sudo dd if=/dev/zero bs=1 count=16 of=/dev/sdX conv=notrunc seek=462
- Boot it again --> Success
- The following boot attempts went fine too and no MBR partition 2 is added.
- All the successful boots finish when I reach full desktop environment after choosing "Try Ubuntu". Then I reboot.

[Thomas Schmitt (scdbackup)]

Hi,

José Marinho wrote:
> I can't believe it... it boots fine again but once, twice, three
> times... and there is no MBR partition anywhere.

Let me make up a theory:

> - Burn on stick the iso that was builded with xorriso-1.5.5.
> - Boot it for the first time --> Success.

During this run of Ubuntu, casper did its "persistent partition creation",
as Steve Langasek puts it in bug 1899308.
This included a run of a GPT partition editor and then patching of MBR
partition slot 2 by the 16 bytes for the dummy partition with boot flag.

> - Boot it again --> Fails

The evil dummy partition does its work.

> sudo dd if=/dev/zero bs=1 count=16 of=/dev/sdX conv=notrunc seek=462

Now it is gone again.

What is not gone is the persistent GPT partition number 3. It covers the
formerly free space on the USB stick. In case of my "128 GB" stick:
  GPT start and size : 3 5505024 240254913

The "$NEW" ISO from xorriso-1.5.5 with mbr_force_bootable=off has only
two GPT partitions. ("$ORIG" has three. Number 3 covers 300 KiB of padding.)

> - Boot it again --> Success

Up to this first success it is no surprise.

To my theory, casper sees no need to add a "persistent partition" and
thus does not run the partition editor and does not write to MBR partition
slot 2.

> - The following boot attempts went fine too and no MBR partition 2 is added.

No new MBR partition 2 = no boot problems.

Have a nice day :)

Thomas

[Leó Kolbeinsson (leok)]

Tested Lubuntu Impish daily ISO 04-06-2021 from QA testing site -

Test machine:
Dell [ Optiplex] MT 7040 ( i7-6700,32 GB, Intel HD Graphics 530,Intel i219-LM GB Ethernet, M.2 256GB SATA SSD)

Very fast boot in BIOS mode - unable to reproduce the error/problem

http://iso.qa.ubuntu.com/qatracker/milestones/424/builds/231678/testcases/1701/results

[Thomas Schmitt (scdbackup)]

Hi,

since the majority of machines seems to work with the current Ubuntu ISOs,
it would be helpful to have manufacturer, model name, and firmware versions
of machines where the ISOs cause long delays or fail.

The problem does not sit in the recognition of boot entry points by the
firmware but occurs later, when GRUB is already running. So probably
some interaction of GRUB and firmware goes wrong, if the MBR partition 2
exists. If this ever shall get debugged, then we need to document which
machines would be able to reproduce the problem.

Have a nice day :)

Thomas

[Chris Guiver (guiverc)]

Lubuntu impish daily boot on (written with mkusb/dus CLONE mode)
- motion computing j3400 (c2d-u9400, 4gb, intel mobile 4 series)
stop watch started at grub/lubuntu
@ 6 mins 34 secs first boot message appeared, only blinking cursor top left till this point
@ 7 mins 04 secs lubuntu's blue plymouth screen appeared
@ 7: mins 56 secs screen black & back to boot messages (but smaller fonts; non-bios fonts)
@ 8 mins 23 secs black screen & Xorg/gui mouse cursor appears
@ 8 mins 54 secs desktop is usable

(~2 mins 53 sec comparison for usable desktop today on `lenovo thinkpad x201 (i5-m520, 4gb, i915)`

ISO was re-written to thumb-drive using mkusb/LIVE mode
times when written that way are slower
(note: most times are ~approx; dog was distracting me)
@ ~7 mins first boot message appeared, only blinking cursor top left till this point
@ ~7 mins 34 secs visual mode changed
@ ~7 mins 49 secs lubuntu's blue plymouth screen appeared
@ ~9 mins 25 secs screen black & back to boot messages (but smaller fonts; non-bios fonts)
@ ~10 mins 05 secs black screen & Xorg mouse cursor appears
@ 10 mins 29 secs desktop is usable

((or 3 mins 30 secs for `hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)` taken from duplicate report so different daily)) lubuntu daily boots faster than Ubuntu (refer duplicates 1929029[ubuntu] or 1928939[lubuntu]) )

I've not followed all of ^, so if my performing some tests would be helpful, please let me know.

I noted Thomas (scdbackup) asked for firmware details; I'm unsure if directed to José Marinho (jmarinho) or general, nor what specific details are required; but `dmidecode` specs for the motion.computer j3400 are attached

[Thomas Schmitt (scdbackup)]

Hi,

> I noted Thomas (scdbackup) asked for firmware details

The motivation is to find enough affected hardware so that the delay
can be reproduced by developers. Obviously it did not happen with the
regular testing machines.

So my request/proposal was for the general public:
Tell which machines are slow at booting the current Ubuntu ISOs.

---------------------------------------------------------------------------

If it becomes reproducible for a developer who understands the boot
sequence well enough to identify the actual step which needs too long,
then a chance for a fix or a workaround could emerge.

My suspicion is that some part of GRUB gets different info from the
firmware, depending on whether the second MBR partition exists or not.
The info in the case of an existing second partition then would cause
this or another part of GRUB to wait for something that won't happen
in reasonable time.
I confess that this is a wide and vague theory. But the insight we have
does not yield more than that.

So we need to bring together a machine with that problem and a developer
with sufficient interest and GRUB knowledge to find out what's happening
and to start discussing the problem with grub-devel mailing list.
I assume that the GRUB developers will not be interested yet, because the
problem is a reaction of a mild violation of GPT specs by Ubuntu's ISO
with its second MBR partition of type 0x00 and boot flag.

---------------------------------------------------------------------------

Another path for a sustainable solution would be if we find enough
affected machines to convince Ubuntu to offer two flavors of ISOs.
One "for picky EFI" with GPT and a fully specs compliant Protective MBR.
(Won't work for old HP laptops.)
One "for picky BIOS" without GPT but rather an MBR partition table.
(Won't work on newer Lenovos.)

We had both versions already when the new Ubuntu ISO layout was developed.
Both could still serve the majority of BIOS and EFI systems. Their success
would differ only on the mentioned machines which show a peculiar
interpretation of EFI specs and BIOS tradition.
"For picky BIOS" was the ISO layout at the beginning of
  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1886148
and "for picky EFI" was the layout at its end.
The attempt to have a single ISO for all yielded the current layout, which
was the result of
  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1899308

Have a nice day :)

Thomas

[José Marinho (jmarinho)]

Hi.
In the next month at least, I will not have much time to dedicate to this so, if something (tests, etc) is required from me, perhaps I may not be able to do it quickly so, please be patient. Even so, I send the output of dmidecode in case it is useful.

Have a nice day

José

[Chris Guiver (guiverc)]

Lubuntu impish daily boot (2021.10.10) on
- motion computing j3400 (c2d-u9400, 4gb, intel mobile 4 series)

start phone-stopwatch at grub menu
~4 mins 40 secs before first boot message appears
~4 mins 47 secs new boot message(s) & switching font (bios generated text? to graphics?)
~4 mins 51 secs plymouth logo appears, blue background
~5 mins 48 secs the plymouth is gone, back to boot type (linux) messages
~5 mins 57 secs mouse pointer appears
~6 mins 27 secs lubuntu background is on screen & LXQt panel etc visible
~6 mins 33 secs system is fully stable & usable

this is ~150 secs faster than last mentioned test (on this device) in comment #76

[Brian Murray (brian-murray)]

I have an HP TouchSmart tm2 (model tm2t-1000) which takes around 10 minutes to boot an Ubuntu 21.10 USB disk created using dd.

dmidecode indicates that the BIOS is revision 18.240 and the Firmware Revision is 71.27.

[Brian Murray (brian-murray)]

If I remove "quiet splash" from the grub command prompt, which I reach quickly on the above system, I see "Booting a command list" and then nothing else until the system is booted.

[Chris Guiver (guiverc)]

I just noted Brian's comment so booted a somewhat recent jammy (2021-12-29) Lubuntu ISO on
- motion computing j3400 (c2d-u9400, 4gb, intel mobile 4 series)

with 'quiet splash' removed I had ~similar response to Brian...

" Booting a command list \n\n" & blinking cursor for a looooong time before eventual boot.

[Brian Murray (brian-murray)]

I modified the grub arguments, using grub shell, so that "set debug=all" was used. Having done that and watching the boot process the last message I saw before a very long pause was:

"lib/relocator.c:1410: chunks = 0xb9fd5c00"

The context before the pause can be found in the attached screenshot.

[Brian Murray (brian-murray)]

Reading the grub manual I found a "pager" option which causes the output to pause after each screenful. By setting that I was able to capture what happens after "lib/relocator.c:1410:".

[Brian Murray (brian-murray)]

There was also a much shorter pause in the boot process at:

"commands/verifiers.c:88: file: /casper/vmlinuz type: 3"

The attached picture shows what happens after that.

[Julian Andres Klode (juliank)]

Are we sure this issue does not happen on 20.10 as well? It seems to spend the same time in 20.10, 21.04, 21.10, and jammy daily when booted in a qemu vm.

I don't know for sure what's going on, I assume it's trying to read the entire partition into memory to verify it, but not 100% sure. Certainly it seems to be trying to verify the initrd, loading it takes longer than the kernel.

[Julian Andres Klode (juliank)]

qemu kvm data, with the iso specified as -hda

- 23s to splash on 20.04 BIOS (isolinux), vs 22s on UEFI
- 30s to splash on 21.04 BIOS, vs 19s on UEFI

I have resized the image using fallocate to be 16G, but that did not increase boot time.

Optimized VM (kvm, virtio HD):

- 3s to splash on 20.04 BIOS, 4s to splash on UEFI
- 9s to splash on 21.04 BIOS, 3s to splash on UEFI

These times are on very fast Samsung NVMe SSD. All times from starting the boot item in the grub menu to splash appearing.

Edit: Hmm, another run in qemu gives me 16s to splash on BIOS, so I don't know.

[Chris Guiver (guiverc)]

I can write a 20.10 ISO to a thumb-drive & re-test it on
- motion computing j3400 (c2d-u9400, 4gb, intel mobile 4 series)

to confirm/deny @juliank's #86 comment on that box. That device is not a common box used for QA by me so it's possible I may not have used it late in the groovy cycle... I won't do it tonight though; it'll have to be tomorrow if helpful.

[Chris Guiver (guiverc)]

Booting Ubuntu Desktop (groovy RC 2020-10-16.4)
- - motion computing j3400 (c2d-u9400, 4gb, intel mobile 4 series)

and plymouth takes ~8 mins to appear on screen
the maybe-ubiquity (try or install) screen took >10 mins

I did mention groovy in comment #17, but I don't believe I filed bugs, just noted on iso.qa.ubuntu.com QA-test reports.

I didn't use stop-watch, but looked at room clock in room (digital without seconds so my times are approx only).. My ISO may also not have been released groovy; just last I zsync'd prior to release for QA.

[Chris Guiver (guiverc)]

Lubuntu 22.04 LTS (jammy) ISO booting on
- motion computing j3400 (c2d-u9400, 4gb, intel mobile 4 series)

00:10 secs enter pressed at grub

(I could reduce 10secs from following times, but have opted not to)

BLINKING CURSOR without change until

7:19 (7 mins 19 secs) first boot message
7:27 lubuntu plymouth is seen
8:04 system messages replace plymouth
8:31 Xorg mouse pointer is seen
9:12 lubuntu wallpaper is drawn
by 9:20 the system is fully functional

[tlk (sarcasticskull)]

Hitting this with Ubuntu MATE 22.04 iso dd'ed onto a USB stick.
Spotted the grub_platform thingy, then had to wait like ~20 min after the GRUB menu for it to start booting.

Run of the mill legacy (I guess) platform:
Gigabyte GA-970A-DS3/GA-970A-DS3, BIOS F7a 01/24/2013
AMD FX-6100
8GB

Kinda frustrating that the fixes/kludges for both "advanced" EFI and legacy crap "proprietary" systems suddenly make booting an Ubuntu LTS live ISO impossible (not having any hint on what's going on).

[Chris Guiver (guiverc)]

I just booted Lubuntu kinetic daily on
- motion computing j3400 (c2d-u9400, 4gb, intel mobile 4 series)

I didn't set up stopwatch to time correctly; but room clock said 13:16 at start of boot, and system was fully functional just as it hit 13:22, which is way faster than comment #90 on same box with jammy ISO.

The ISO wasn't cloned to thumb-drive; but written using

`sudo -H dus-iso2usb kinetic-desktop-amd64.iso /dev/sdb msdos grub-2.0.4 persistent`

guiverc@d960-ubu2:/de2900/lubuntu_64$ apt-cache policy mkusb
mkusb:
  Installed: 22.0.1-1ubuntu2
  Candidate: 22.0.1-1ubuntu2
  Version table:
 *** 22.0.1-1ubuntu2 500
        500 http://ppa.launchpad.net/mkusb/unstable/ubuntu kinetic/main amd64 Packages

This comparison isn't complete; as different ISOs used (jammy vs kinetic) & different writes (clone/dd vs mkusb-altered), but noted here.

[Thomas Schmitt (scdbackup)]

Hi,

Chris Guiver wrote:
> The ISO wasn't cloned to thumb-drive; but written using
> `sudo -H dus-iso2usb kinetic-desktop-amd64.iso /dev/sdb msdos grub-2.0.4 persistent`

Ahum. Congrats to sudodus then. :))

I read in
  https://help.ubuntu.com/community/mkusb#Installation
  "The main function of dus-iso2usb is to
   1. extract the first partitions and the grub structure from compressed
      image files
   2. create new partitions, one for an iso file labeled 'isodevice',
      and one optional for persistence labeled 'writable' (or in older
      systems labeled 'casper-rw').
   3. copy iso file to 'isodevice' and add some tweaks.
   "

So this indicates that GRUB alone is not to blame for the slow booting.
Something in the ISO's partitioning or GRUB configuration is needed for
the problem.

What do you get reported about the USB stick partitions by fdisk -l ?

I wonder what is meant by "add some tweaks".

Have a nice day :)

Thomas

[sudodus (nio-wiklund)]

Hi Thomas Schmitt,

> I wonder what is meant by "add some tweaks".

1. Made grub use UUID instead of device name to identify partition where to find the grub files

---
# create grub.cfg to match iso file's name

echo -e "$inversvid tweak grub ...$resetvid"

# tweak 3: search by UUID

sleep 2
partprobe
sleep 2
targ1=$(mktemp -d --tmpdir dus.XXXXXXXXXX)
uid2=$(lsblk -o uuid,label "$target" |grep -i 'usbboot'|sed "s% *usbboot%%")
uid3=$(lsblk -o uuid,label "$target" |grep 'isodevice'|sed "s% *isodevice%%")
echo "uid2=$uid2"
echo "uid3=$uid3"
mount -U "$uid2" "$targ1"
sleep 2
if [ "$uid3" != "" ]
then
 sed -i '/set isofile/'a" search --set=root --fs-uuid $uid3" "$targ1"/boot/grub/grub.cfg
 sed -i 's/(hd0,3)/($root)/' "$targ1"/boot/grub/grub.cfg
 if [ $? -ne 0 ]
 then
  error="$error - sed: tweak 3 grub.cfg"
  result="target drive might work, but setting 'search by UUID' failed :-("
  echo "$result"
 fi
fi
umount "$targ1"
rmdir "$targ1"
sleep 2
partprobe
sleep 1
}
---
# tweak 1: maybe modify label of partition for persistence

mntpt=$(mktemp -d --tmpdir dus.XXXXXXXXXX)
mount -o loop "$source" "$mntpt"
syslbl=$(lsblk -o label,mountpoint | grep "$mntpt"|sed "s% *$mntpt%%")
umount "$mntpt"
sleep 2
isover=$(<<< "$syslbl" grep -o ' [0-9]*\.'|grep -o '[0-9]*')
pptarg=$(lsblk -lo name,label "$target" |grep writable|sed -e 's/ .*//' -e 's#^#/dev/#')

if [ $isover -lt 20 ]
then
 tune2fs -L casper-rw "$pptarg"
 sleep 1
fi

# tweak 2:
# To make Firefox work in Jammy & newer:
# apparmor.service.d/30_live_mode.conf: ConditionPathExists=

if [ $isover -ge 22 ]
then
 targ1=$(mktemp -d --tmpdir dus.XXXXXXXXXX)
 /bin/echo -e "$inversvid To make Firefox work in Jammy & newer $resetvid
 apparmor.service.d/30_live_mode.conf: ConditionPathExists="
 umount "$targ1" 2>&1
 targ0=$(mktemp -d --tmpdir dus.XXXXXXXXXX)
 mount -o loop "$source" "$targ0" 2>&1
echo '$pptarg $targ1'": $pptarg $targ1"
 mount "$pptarg" "$targ1" 2>&1
 tghme=$(find "$targ0" -name '*ubuntu*.seed')
# echo "$tghme"
 tghme=${tghme##*/}
 tghme=${tghme%.seed}
 if [ "$tghme" == "ubuntustudio" ]
 then
  tghme='ubuntu-studio'
 elif [ "$tghme" == "" ]
 then
  tghme=$(grep -om1 'ubuntu-kylin' "$targ0"/casper/filesystem.manifest)
 fi
 echo "target's home directory: /home/$tghme"
 mkdir -p "$targ1"/upper/home/"$tghme"/.config/autostart
 mkdir -p "$targ1"/upper/usr/local/sbin
 cp /usr/share/mkusb/fixfox.desktop "$targ1"/upper/home/"$tghme"/.config/autostart
 cp /usr/share/mkusb/fixfox "$targ1"/upper/usr/local/sbin
 umount "$targ1" "$targ0" 2>&1
 rm -r "$targ1" "$targ0"
# read -p "press Enter to continue"
fi
---

I think tweak 3 is most relevant in this case (of this bug report).

I have also noticed that it is difficult to 'make all computers happy with the same configuration' even within the group that want an MSDOS partition table.

The bug of this report seems to be somewhat worked around by the version in dus-iso2usb version 22.0.1, but I think old HP computers with core2duo processors want a bootable flag on the partition where grub is located. However, adding that boot flag seems to stop booting in UEFI mode. I am trying to make a configuration work for all these test cases ...

[Thomas Schmitt (scdbackup)]

Hi,

sudodus wrote:
> # tweak 3: search by UUID
> sed -i '/set isofile/'a" search --set=root --fs-uuid $uid3" "$targ1"/boot/grub/grub.cfg
> sed -i 's/(hd0,3)/($root)/' "$targ1"/boot/grub/grub.cfg

Where does the grub.cfg come from, in which you find "set isofile"
and "(hd0,3)" ?

I looked into /boot/grub/grub.cfg of ubuntu-22.04-desktop-amd64.iso and
of ubuntu-20.04.1-live-server-amd64.iso . Nothing to see of above search
texts.

Have a nice day :)

Thomas

[sudodus (nio-wiklund)]

Hi again Thomas,

The grub comes from a compressed tarball, that is extracted onto the target drive. This makes it independent of the current operating system and its version of grub.

So in order to look at it, use dus-iso2usb, or if you want only the basics, extract only the tarball, in this case if you installed mkusb 22.0.1

/usr/share/mkusb/dd_grub-boot-template-for-uefi-n-bios_msdos_grub-2.0.2-n-2.0.4.img.xz

If you don't want to install mkusb into your main system, you can install it into a persistent live system in a USB pendrive or into a system in a virtual machine.

[Thomas Schmitt (scdbackup)]

Hi,

sudodus wrote:
> The grub comes from a compressed tarball,

Well, then the hope to find a simple explanation from tweak 3 was an
illusion.

I am staring at the /boot/grub/grub.cfg of ubuntu-22.04-desktop-amd64.iso
and compare it with Chris Guiver's comment #90:
>...> "00:10 secs enter pressed at grub
>...> BLINKING CURSOR without change until
>...> 7:19 (7 mins 19 secs) first boot message

So i assume that /boot/grub/grub.cfg was executed. But then would last very
long to get from
        linux /casper/vmlinuz file=/cdrom/preseed/ubuntu.seed maybe-ubiquity quiet splash ---
        initrd /casper/initrd
to a usable system.

Do i look at the wrong ISO ?
(I could not find Ubuntu "live" ISOs in the web.)

Have a nice day :)

Thomas

[sudodus (nio-wiklund)]

At

https://releases.ubuntu.com/22.04/

you find the amd64 iso file and its sha256sum, and at

http://cdimage.ubuntu.com/lubuntu/releases/22.04/release/

you find the corresponding Lubuntu files.

[Thomas Schmitt (scdbackup)]

Hi,

sudodus wrote:
> http://cdimage.ubuntu.com/lubuntu/releases/22.04/release/

The "Try or Install" menu item of grub.cfg in lubuntu-22.04-desktop-amd64.iso
differs from the one of ubuntu-22.04-desktop-amd64.iso mainly by kernel
parameter file=/cdrom/preseed/lubuntu.seed instead of ubuntu.seed.

The .seed files differ by:

--- ubuntu.seed 2022-05-15 19:02:13.187611554 +0200
+++ lubuntu.seed 2022-05-15 19:01:54.003611482 +0200
@@ -1,9 +1,8 @@
+# The Lubuntu seeds assume that installation of Recommends is disabled.
+d-i base-installer/install-recommends boolean true
 # Enable extras.ubuntu.com.
 d-i apt-setup/extras boolean true
-# Install the Ubuntu desktop.
-tasksel tasksel/first multiselect ubuntu-desktop
-# On live DVDs, don't spend huge amounts of time removing substantial
-# application packages pulled in by language packs. Given that we clearly
-# have the space to include them on the DVD, they're useful and we might as
-# well keep them installed.
-ubiquity ubiquity/keep-installed string icedtea6-plugin openoffice.org
+# Install the Lubuntu desktop.
+tasksel tasksel/first multiselect standard, lubuntu-desktop
+# No LXDE translation packages yet.
+d-i pkgsel/language-pack-patterns string

It is noteworthy that ubuntu.seed talks of avoiding "huge amounts of time",
whereas lubuntu.seed does not.

Does ubuntu-22.04-desktop-amd64.iso boot faster than the lubuntu ISO ?

Have a nice day :)

Thomas

[sudodus (nio-wiklund)]

@ guiverc,

I think you have suitable hardware and the best experience of us to answer the question in comment # 99.

[Chris Guiver (guiverc)]

Booting the released image of Ubuntu Desktop 22.04 LTS

- 00:00 word 'GRUB' appeared on screen (where I started stopwatch, so these results will be ~10 secs slower sorry)

- 00:09 enter pressed at GRUB
- 08:06 screen blacked
- 08:20 ubuntu plymouth first appeared
- 09:52 plymouth is gone, system text message appears
- 10:10 maybe-ubiquity TRY INSTALL prompt
- 10:37 system fully operational

quick comparison with comment #90 (https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1922342/comments/90) shows ~minute longer to be fully functional; given a 1-5 secs gets lost as touch 'try ubuntu' it's still longer

lubuntu (jammy) plymouth seen at 7:27 (#90)
 ubuntu (jammy) plymouth seen at 8:20 (#101)

[Thomas Schmitt (scdbackup)]

Hi,

Chris Guiver wrote:
> lubuntu (jammy) plymouth seen at 7:27 (#90)
> ubuntu (jammy) plymouth seen at 8:20 (#101)

So it is not about the "huge amounts of time" which ubuntu.seed wants
to avoid.
Are those times stable or do they vary by minutes ?

We know that the firmware executes GRUB without much delay.
I am pondering how we could find out when GRUB hands over execution
to vmlinuz. Any ideas anybody ?

Have a nice day :)

Thomas

[Chris Guiver (guiverc)]

I grabbed my phone (stopwatch app) & lubuntu (kinetic; what was used last time) thumb-drive & timed a boot of Lubuntu kinetic (no new write of ISO)

Purpose:
> (#102) Are those times stable or do they vary by minutes ?

I hoped it would be a re-creation of #92 (but didn't read was there so it wouldn't influence result), alas I didn't use stopwatch then & didn't note which option I used

> (#92) I didn't set up stopwatch to time correctly; but room clock said 13:16 at start of boot, and system was fully functional just as it hit 13:22

ie. system operation in ~5 mins (<5 I believe as clock had just flipped 13:22 as it appeared), but I didn't say if persistent OR live was used.

OPTION USED THIS TIME = LIVE only (not persistent)

BOOT 1: LIVE
started stopwatch as I pressed ENTER at grub
by 2 mins 43 secs the system appeared functional

BOOT2: LIVE
00:25 secs & screen blanks & messages
00:35 plymouth visible
01:11 message(s) again & plymouth gone
02:23 system fully-operational

BOOT3: LIVE
00:29 secs & screen blanks & messages
00:35 plymouth visible
01:10 message(s) again & plymouth gone
02:25 system fully-operational

BOOT4: PERSISTENCE
00:27 secs & screen blanks & messages
00:34 plymount visible
01:41 message(s) again & plymouth gone
03:06 system fully-operational

(boot 4 and there was a longer delay between wallpaper being drawn & the menu responding to wacom-pen or SUPER key being pressed & menu appearing which is what I use to detect 'fully-operational)

Boot 2 & 3 are identical; any differences will be me needing to press the screen on the phone more than once to get it to register 'lap' time.

Re-creating test #101 using Ubuntu Desktop 22.04 LTS (comment #101)

> - 00:09 enter pressed at GRUB
> - 08:06 screen blacked
> - 08:20 ubuntu plymouth first appeared
> - 09:52 plymouth is gone, system text message appears
> - 10:10 maybe-ubiquity TRY INSTALL prompt
> - 10:37 system fully operational

(I started time at ENTER PRESSED at GRUB, so 9 secs difference expected)

08:01 screen blanked
08:08 ubuntu plymouth first appeared
09:41 plymouth is gone, system text message(s) appear
10:08 maybe-ubiquity; TRY/INSTALL prompt
10:55 system fully operational

Times here start out consistent (9 secs difference expected as I started time with right hand as I hit enter at grub with left, last time it was when I saw 'grub' prior to grub.menu appearing), however at MAYBE-UBIQUITY times differed a bit.. and final-operational differs well beyond any user/me error(s)

Picking when a system is 'fully operational' is subjective, so a few secs difference can be just me using wacom-pen touching screen OR hitting SUPER at the right sec, versus a short delay (as I'm not hitting key/pen rapidly), but we have ~30 secs difference (10mins37secs-9sec 10m55s?)

> (#102) Are those times stable or do they vary by minutes ?

To me as a user they feel ~stable; but BOOT 1 of Lubuntu LIVE was slower than BOOTS 2 & 3 (which were consistent; boot 4 was different mode).. and some some times of Ubuntu Desktop were identical; but by end they differed beyond I believe could be me.

They don't vary by minutes though in my opinion; secs like this is all I've generally noted (a...

[Thomas Schmitt (scdbackup)]

Hi,

Chris Guiver wrote:
> BOOT2: LIVE
> 00:25 secs & screen blanks & messages
> 00:35 plymouth visible
> 01:11 message(s) again & plymouth gone
> 02:23 system fully-operational

The overall boot time is still very long. But no particular stage stands
out as the big time waster.

>...> motion computing j3400

This one is possibly allowed to be somewhat slow.

> Ubuntu Desktop 22.04 LTS
> 08:01 screen blanked

I really wonder which side, GRUB or vmlinuz, causes this delay.

After wasting time with clueless experiments i re-read what we have and
what i vastly forgot.

----------------------------------------------------------------------
This prompts another test request @ Chris Guiver:

Does it help to remove MBR partition 2 similarly to what i proposed a year
ago to José Marinho and what helped to boot:

  # (When replacing the "X", take care not to zap your system disk)
  STICK=/dev/sdX
  dd if=/dev/zero bs=1 count=16 of="$STICK" conv=notrunc seek=462

Last year this worked only once, because casper added the partition again
during "persistent partition creation". This could be fixed now:
  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1899308/comments/60
If not fixed: A second dd treatment after the first boot was not overridden
by casper in the next Live system run.

----------------------------------------------------------------------
Summary of my re-reading:

José Marinho wrote in
https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1922342/comments/35
> With the iso without any change:
> - Plymouth --> 7 minutes
> - Desktop environment --> 7 minutes 50 seconds
> Marking the first partition as bootable legacy BIOS:
> - Plymouth --> 54 seconds.
> - Desktop environment --> 1 minute 45 seconds
> Without marking any partition as bootable but changing the partition table from GPT to MBR:
> - Plymouth --> 25 seconds.
> - Desktop environment --> 1 minute 20 seconds.

So we had already a good suspect for a delay of about 6 minutes.

Then we had the red herring of these lines in grub.cfg which for a while
seemed to be the culprit. I proposed a change in #39. But in #55,
José Marinho finally reported that the intermediate success was rather
due to inadverted partition table changes during the experients.

In #60 i propose to remove partition 2 from the MBR.
(This partition exists to lure in some HP laptops which want to see a
boot flag. This flag is forbidden for the GPT-announcing partition slot.)
José Marinho reports in #61 that this helps for one time booting, but not
for further attempts to boot.
In #70 i demonstrated that the software in the ISO manipulates the MBR
partition table of the stick.
In #73 i point to "casper" and its "persistent partition creation".

Chris Guiver introduced the j3400 tablet to us in #76.

Chris Guiver and Brian Murray let GRUB be verbose in #82 and #83.
It prints:
> "Booting a command list"
Screenshots are at #84 and #85.

Since the messages do not look to me like from a Linux kernel, it appears
that the delay is in GRUB.
We could ask at grub-devel for help. But it happens only to a few (old ?)
machines and it is about the boot flag, which at least Vladimir Serbinko
rejected firmly when the HP laptops showed...

[sudodus (nio-wiklund)]

I agree with Thomas about the problems to boot some old computers.

I'm continuing to develop/tweak dus-iso2usb, added an extra (optional) parameter 'boot-flag' that can be added when mkusb partition table is selected. It helps HP computers, but may spoil booting some other computers.

So I think we probably need ways to get boot drive versions with and without a boot flag. One way would be two different iso files, another way is to use some special tool (like dus-iso2usb) for the corner cases that need a boot flag.

[Thomas Schmitt (scdbackup)]

Hi,

i have to correct a statement of mine in #104:

> Last year this worked only once, because casper added the partition again
> during "persistent partition creation". This could be fixed now:
> https://bugs.launchpad.net/ubuntu-cdimage/+bug/1899308/comments/60

It's probably not fixed.
The comment by Steve Langasek rather announces the problem which José
Marinho experienced:
casper was probably adjusted to create the boot flag when creating the
persistent partition. Originally it did not preserve it.

We will see what the j3400 does on first and second boot attempt.

Have a nice day :)

Thomas

[Chris Guiver (guiverc)]

Thomas Schmitt in #104

>...> motion computing j3400
> This one is possibly allowed to be somewhat slow.

From me (comment #76)
> ((or 3 mins 30 secs for `hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)`

The time for lubuntu kinetic actually beat a comparison time of one of an old hp.dc7700 I use heaps in QA-testing.

> This prompts another test request @ Chris Guiver:
> Does it help to remove MBR partition 2 similarly to what i proposed a year ago to José Marinho and what helped to boot:

I performed the command

`sudo dd if=/dev/zero bs=1 count=16 of=/dev/sdb conv=notrunc seek=462`

to Ubuntu Desktop 22.04 LTS thumb-drive (K)

Booted on j3400

(note: times for BOOT1 will be ~3-8 secs slower than reported due my-error operating phone-stopwatch)

BOOT1:
(I won't give times as I can't be sure which are which)
00:18, 00:27, 01:57, 02:18, 02:45
HOWEVER either way it was fully-operational at 2 min 45 secs (or 3-8 secs after that anyway)

system just rebooted (alt-sysreq+reisub)

BOOT2:
00:25 screen clears & message(s)
00:28 plymouth seen
02:17 maybe-ubiquity; try/install selection (I may have been a tad slow here in selecting try)
02:41 (i forget sorry)
03:11 system fully-operational

Maybe my [estimated] 3-8 secs is inaccurate; but I'd suggest it was similar.

FYI: If you need more testing; just ask (or I miss anything I was asked).. I'll boot the thumb-drive on a few boxes to confirm no issues with the

I'll go & boot the thumb-drive on some boxes to ensure it's usable elsewhere.. I'll edit this and add boxes it was tested on (I'm not expecting any issues)

It booted successfully on (got to maybe-ubiquity/try/install anyway; then I sysrq-reisuo)
- hp 8200 elite sff (i5-2400, 8gb, nvidia quadro 600)
- dell [optiplex] 755 (c2d-e6850, 5gb, amd/ati radeon rv516/x1300/x1550)
- dell [optiplex] 780 (c2q-q9400, 8gb, amd/ati cedar radeon hd 5000/6000/7350/8350)
- sony vaio svp112a1cw (i5-9400u, 4gb, intel haswell-ULT)
- samsung 700t1c-p10aat (i5-3317u, 4gb, 3rd gen.core.intel.graphics.4000)

BUT FAILED TO BOOT on
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)

[Thomas Schmitt (scdbackup)]

Hi,

Chris Guiver wrote:
> I performed the command
> `sudo dd if=/dev/zero bs=1 count=16 of=/dev/sdb conv=notrunc seek=462`
> to Ubuntu Desktop 22.04 LTS thumb-drive (K)
> Booted on j3400
> BOOT1:
> it was fully-operational at 2 min 45 secs

So it looks like the dd run brought about the best boot-up speed we can
expect from this hardware. (Please contradict me if i'm wrong.)

> system just rebooted (alt-sysreq+reisub)
> 03:11 system fully-operational

So casper did not re-install the dummy partition 2 when creating a persistent
partition.

Well, did it create a new partition at all ?
What do you get from

  xorriso -indev /dev/sdb -report_system_area plain

Expectation:
The report about the MBR partition table should list only one partition

  MBR partition table: N Status Type Start Blocks
  MBR partition : 1 0x00 0xee 1 5090363

with possibly the Blocks number adjusted to the size of your USB stick.
(The original ISO would show a MBR partition 2
  MBR partition : 2 0x80 0x00 0 1
which obviously makes the trouble with j3400.)

The GPT part of the report would list a fourth GPT partition, if casper
created one.

> BUT FAILED TO BOOT on
> - hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)

This is obviously one of the machines which demand a boot flag in MBR.

Does it boot after you (rawly and naughtily) set the boot flag to MBR
partition 1:

  echo $'\x80' | dd of=/dev/sdX bs=1 count=1 conv=notrunc seek=446

(If Ubuntu decides to leave these machines behind, this run could be the
remedy for hp dc7700 and others. It is simpler than a dd run which installs
partition 2. On the other hand it is clearly violating the specs.)

What does the j3400 do with this state of the USB stick ?

(If it boots slowly:
Above dd run can be revoked by "cat /dev/zero" instead of "echo $'\x80'".)

-----------------------------------------------------------------------

It looks like a fundamental decision by Ubuntu is needed:

Since GPT specs forbid the boot flag with the MBR partition of type 0xee
and since some EFI implementations indeed take offense if it is there,
the MBR partition with its boot flag was the best compromise ... until
j3400 and José Marinho's machine of which i could not find a model name.

But now seems that Ubuntu needs to decide with its ISOs whether to leave
behind either j3400 or hp dc7700.
A good reason for keeping j3400 bootable is that this partition layout is
fully specs compliant and tasty to all EFI implementations (... so far).

The only partition layout for hp dc7700 which complies to UEFI and its GPT
specs would be a MBR partition table marking the EFI partition by type 0xEF
and no GPT.
But although this is covered by the specs, there are EFI implementations
known which don't boot from a device without GPT.

(Really annoying to me is the fact that the old ISOLINUX/GRUB jackalope works
for all machines with its MBR partition table and a dead GPT strapped on its
back. Somehow i hope for a widely used EFI which hates it and forces it
out of the world.)

I guess that it is not an option to provide full ISO sets for both, the
mainstream and some old exotic fi...

[Chris Guiver (guiverc)]

My thumb-drive K (what was used in #107 & #101) which contains Ubuntu 22.04 LTS (20220416) resulted in

PLEASE NOTE: I just noted the ISO date shows it wasn't the released image.. It'll be the last daily/final/RC i actually used in testing... I'd expect 20220419 for the released ISO. I don't expect it'll impair these results though.

(parts of lsblk & blkid are included... I'm not sure the provided command xorriso contains I expected)

sdb 8:16 1 7.5G 0 disk
├─sdb1 8:17 1 3.4G 0 part /media/guiverc/Ubuntu 22.04 LTS amd64
├─sdb2 8:18 1 4.1M 0 part
├─sdb3 8:19 1 300K 0 part
└─sdb4 8:20 1 4G 0 part /media/guiverc/writable
sr0 11:0 1 1024M 0 rom

guiverc@d960-ubu2:~/uwn/issues/735$ xorriso -indev /dev/sdb -report_system_area plain
xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.

xorriso : FAILURE : Drive address '/dev/sdb' rejected because: not MMC and -drive_class 'caution' '/dev'
xorriso : HINT : If the address is a legitimate target, prepend "stdio:"
xorriso : aborting : -abort_on 'FAILURE' encountered 'FAILURE'
guiverc@d960-ubu2:~/uwn/issues/735$ lsblk^Crriso -index /dev/sdb -report_system_area plain
guiverc@d960-ubu2:~/uwn/issues/735$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
.. redacting loop devices & my sda disk
sdb 8:16 1 7.5G 0 disk
├─sdb1 8:17 1 3.4G 0 part /media/guiverc/Ubuntu 22.04 LTS amd64
├─sdb2 8:18 1 4.1M 0 part
├─sdb3 8:19 1 300K 0 part
└─sdb4 8:20 1 4G 0 part /media/guiverc/writable

-- I stopped at this point & haven't performed
> This is obviously one of the machines which demand a boot flag in MBR. Does it boot after you (rawly and naughtily) set the boot flag to MBR

Please advise if I should continue, or other (the xorriso output didn't appear right to me)

[Thomas Schmitt (scdbackup)]

Hi,

Chris Guiver wrote:
> └─sdb4 8:20 1 4G 0 part /media/guiverc/writable

So it seems that casper added its persistent partition without creating
a dummy MBR partition with boot flag.
This simplifies the task of making current Ununtu ISOs digestible for
the j3400. One run of

  dd if=/dev/zero bs=1 count=16 of="$STICK" conv=notrunc seek=462

suffices.

Chris Guiver wrote:
> guiverc@d960-ubu2:~/uwn/issues/735$ xorriso -indev /dev/sdb -report_system_area plain
> xorriso : FAILURE : Drive address '/dev/sdb' rejected because: not MMC and -drive_class 'caution' '/dev'

My mistake. The program is right. The command proposal should have been

  xorriso -indev stdio:/dev/sdb -report_system_area plain

Please run this to verify that the dummy partition is really not there
after the first successful booting of the USB stick.

(The demand for the "stdio:" prefix is a safety precaution to protect
system disks from dangerous superuser activities.
I forgot about it because i have a file /etc/opt/xorriso/rc which declares
  -drive_class harmless '/dev/sd[c-e]*'
so that i don't have to use "stdio:" with my USB sticks.
Whatever, -indev without -outdev to the same device prevents writing
and -report_system_area "plain" does not want to write to the device.
So it would be safe even for the system disks.)

Have a nice day :)

Thomas

[Chris Guiver (guiverc)]

My thumb-drive K does not boot on
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
- hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)
as it stands now.. (No big news as dc7700 & dc7900 usually react the same; but confirmed it won't boot on dc7900)

Likely not required; but again
--
guiverc@d960-ubu2:~/uwn/issues/735$ xorriso -indev stdio:/dev/sdb -report_system_area plain
xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.

libburn : SORRY : Failed to open device (a pseudo-drive) : Permission denied
xorriso : FAILURE : Cannot acquire drive 'stdio:/dev/sdb'
xorriso : aborting : -abort_on 'FAILURE' encountered 'FAILURE'
--
Executed
--
[sudo] password for guiverc:
root@d960-ubu2:~# echo $'\x80' | dd of=/dev/sdb bs=1 count=1 conv=notrunc seek=446
1+0 records in
1+0 records out
1 byte copied, 0.000132766 s, 7.5 kB/s
root@d960-ubu2:~#
--

Safely ejected thumb-drive; tested and it NOW BOOTS ON
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
- hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)

I only booted to maybe-ubiquity|try/install then used sysrq-REISUO. If you needed me to complete boot & shutdown like a user, you'll have to ask me to do that.

I tested boot again on
- dell [optiplex] 755 (c2d-e6850, 5gb, amd/ati radeon rv516/x1300/x1550)
- dell [optiplex] 780 (c2q-q9400, 8gb, amd/ati cedar radeon hd 5000/6000/7350/8350)
and sysrq-REISUO when it reached maybe-ubiquity|try/install

The following would NOT BOOT thumb-drive K now
- sony vaio svp112a1cw (i5-9400u, 4gb, intel haswell-ULT)
(I didn't get it to boot on samsung 700t1c-p10aat either, but I can't rule out me; as I find that device a pain booting external media)

Inserting thumb-drive K into my primary box
--
guiverc@d960-ubu2:~/uwn/issues/735$ xorriso -indev stdio:/dev/sdb -report_system_area plain
xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.

libburn : SORRY : Failed to open device (a pseudo-drive) : Permission denied
xorriso : FAILURE : Cannot acquire drive 'stdio:/dev/sdb'
xorriso : aborting : -abort_on 'FAILURE' encountered 'FAILURE'
guiverc@d960-ubu2:~/uwn/issues/735$ xorriso -indev stdio:/dev/sdb -report_system_area plain
xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.

libburn : SORRY : Failed to open device (a pseudo-drive) : Permission denied
xorriso : FAILURE : Cannot acquire drive 'stdio:/dev/sdb'
xorriso : aborting : -abort_on 'FAILURE' encountered 'FAILURE'

guiverc@d960-ubu2:~/uwn/issues/735$ sudo xorriso -indev stdio:/dev/sdb -report_system_area plain
[sudo] password for guiverc:
xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.

xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE : 940 nodes read in 1 seconds
libisofs: NOTE : Found hidden El-Torito image for EFI.
libisofs: NOTE : EFI image start and size: 1782345 * 2048 , 8496 * 512
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -indev 'stdio:/dev/sdb'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record : El Torito , MBR protective-msdos-label grub2-mbr cyl-align-off GPT
Med...

[Thomas Schmitt (scdbackup)]

Hi,

Chris Guiver wrote:
> libburn : SORRY : Failed to open device (a pseudo-drive) : Permission denied

Oops. No read permission for normal users on USB sticks.
(I should really operate my workstation with a more conventional setup
so that i better anticipate other's adventures.)

> My thumb-drive K does not boot on
> - hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
> - hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)
> as it stands now.

Obviously the addicted HPs don't see a boot flag.

> echo $'\x80' | dd of=/dev/sdb bs=1 count=1 conv=notrunc seek=446
> it NOW BOOTS ON
> - hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
> - hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)

Another confirmation that the flag was not set before.

The only small chance for a red herring would be a prolonged history of
experiments with the ISO on the stick, which would have caused casper to
not do what it normally does. We had that in the #50s comments.

--------------------------------------------------------------------------

So if we want to propose a workaround for the current layout, it would
be a good base if you could confirm that j3400 boots after the plain
procedure with no other experiments inbetween:

  # Patch the ISO already as image on hard disk
  ISO=ubuntu-22.04-desktop-amd64.iso
  dd if=/dev/zero bs=1 count=16 of="$ISO" conv=notrunc seek=462

  # Put it onto the USB stick as you usually do. E.g.
  sudo dd bs=4M oflag=sync status=progress of=/dev/sdb if="$ISO"

Then you would check if it boots twice on the j3400. If the second boot
succeeds and shows a writable big partition as last one, then we could be
sure that this procedure is a valid workaround.

A run of

  sudo xorriso -indev stdio:/dev/sdb -report_system_area plain

would (hopefully) confirm that there is no second MBR partition with boot
flag.

Of course it would be enough if you can confirm that you did this already
during the experiments, with no intermediate manipulations.
But given the curvy way of this bug report, we need to be sure.

--------------------------------------------------------------------------

Assumed that

  dd if=/dev/zero bs=1 count=16 of="$ISO" conv=notrunc seek=462

is really a reliable way to make the Ubuntu ISOs boot on the few old
BIOS machines, which slow down GRUB when encountering the combination of
GPT and MBR dummy partition, we would need a way to inform the downloaders
of Ubuntu ISOs about this trick.

To whom would we have to talk to get this proposal onto sites like
  https://ubuntu.com/tutorials/install-ubuntu-desktop
which gets pointed to by a link on
  https://ubuntu.com/download/desktop

I think it should be mentioned in
  https://ubuntu.com/tutorials/install-ubuntu-desktop#4-boot-from-usb-flash-drive
like
  "On some very old and meanwhile rare machines it can last 8 minutes or
   longer until you see this welcome screen.
   If you plan just a single installation then simply wait until the screen
   appears.
   But if you plan to repeatedly use the "Try Ubuntu" offer on such an old
   machine, then see [link to new page tutorials/remove-boot-flag-from-iso]
   for a way to substantially shorten this time span."

The ...

[Thomas Schmitt (scdbackup)]

Hi,

i wrote towards Chris Guiver:
> it would
> be a good base if you could confirm that j3400 boots after the plain
> procedure with no other experiments inbetween

I should have written:

  it would be a good base if you could confirm that j3400 boots without
  the extra delay of ~6 minutes after the plain procedure with no other
  experiments inbetween

Reading my text proposal for the tutorial, i now think it should give
less info. Like

  On some very old and meanwhile rare machines it can last 10 minutes or
  longer until you see this welcome screen.
  If you really have to wait that long, consider to read the article [link]
  about a possible cause and a workaround.

The article would explain that
- the Ubuntu ISOs contain a boot flag as inavoidable workaround for booting
  some old and not so rare machines,
- which causes the long boot delay with some other old and rarer machines,
- and can be disabled by the dd procedure.

Have a nice day :)

Thomas

[Chris Guiver (guiverc)]

-- reply to @Thomas #112 (start paste)
So if we want to propose a workaround for the current layout, it would be a good base if you could confirm that j3400 boots after the plain procedure with no other experiments inbetween:

  # Patch the ISO already as image on hard disk
  ISO=ubuntu-22.04-desktop-amd64.iso
  dd if=/dev/zero bs=1 count=16 of="$ISO" conv=notrunc seek=462

  # Put it onto the USB stick as you usually do. E.g.
  sudo dd bs=4M oflag=sync status=progress of=/dev/sdb if="$ISO"

Then you would check if it boots twice on the j3400. If the second boot succeeds and shows a writable big partition as last one, then we could be sure that this procedure is a valid workaround.

A run of

  sudo xorriso -indev stdio:/dev/sdb -report_system_area plain

would (hopefully) confirm that there is no second MBR partition with boot flag.

Of course it would be enough if you can confirm that you did this already during the experiments, with no intermediate manipulations. But given the curvy way of this bug report, we need to be sure.
-- (end paste)

I grabbed my thumb-drive 2; Ubuntu-MATE jammy (2022-04-19)
- plugged into j3400; enter pressed [just] @12:28 using room clock, plymouth seen at 12:36, maybe-ubiquity appeared at 12:38

This wasn't required/requested; but I omitted `sudo` for last; so added here

--- (start paste)
guiverc@d960-ubu2:~/uwn/issues/735$ sudo xorriso -indev stdio:/dev/sdb -report_system_area plain
[sudo] password for guiverc:
xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.

xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE : 774 nodes read in 1 seconds
libisofs: NOTE : Found hidden El-Torito image for EFI.
libisofs: NOTE : EFI image start and size: 1422522 * 2048 , 8496 * 512
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -indev 'stdio:/dev/sdb'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record : El Torito , MBR protective-msdos-label grub2-mbr cyl-align-off GPT
Media summary: 1 session, 1424812 data blocks, 2783m data, 4849m free
Volume id : 'Ubuntu-MATE 22.04 LTS amd64'
System area options: 0x00004201
System area summary: MBR protective-msdos-label grub2-mbr cyl-align-off GPT
ISO image size/512 : 5699248
Partition offset : 16
MBR heads per cyl : 0
MBR secs per head : 0
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x00 0xee 1 15630335
MBR partition : 2 0x80 0x00 0 1
GPT : N Info
GPT disk GUID : 92da1293cf19e54b80558e26d983bc85
GPT entry array : 2 248 separated
GPT lba range : 64 15630272 15630335
GPT partition name : 1 490053004f003900360036003000
GPT partname local : 1 ISO9660
GPT partition GUID : 1 92da1293cf19e54b80548e26d983bc85
GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 1 0x1000000000000001
GPT start and size : 1 64 5690024
GPT partition name : 2 41007000700065006e006400650064003200
GPT partname local : 2 Appended2
GPT partition GUID : 2 92da1293cf19e54b80578...

[Thomas Schmitt (scdbackup)]

Hi,

Chris Guiver wrote:
> On j3400; pressed ENTER (at grub) at 13:10 (room clock).. I didn't note
> time of plymouth but I had maybe-ubiquity @ 13:13 (room clock) for
> altered version of 2022-04-19 ISO
> [...]
> so I'll now boot twice
> [...]
> alas should have noted clock.. it 'feels' slow... nah it's slow :(
> [...]
> another boot on j3400, hit ENTER (at grub) at 13:31, maybe-ubiquity jingle
> waking me from sleep at 13:41
> [...]
> MBR partition : 2 0x80 0x00 0 1

Looks like casper is still re-installing MBR partition 2 on the first
run of the Live system, although it was not there before this run.

If you now apply the remedy to the stick
  dd if=/dev/zero bs=1 count=16 of=/dev/sdb conv=notrunc seek=462
then i expect it to be permanently booting in reasonable time.
(Not because it was applied to the stick, but because casper will not
manipulate the partition table any more.)

Please confirm.

----------------------------------------------------------------------

I had hoped that capser would save the partition entry and re-install it after
the creation of the persistent overwrote the MBR partition table. But it
seems to just write a pre-recorded partition entry with the boot flag.

So we'd need a change in casper if the description of the workaround
shall be reasonably safe.
I don't consider dd of=/dev/sdb safe considered that the Ubuntu tutorial
advises to use Balena Etcher for the copy-to-USB-stick step.

Casper would have to save before the additon of the persisten partition
at least 16 bytes beginning at offset 462, or 48 bytes to cover all three
unused MBR partitions. After creation of the partition the saved bytes
would be written back to the MBR of the USB stick.

In
  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1899308/comments/66
i read:

   casper (1.455) groovy; urgency=medium
     [...]
     * Limit our dd to only change 16 bytes in the MBR regardless of input,
     [...]
   -- Steve Langasek <email address hidden> Fri, 16 Oct 2020 09:51:20 -0700

But this is not what Chris Guiver observes.

In
  https://git.launchpad.net/ubuntu/+source/casper/commit/?id=5c3637a224e7eca4c8998d72ac1711cd8b58b335
i see unconditional insertion of the boot flag partition:

  + echo $escape_arg -n '\0200\00\01\00\00\00\01\00\00\00\00\00\01\00\00\00' \
  + | dd of=$DEVICE bs=1 seek=462 conv=notrunc count=16

The current state in
  https://git.launchpad.net/ubuntu/+source/casper/tree/scripts/casper-helpers
still has this gesture:

  find_or_create_persistent_partition () {
      [...]
      echo "start=$start" | sfdisk --no-reread -q $DEVICE -a || return
      [...]
      if sfdisk -d $DEVICE | grep -q 'label: gpt'; then
          [...]
          echo $escape_arg -n '\0200\00\01\00\00\00\01\00\00\00\00\00\01\00\00\00' \
          | dd of=$DEVICE bs=1 seek=462 conv=notrunc count=16
      fi
      [...]

I would propose to do something like:

 + MBR_STASH=/tmp/mbr_stash_$$
 + dd if="$DEVICE" bs=1 skip=462 count=48 of="$MBR_STASH"
      echo "start=$start" | sfdisk --no-reread -q $DEVICE -a || return
      [...]
 - echo $escape_arg -n '\0200\00\01\00\00\00\01\00\00\00\00\00\01\00\00\00' \
 - ...

[Chris Guiver (guiverc)]

> If you now apply the remedy to the stick
> dd if=/dev/zero bs=1 count=16 of=/dev/sdb conv=notrunc seek=462
>then i expect it to be permanently booting in reasonable time.
> .. Please confirm.

Inserted thumb-drive(2) [Ubuntu-MATE jammy] into box and

-- start paste
guiverc@d960-ubu2:/de2900/ubuntu_mate_64$ sudo dd if=/dev/zero bs=1 count=16 of=/dev/sdb conv=notrunc seek=462
[sudo] password for guiverc:
16+0 records in
16+0 records out
16 bytes copied, 0.0107609 s, 1.5 kB/s
-- end paste

booted it on j3400
- motion computing j3400 (c2d-u9400, 4gb, intel mobile 4 series)

on boot (noting when at grub) it had reached maybe-ubiquity in ~3 mins using room clock (no seconds as digital). I made an error in typing SysRq-REO (intended REISUB) so as no sync.. ignoring this bot.

turned box back on & booted again; again at maybe-ubiquity in ~3 mins by room clock. Correctly used SysRq-REISUB to restart

another boot & again ~3 mins by room clock. SysRq-REISUO now

NOTE: I didn't wait for clock to change minute & press enter this time, thus the extra seconds (making 2+ mins ~3 mins) is just ME not starting hitting the timing at ~:01 secs or hitting ENTER just after minute changes.. Either way ~3 mins is NOT TEN minutes as the slow boot takes to reach maybe-ubiquity.

This confirms what I believe you wanted.

[sudodus (nio-wiklund)]

Congratulations Thomas (for theory) and Chris (for experiments). This is like development of physics ;-)

[Thomas Schmitt (scdbackup)]

Hi,

Chris Guiver wrote:
> This confirms what I believe you wanted.

Yes.
The stick needs the remedy once again after casper created the persistent
partition.

sudodus wrote:
> This is like development of physics ;-)

Like a unification of relativity theory and quantum mechanics ?

My apologies again for not remembering the results from one year ago
and wasting everybody's time with my first questions of this year.
Between #94 and #103 i was off track.

I'll wait a few days whether Steve Langasek shows up at this bug report.
If not, i plan to file a new wishlist bug for casper which describes
our findings and proposes the code change as of #115.

I re-re-read this report to list the affected machines:
- a tablet computer "motion computing j3400" of Chris Guiver,
- a Gigabyte H61M-D2H-USB3 mainboard of José Marinho (see also
  https://launchpadlibrarian.net/531427797/lshw.txt),
- (a Gigabyte GA-970A-DS3, BIOS F7a 01/24/2013 of tlk in #91, of which we
   have no confirmation that removing MBR partition 2 really helps).

Have a nice day :)

Thomas

[tlk (sarcasticskull)]

which post details the "removing MBR partition 2"?
I'll try when I come around to it.

[Thomas Schmitt (scdbackup)]

Hi,

tlk wrote:
> which post details the "removing MBR partition 2"?

#112 by me, confirmed by #114 by Chris Guiver:

If the USB stick with the the slowly booting ISO is overwritten meanwhile:

  # Patch the ISO already as image on hard disk
  ISO=ubuntu-22.04-desktop-amd64.iso
  dd if=/dev/zero bs=1 count=16 of="$ISO" conv=notrunc seek=462

  # Put it onto the USB stick as you usually do. E.g.
  sudo dd bs=4M oflag=sync status=progress of=/dev/sdX if="$ISO"

In any case after the USB stick has already booted once, you have to apply
the change again directly to it, because casper installed MBR partition 2
again after creating its persistent partition:

  # Take care not to name any of your valuable disks as USB_STICK
  USB_STICK=/dev/sdX
  sudo dd if=/dev/zero bs=1 count=16 of="$USB_STICK" conv=notrunc seek=462

Have a nice day :)

Thomas

[Thomas Schmitt (scdbackup)]

Hi,

i opened a separate bug with my wish for not needing a second zeroizing dd
run when the ISO is already on the USB stick:

  https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1977644
  "Please preserve MBR partition entries 2 to 4 when creating persistent partition"

Have a nice day :)

Thomas

[tlk (sarcasticskull)]

I can confirm that "removing MBR partition 2" indeed WORKS.

I've tested the "second" step as I've already had booted from the stick a couple of times (and always had experienced the daunting delay).
Thus I've patched the partition on the stick directly as you've advised.

As expected the insanely long pause after hitting enter on the GRUB menu is gone.

Thank you for your investigation!

Incidentally, I'm assembling a new desktop system for me right now, it's a modern platform obviously... so I'm expecting all this to not be needed for it.

[Brian Murray (brian-murray)]

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.

[Launchpad Janitor (janitor)]

This bug was fixed in the package casper - 1.473

---------------
casper (1.473) kinetic; urgency=medium

  * casper-helper: preserve entries 2-4 in the MBR when creating a new
    partition. (LP: #1977644, #1922342)

 -- Michael Hudson-Doyle <email address hidden> Fri, 05 Aug 2022 17:45:05 +1200

[Thomas Schmitt (scdbackup)]

Hi,

the fix for #1977644 does not solve the problem of #1922342. It only reduces
the need for ISO patching to a single operation which can be done on the
ISO image file before putting it onto a USB stick (and not a second time
directly on /dev/sdX after the first boot):

  # Patch the ISO already as image on hard disk
  ISO=ubuntu-22.04-desktop-amd64.iso
  dd if=/dev/zero bs=1 count=16 of="$ISO" conv=notrunc seek=462

I think that #1922342 can only be considered solved if the users of Ubuntu
ISOs can easily find this procedure and an explanation when it should be
applied.
My proposal would be to put a brief hint into
  https://ubuntu.com/tutorials/install-ubuntu-desktop#4-boot-from-usb-flash-drive
and to add a new page
  https://ubuntu.com/tutorials/remove-boot-flag-from-iso
which gives details.

The brief hint would be like:

  "On some very old and meanwhile rare machines it can last 8 minutes or
   longer until you see this welcome screen.
   If you plan just a single installation then simply wait until the screen
   appears.
   But if you plan to repeatedly use the "Try Ubuntu" offer on such an old
   machine, then see [link to new page tutorials/remove-boot-flag-from-iso]
   for a way to substantially shorten this time span."

Does anybody here know how to contact the custodians of the tutorials ?

Have a nice day :)

Thomas

[sudodus (nio-wiklund)]

Hi Thomas,

It might work to ask [the same question] at

https://discourse.ubuntu.com/

in order to find how to contact the custodians of the tutorials.

[Chris Guiver (guiverc)]

Lubuntu daily booted (2022-09-17) on
- motion computing j3400 (c2d-u9400, 4gb, intel mobile 4 series)

boot started (used arrow keys at grub to stop booting, hit enter at grub prompt) at 16:44
just blank screen & cursor top left until 16:52 so nothing seen for 'ages'
at 16:52 the lubuntu plymouth started
by 16:53 the session had started

system boots fine... just slow with >8 mins of nothing changed on screen but a non-blinking cursor ... which leads people to think the boot failed as per initial bug report in my opinion.

(I've not tried Ubuntu desktop; can if helpful; but it always responded the same except slightly slower.. I'd expect the same here unless it was changed & Lubuntu/flavors weren't)

[Thomas Schmitt (scdbackup)]

Hi,

following sudodus' advice i found
  https://discourse.ubuntu.com/t/install-ubuntu-desktop/25312
matching
  https://ubuntu.com/tutorials/install-ubuntu-desktop#4-boot-from-usb-flash-drive
but the "Sign Up" button first urges me to enable JavaScript (booh !)
and then says my browser is too old for its "rich content".
I could probably work around, but this is a demand which i only accept
from my local tax authorities (teeth-gnashingly).

-----------------------------------------------------------------
@Chris Guiver:

On my way to that discourse page i came to
  https://discourse.ubuntu.com/t/create-a-bootable-usb-stick-on-ubuntu/14011
where i see comments from "guiverc".
So, i assume that you can point the participants of
  https://discourse.ubuntu.com/t/install-ubuntu-desktop/25312
to my summary in

  https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1922342/comments/125

Better warn them that the overall thread is long and full of wrong
theories. But meanwhile the diagnosis is clear.

As background you can point to

  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1886148

where Ubuntu ISO production had to switch from MBR partition table to
GPT partitioning, because some buggy EFI implementations need it,
and to

  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1899308

where Ubuntu ISO production had to add a MBR partition of type 0x00 with
boot flag for getting some buggy BIOS implementations to work again.
This boot flag partition is what slows down the boot process on the machines
which are reported at this bug (1922342).

If the need arises for discussing with me, then please on this bug report.

Have a nice day :)

Thomas