additional memory management changes

[Julian Andres Klode]

[Impact]

The 2.12 mm changes did not take into account grub's mm overhead when requesting pages from the firmware and hence can cause out of memory issues (i.e. we need 4 KiB, it allocates 4 KiB from the firwmare, but then only has like 4000 byte left or something left that, because it itself needs to store some metadata).

Also two performance improvements to those mm changes.

Full commits with more details:

commit 873be3c86c35f8c1d539b91bd5cc88a54036070a
Author: Zhang Boyang <email address hidden>
Date: Sun Jan 29 19:49:33 2023 +0800

    mm: Avoid complex heap growth math in hot path

    We do a lot of math about heap growth in hot path of grub_memalign().
    However, the result is only used if out of memory is encountered, which
    is seldom.

    This patch moves these calculations away from hot path. These
    calculations are now only done if out of memory is encountered. This
    change can also help compiler to optimize integer overflow checks away.

    Signed-off-by: Zhang Boyang <email address hidden>
    Reviewed-by: Daniel Kiper <email address hidden>

    Gbp-Pq: Topic 2.12-mm

commit ff73d7484d7f885cd1da82fe278fe607fb868b2d
Author: Zhang Boyang <email address hidden>
Date: Sun Jan 29 19:49:32 2023 +0800

    mm: Preallocate some space when adding new regions

    When grub_memalign() encounters out-of-memory, it will try
    grub_mm_add_region_fn() to request more memory from system firmware.
    However, it doesn't preallocate memory space for future allocation
    requests. In extreme cases, it requires one call to
    grub_mm_add_region_fn() for each memory allocation request. This can
    be very slow.

    This patch introduces GRUB_MM_HEAP_GROW_EXTRA, the minimal heap growth
    granularity. The new region size is now set to the bigger one of its
    original value and GRUB_MM_HEAP_GROW_EXTRA. Thus, it will result in some
    memory space preallocated if current allocations request is small.

    The value of GRUB_MM_HEAP_GROW_EXTRA is set to 1MB. If this value is
    smaller, the cost of small memory allocations will be higher. If this
    value is larger, more memory will be wasted and it might cause
    out-of-memory on machines with small amount of RAM.

    Signed-off-by: Zhang Boyang <email address hidden>
    Reviewed-by: Daniel Kiper <email address hidden>

    Gbp-Pq: Topic 2.12-mm

commit ef6115f3b411e63ce9125c60e572154024063d07
Author: Zhang Boyang <email address hidden>
Date: Sun Jan 29 19:49:31 2023 +0800

    mm: Adjust new region size to take management overhead into account

    When grub_memalign() encounters out-of-memory, it will try
    grub_mm_add_region_fn() to request more memory from system firmware.
    However, the size passed to it doesn't take region management overhead
    into account. Adding a memory area of "size" bytes may result in a heap
    region of less than "size" bytes really available. Thus, the new region
    may not be adequate for current allocation request, confusing
    out-of-memory handling code.

    This patch introduces GRUB_MM_MGMT_OVERHEAD to address the region
    management overhead (e.g. metadata, padding). The value of this new
    constant must be large enough to make sure grub_memalign(align, size)
    always succeeds after a successful call to
      grub_mm_init_region(addr, size + align + GRUB_MM_MGMT_OVERHEAD),
    for any given addr and size (assuming no integer overflow).

    The size passed to grub_mm_add_region_fn() is now correctly adjusted,
    thus if grub_mm_add_region_fn() succeeded, current allocation request
    can always succeed.

    Signed-off-by: Zhang Boyang <email address hidden>
    Reviewed-by: Daniel Kiper <email address hidden>

    Gbp-Pq: Topic 2.12-mm

[Test plan]
Boot a couple platforms make sure it works fine, let it rest in devel for a couple weeks until after the point release.

[where problems could occur]
It could fail to allocate memory in other ways now of course. Security issues because of overflows or stuff are always possible too. I did do some basic review though and that seemed fine.

[Marc Püschel (newcomer-01)]

@Julian Andres Klode (~juliank):

to add this ppa's

ppa:ubuntu-uefi-team/ubuntu/

ppa:ubuntu-uefi-team/ubuntu/grub2-signed/1.187.4~22.04.1

solves the problem for me!

After i added this ppa's I was able to install the updates over the Ubuntu Store App successfully.
=> https://abload.de/img/bildschirmfotovom2023gpeui.png

Thank you so much for your work!

Should I now remove the on top named ppa's again or can I let them in?
=> https://abload.de/img/bildschirmfotovom2023zqfhq.png
=> https://abload.de/img/bildschirmfotovom2023awigo.png

[Julian Andres Klode (juliank)]

Adding block-proposed for stable releases as we do not plan immediate SRU releases, just have them in proposed for at least two weeks.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu17

---------------
grub2-unsigned (2.06-2ubuntu17) lunar; urgency=medium

  * Cherry-pick more upstream memory patches (LP: #2004643)
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Julian Andres Klode <email address hidden> Mon, 20 Feb 2023 17:24:10 +0100

[Steve Langasek (vorlon)]

Hello Julian, or anyone else affected,

Accepted grub2-unsigned into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu17 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Julian, or anyone else affected,

Accepted grub2-signed into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.193 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Julian, or anyone else affected,

Accepted grub2-unsigned into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu14.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Julian, or anyone else affected,

Accepted grub2-signed into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.187.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Julian, or anyone else affected,

Accepted grub2-unsigned into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu14.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Julian, or anyone else affected,

Accepted grub2-signed into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.187.4~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Julian, or anyone else affected,

Accepted grub2-unsigned into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu14.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Julian, or anyone else affected,

Accepted grub2-signed into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.187.4~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Julian, or anyone else affected,

Accepted grub2-unsigned into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu14.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Julian Andres Klode (juliank)]

Lunar is the same binary as devel and this has been out for a while without reported regressions so we are ready to release.

[Mate Kukri (mkukri)]

Verified on focal, jammy, and lunar.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu17

---------------
grub2-unsigned (2.06-2ubuntu17) lunar; urgency=medium

  * Cherry-pick more upstream memory patches (LP: #2004643)
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Julian Andres Klode <email address hidden> Mon, 20 Feb 2023 17:24:10 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.193

---------------
grub2-signed (1.193) lunar; urgency=medium

  * Rebuild against grub2 2.06-2ubuntu17 (LP: #2004643)

 -- Julian Andres Klode <email address hidden> Wed, 29 Mar 2023 10:24:29 +0200

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for grub2-unsigned has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu14.2

---------------
grub2-unsigned (2.06-2ubuntu14.2) kinetic; urgency=medium

  * Cherry-pick more upstream memory patches (LP: #2004643)
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Julian Andres Klode <email address hidden> Mon, 20 Feb 2023 17:29:00 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.187.4~22.04.1

---------------
grub2-signed (1.187.4~22.04.1) jammy; urgency=medium

  * Rebuild against grub2 2.06-2ubuntu14.2 (LP: #2004643)

 -- Julian Andres Klode <email address hidden> Wed, 29 Mar 2023 13:06:27 +0200

[Andreas Hasenack (ahasenack)]

Kinetic is EOL

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu14.2

---------------
grub2-unsigned (2.06-2ubuntu14.2) kinetic; urgency=medium

  * Cherry-pick more upstream memory patches (LP: #2004643)
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Julian Andres Klode <email address hidden> Mon, 20 Feb 2023 17:29:00 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.187.4~20.04.1

---------------
grub2-signed (1.187.4~20.04.1) focal; urgency=medium

  * Rebuild against grub2 2.06-2ubuntu14.2 (LP: #2004643)

 -- Julian Andres Klode <email address hidden> Wed, 29 Mar 2023 13:06:27 +0200