[SRU] Calamares 3.2 Series for 22.04 LTS

[Simon Quigley]

[Impact]

 * The version of Calamares currently present in Ubuntu has quite a few
   bugs, which could potentially impact the user in a negative way during
   the initial installation process of Ubuntu flavours that make use of
   the Calamares installer.

 * One such bug is Calamares upstream bug #1817. Attempting to install an
   OS using Calamares and LVM results in an almost immediate installation
   failure. Testing has revealed that this bug affects both Lubuntu and
   Ubuntu Studio 22.04.

 * This SRU updates the version of Calamares to the latest stable release,
   fixing this bug and a number of other bugs which can be seen at
   https://github.com/calamares/calamares/blob/calamares/CHANGES-3.2 (the
   current version of Calamares is 3.2.41, so everything up from there are
   the changes that will be applied). Additionally, a flaw in the
   debian/rules file of the packaging was repaired, and an obsolete
   manpage was removed to avoid potential user confusion. All the details
   of these changes are documented in the package changelog.

[Test Plan]

 * Using a Lubuntu or Ubuntu Studio 22.04 ISO, follow the exact steps
   mentioned in the upstream bug report here:
   https://github.com/calamares/calamares/issues/1817

 * (Note: There's a typo in the line that reads `sudo vgcreate myvg
   /dev/sda1k` - the "k" at the end doesn't belong. The line should read
   `sudo vgcreate myvg /dev/sda1` - I verified this in my testing.)

 * The installation will fail. Repeated attempts to install will result in
   repeated failures (I tested by attempting to install in this
   configuration twice).

 * Install Calamares 3.2.60 into the live ISO using the updated package.

 * Repeat the steps to reproduce the bug.

 * The installation will succeed, and you will be able to boot into the OS
   after installation.

[Where problems could occur]

 * If a bug was introduced in this version of Calamares, or if a
   configuration change is needed in calamares-settings-ubuntu, it could
   result in installations failing, being unable to boot after a
   "successful" installation, or being bootable but too damaged to use.
   Due to the highly obvious nature of such issues, it will be easy to
   catch it if something like this happens.

[Other Info]

 * The Lubuntu team has an intensive test suite prepared for detecting
   such problems, and the new Calamares package will be put through the
   full test suite, possibly multiple times, to ensure that it is stable
   and ready for production use.

 * As this change will affect both Lubuntu and Ubuntu Studio, we intend to
   run both flavors through the full test suite to minimize the potential
   for unforseen breakage.

 * The full Calamares test suite used by Lubuntu can be located here:
   https://phab.lubuntu.me/w/release-team/testing-checklist/ The way the
   checklist works is explained here:
   https://discourse.lubuntu.me/t/testing-checklist-understanding-the-testcases/2743

---

Original bug report:

(This is starting as a request for comment from the Ubuntu Stable Release Update Team and will eventually be edited to be an SRU bug.)

In Lubuntu and Ubuntu Studio 22.04 LTS, we dropped the ball by including the same version of Calamares as we did in 21.10. Upstream considers 3.2 to be an "LTS" series, so we would like to follow the upstream release cycle to deliver the latest 3.2 point release to our users.

Here is a specific link to the code changes included between the two versions: https://github.com/calamares/calamares/compare/v3.2.41.1...v3.2.60

It's quite a lengthy update. As far as I can tell, we have three options:
 A) Update to the latest upstream milestone in 22.04. Go through a thorough test suite, which will be outlined in the SRU bug report, and continue to follow upstream point releases. The benefit of this route is that 3.2.60 has many more eyes on it than 3.2.41.1. Bugs are more easily fixed, and problems are likely to be experienced between multiple distributions, not just us.
 B) We patch 3.2.41.1 with just translations and all bugfixes, only. This will take quite a bit of work to sort through what qualifies as a bugfix, what can be backported, and the rationale for each bugfix. Would not recommend but perhaps this is the most "standard" option.
 C) Only backport specific high-profile bugfixes to 3.2.41.1.

I'd like to just go with A, and speaking on behalf of Lubuntu I think that would be the most sustainable option. I'm *very* rusty on this entire process, so please let me know what other information you may need to help in processing this request.

Thank you.

[Erich Eickmeyer (eeickmeyer)]

*with Ubuntu Studio hat on*

Honestly, option A seems very sane. I'm against going with dozens or hundreds of patches to accomplish the same goal, and getting Calamares into better shape is definitely a better option.

I did bring up with the Lubuntu team that Calamares did seem to be needing an update, but this was a bit beyond feature freeze, and nobody (including myself) felt comfortable at the time with uploading a new version that close to beta freeze.

However, we have seen what can happen if the installer does not have LTS maintainability and stability, and this does have a detrimental effect on the product. With that, I agree with everything Simon has written and the justification seems sound.

Unfortunately, we have just a month as 22.04.1 is approaching quickly. I have had several recent experiences with SRUs, so I can assist Simon with the SRU if need-be.

[Robie Basak (racb)]

calamares doesn't appear to have *ever* received any updates (neither to -updates nor -security) in the history of the package in Ubuntu. So what has changed that following an upstream "LTS" release will make a difference now?

If you weren't able to update it before Jammy's release according to the regular release schedule, what gives you confidence that you have the capacity to update it now, and will be able to give it the additional care required for a post-release update?

[Simon Quigley (tsimonq2)]

Dear Robie,

Thank you for your consideration on this update and the rationale behind it. You make a valid point, Calamares has never received a Stable Release Update, and we have not made such a request before. If this were to be a typical application, we would agree with your assessment that this could be inappropriate for users. Here is why this is an exceptional case:
 - Calamares is simply a system installer. It gets automatically removed once the system is installed, and users typically will not update the system prior to installing it, usually after.
 - Lubuntu has a thorough test suite covering all typical ways a user would install a system, which Ubuntu Studio also uses. This would be exponentially above and beyond for manual testing on a normal SRU: https://phab.lubuntu.me/w/release-team/testing-checklist/
 - Both official flavors of Ubuntu that utilize this application in any official capacity are giving this a green light. We are willing and prepared to test this update thoroughly.

To answer the question of "why hasn't this been done before?," Calamares was a fresh addition to Lubuntu in 20.04 and changes were happening too quickly. We are confident that upstream has stabilized that branch and tests it enough that we should not run into any major, showstopper issues. We believe that this will be an upgrade in stability and long-term maintainability, not a downgrade with breaking changes that will negatively affect the user experience.

If you have any further questions, please feel free to ask.

Thanks again,
Simon Quigley and Erich Eickmeyer for the Lubuntu and Ubuntu Studio Teams, respectively

[Aaron Rainbolt (arraybolt3)]

Coming from an end-user standpoint, I believe that updating Calamares in Ubuntu 22.04 will have a direct, positive effect on our user base.

The installer is one of the first things (if not the first thing) a user notices about a distro, and having installed many distros, you can usually tell a lot about how usable and stable a distro is based on how intuitive and stable the installer is. If I have a bad first experience with an installer, I expect the distro as a whole to be difficult to use and potentially buggy. Likewise, if the installer works great, I expect the distro to be well-developed and usable. So far, this has proven true with virtually every distro I've tried.

As Simon and Erich pointed out above, the installer is only present in a live session, and is removed after installing the system. As a result, it is unlikely to have any effect on the out-of-box experience for a user after installation, so long as the installer itself doesn't cause any problems to the system during installation. I would expect this update to improve the installer's functionality, making it deliver a more stable and usable distro.

Since the installer will either improve or at least not interfere with the out-of-box experience, it's my opinion that this is low-hanging fruit. It's a change we can safely make, that may have a highly positive effect on some of our users for not one, but two flavours of Ubuntu.

As part of the testing process for the updated Calamares version, I have a dedicated testing laptop that will allow me to assist in the testing process, helping to ensure that an updated version of Calamares won't result in a degradation of quality for our end user base. In addition to the testing done by our usual testing team, I will be personally looking for bugs and helping to evaluate Calamares 3.2.60's suitability for being SRU'd into Ubuntu 22.04.

I hope this message finds you well.

Reviewed-by: Simon Quigley <email address hidden>
Reviewed-by: Erich Eickmeyer <email address hidden>

[Brian Murray (brian-murray)]

I'm okay with the installer being updated in principle but I'd like know if there are any changes in calamares that would result in a system installed from 22.04 media being substantially different than a system installed from 22.04.1 media. (For example are there package management changes or configuration file changes?)

[Chris Guiver (guiverc)]

Giving another end-user view, and from my own QA-testing of Lubuntu releases found on our checklist, I'm unaware of any differences.

> For example are there package management changes or configuration file changes?

I'm not a developer, but my name is found in the checklist (not as often as Leó's) and my testing post-install is identical for all releases. I'll let the devs speak to Brian's question.

We have a 'install using existing partition' which is a no-format install, so end-user manually-installed packages get re-installed, user files are untouched, which would allow a user to re-install without any real consequence & i test it almost weekly (2022-07-14 my last test using jammy/22.04.1 daily). I believe I could use that install to change a 22.04 install to kinetic & back (to 22.04 using currently dailies & the different calamares versions) without issue. I can test that if helpful.

The box I'm using now still has a 21.10/impish install, impish is now EOL & that install will become my kinetic install on this box using the 'install using existing partition'. I could switch it to 'kinetic' (newer calamares), check it out, then re-install older 20.04 using current dailies & older calamares, and I'd expect the checklist install to pass.. ie.
- my data files remain (due to lack of format in install)
- my manually installed packages will get re-installed; I use `clementine` to play music which isn't installed by default with Lubuntu, & expect that to continue playing my playlist post-install using this install method as I talked about in the 'understanding the testcases' doc.

Alas that doesn't meet Brian's concern as I read it; ie. config files for calamares will from be different releases if I performed this test, but I'd expect the user-viewpoint to be identical, being what I'm trying to say.

If some testing would be helpful; I'll do it.

FYI: Description of our testing checklist can be found here https://discourse.lubuntu.me/t/testing-checklist-understanding-the-testcases/2743

[Łukasz Zemczak (sil2100)]

Seeing that there is a group of dedicated individuals willing to work on this and getting it thoroughly tested, I think I am willing to give it a green light. 22.04.1 is right around the corner, but we could still get the new version into -proposed for testing and then, in case we deem it to risky, simply dropping it from -proposed without moving to -updates.

I agree with Brian's points - when preparing the calamares SRU please be sure to double check the changes to make sure the resulting installation is the same as with the previous version (or all the other behavioral differences documented and understood). But from my POV, I give a green light to at least get the new version into -proposed.

[Aaron Rainbolt (arraybolt3)]

Calamares 3.2.41 > 3.2.60 debdiff.

[Simon Quigley (tsimonq2)]

To answer the concern regarding whether the installed system will be changed, the short answer is =~ no.

Our configuration stored in calamares-settings-ubuntu dictates precisely how the installer should function. As noted in the SRU paperwork, we are unaware of any configuration changes required to make this update work correctly.

Many of the upstream changes are disguised as new features but, in reality, address bugs. As Calamares simply handles partitioning, unpacking of the squashfs, and basic initial setup of the system, our QA processes are likely to detect if there are any changes in the installed system. As this release is used in many distributions (which is the appeal of using Calamares), any such changes should have been submitted upstream by this point.

I hope this answers your questions. Thanks.

[Łukasz Zemczak (sil2100)]

Thank you for shedding some light on the topic, Simon! And thank you for uploading the new version of calamares. It's a bit on me for not engaging in this topic earlier, and now we're in this weird situation where we're trying to land things very late for .1.

That being said, since both Lubuntu and Studio leads, so the only flavors currently using calamares, are +1 on having this for .1, I am willing to get this SRU landed and trying to ship it with .1. It's a risky move, so I'd like to set some ground rules here:
 * I will review and (hopefully) accept the calamares packages today, making them build in -proposed
 * I'll kick a manual build using -proposed of the affected flavors and ping for testing here and on IRC. I would appreciate a quick look by flavors to assess if there is nothing bad with the update - would be welcome if validation flags would be switched here
 * If all looks good, we promptly promote the calamares packages to jammy-updates tomorrow (on a Friday, oh no!). This should be fine as this is basically only an installer package, so no worries about breaking existing users
 * But since the -proposed validation period is so short, some of the real-life-validation will most probably have to happen with the RCs for .1. So the rule is: if we find a critical regression with the new calamares, we revert back to the old package and rebuild, since that's the only 'safe' way to proceed. Other options are delaying the .1 release (which I don't think is feasible) or skipping flavors (...I don't think anyone wants that either!)

Hope this plan is okay. I'll start working on it in a moment.

[Łukasz Zemczak (sil2100)]

Hello Simon, or anyone else affected,

Accepted calamares into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/calamares/3.2.60-0ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Aaron Rainbolt (arraybolt3)]

We are hoping to also get in the fix for the BTRFS installation bug in before the release of 22.04.1 - the bug number is #1966774.

[Erich Eickmeyer (eeickmeyer)]

I don't see any real installation difference whatsoever and everything is working exactly as it did before, perhaps better. If anything, I see substantial performance improvements, which I consider a win. So far, this looks like a "go" from Ubuntu Studio. I'm stopping short of marking as "verification-done" and leaving that up to the rest of the combined Lubuntu/Ubuntu Studio team to finish with testing.

[Chris Guiver (guiverc)]

I've performed some installs

full disk (uEFI) to
- sony vaio svp112a1cw (i5-9400u, 4gb, intel haswell-ULT)
install alongside & replace partition (BIOS) on
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
and a re-install test (install using existing partition in lubuntu testing checklist; ie. no format & check to see no data is lost & manually installed packages get re-installed)
- dell [optiplex] 780 (c2q-q9400, 8gb, amd/ati cedar radeon hd 5000/6000/7350/8350)

all of which were perfect.

I haven't noticed any differences in speed, however did notice https://bugs.launchpad.net/ubuntu/+source/calamares/+bug/1904266 is fixed with this version :)

I'll continue testing, but Leó expected to perform some Secure-uEFI testing soon

lubot> [telegram] <Leokolb> I will be testing uefi+secure boot on bare metal boxes in about 2 hours time

[Chris Guiver (guiverc)]

The lubuntu checklist is now completed (excluding BTRFS which needs a different package change/SRU thus was skipped).

Some more testing by me, and many by Leó (leokolb)

Lubuntu testing checklist can be see here - https://phab.lubuntu.me/w/release-team/testing-checklist/ (second set of tables is the PROPOSED version of calamares; ie. 3.2.60-0ubuntu0.1 found on ISO dated 2022-07-28.1

Results/notes of installs can be found here - http://iso.qa.ubuntu.com/qatracker/milestones/429/builds/254612/testcases/1701/results/

I've also spoken to Dan/kc2bez (#lubuntu-devel)

<lubot> [telegram] <kc2bez> OK. I don't think we need anything beyond the testing we have. I am good if you mark it verified

so will mark it done/verified

[Łukasz Zemczak (sil2100)]

Thank you for all the testing! Let me release those and get the machinery moving.

[Launchpad Janitor (janitor)]

This bug was fixed in the package calamares - 3.2.60-0ubuntu0.1

---------------
calamares (3.2.60-0ubuntu0.1) jammy; urgency=medium

  * Updated Calamares to version 3.2.60 for SRU (LP: #1980180).
  * Updated copyright file.
  * Removed obsolete manpage for Calamares.
    - Reasoning: Calamares provides a man page upstream.
  * Imported upstream's signing key into debian/upstream/signing-key.asc.
  * Updated debian/watch to version 4, adding automatic GPG verification
    in the process.
  * Fixed a typo and a variable assignment bug in debian/rules.
    - Details: Originally, the rules file exported the
      DEB_BUILD_MAINT_OPTIONS variable twice, once to enable hardening, and
      once to help make build paths reproducible. I believe this would
      result in hardening being disabled due to clobbering. Additionally,
      there was a typo in the second variable export (a missing "=") that
      caused sbuild to show me a dh warning. Both problems are now resolved -
      the variable is only exported once, with both options set, and the
      typo is fixed.

 -- Aaron Rainbolt <email address hidden> Thu, 21 Jul 2022 13:09:04 -0500

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for calamares has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.