Update (python-)zhmcclient to latest release

[Frank Heimes]

[Impact]

In order to remotely power-cycle IBM z14 GA2 (or newer) or IBM LinuxONE III (or newer) machines/LPARs from within a MAAS controller, the python3-zhmcclient libraray (>= v0.29) is needed, to provide all the functions required for controlling the HMC.

[Test Plan]
 * setup a MAAS controller inside a LPAR
 * use that controller to deploy a new Ubuntu Server instance in an additional LPAR
 * use that controller to deploy a new Ubuntu Server instance inside a KVM host (inside another LPAR)
 * Power-cycle the LPAR instance
 * Power-cycle the KVM instance
 * on top and for regression testing the example (as shipped by the zhmcclient) can be ran as well (as far as possible on a non-DPM HMC setup):
https://github.com/zhmcclient/python-zhmcclient/tree/master/examples

Also get confirmation from the MAAS team that things are working as expected.

[Where problems could occur]

This library is s390x only and zhmcclient isn’t widely used yet, it has very limited exposure within the s390 environment. If we would be hit by a regression, we would not be able to power-cycle or deploy any LPAR (or KVM host inside LPAR) machine, using a Focal-based MAAS controller.
This is basically also the status quo, as the current version in Focal (v0.22) does not allow for that functionality either.

Reviewing the changes/commits starting from 0.28 to 0.31 (68 + 37 + 27 + 15 commits) let me think that these are largely unintrusive, just a few new features were added (like DPM Capacity Groups), the docs, examples and lots of meta updated, error and exception handling refined <important>, and last but not least several important bugs fixed (for installation, .find for objects <again important>, certificates, missing argument in method) - some of them were identified during the development of DPM/LPAR support in MAAS earlier this year (for example LP#1916475).

Therefore, and because the zhmcclient is not wide-spread yet (since MAAS v3.0 is the first version that can make use of it and was just made available on June the 22nd), it's for accessing and managing Z systems only via the HMC API, which not many organizations allow (in general - for security reasons) and because v0.31 was the version that was already used for development _nd final product testing (these days as a git clone), makes me rating the regression risk as low.

[Other Info]
* https://maas.io/docs/snap/3.0/ui/power-management#heading--configure-use-ibm-z
* As this is only used on MAAS controllers, running LTS releases, the backport is not needed for Hirsute 21.04
* The new package dependencies are fulfilled in Focal:
  - python3-requests >= 2.20.1 (focal contains 2.22.0-2ubuntu1 in main)
  - python3-decorator >= 4.0.11 (focal contains 4.4.2-0ubuntu1 in main)

=== Original description ===

While working on MAAS DPM/LPAR it turned out that a newer python-zhmcclient (package zhmcclient) is needed to get certain functions work, like especially storage group support.

The latest is package is v0.27 on hirsute and v0.22 on focal.

v0.28 is currently the latest release, but 0.29 is coming shortly.
https://github.com/zhmcclient/python-zhmcclient/issues/731

Hence an updated package (ideally to 0.29) is needed for focal (knowing that this will require a good justification: https://wiki.ubuntu.com/StableReleaseUpdates).
Please notice that MAAS was the only reason why the zhmcclient package was created at all.

The updated package is especially needed for focal, since focal is the recommended base distro for MAAS DPM/LPAR, hence this would require a special SRU (for this package version bump).

[Frank Heimes (fheimes)]

Setting to incomplete until 0.29 is officially out.

[Frank Heimes (fheimes)]

v0.29 was just released:
https://github.com/zhmcclient/python-zhmcclient/releases
https://github.com/zhmcclient/python-zhmcclient/releases/tag/0.29.0

[Frank Heimes (fheimes)]

v0.30.0 was just released
https://github.com/zhmcclient/python-zhmcclient/releases

[Matthieu Clemenceau (mclemenceau)]

v0.31.0 is available on github, I will use that one and provide a debdiff for sponsoring once it is finished building successfully.

[Matthieu Clemenceau (mclemenceau)]

packaged and built version 0.31.0 for impish on my ppa:mclemenceau/distro-work

It is available for your test and verification. I will look for a sponsor now unless a problem is found.

[Lukas Märdian (slyon)]

LGTM, thanks for the debdiff!

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "diff from 0.27.0 to 0.31.0" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Launchpad Janitor (janitor)]

This bug was fixed in the package zhmcclient - 0.31.0-0ubuntu3

---------------
zhmcclient (0.31.0-0ubuntu3) impish; urgency=medium

  * d/control: fix armhf autopkgtest by adding python3-requests dependency

 -- Lukas Märdian <email address hidden> Mon, 21 Jun 2021 18:16:51 +0200

[Robie Basak (racb)]

It sounds like this type of update would qualify under the same principle that MAAS (were it still in the archive) adds new features to stable releases.

However, please see the requirements at https://wiki.ubuntu.com/StableReleaseUpdates#Other_safe_cases. Specifically:

I see no upload for Hirsute, but this is a requirement for new features in an LTS so that there isn't a regression and lack of ability to receive updates if users upgrade from the updated Focal to Hirsute. Please provide an upload for Hirsute.

It must be demonstrated that the changes you propose are unintrusive, have a minimal regression potential, and have been tested properly. You could cherry-pick the required features, or you could update to a newer upstream release wholesale as you're proposing. Please provide an analysis and explanation as to why you think the latter approach is better in this case with respect to these requirements, and what you're doing to meet them. For example, I see nothing in your Test Plan that seeks to ensure that unrelated behaviour in this package has not regressed; nor do I see any analysis of the upstream changes to ensure that functional/behavioural changes have not been made. And what about other non-MAAS use cases for this package that users might have established? How do you intend to mitigate regressing such use cases?

[Frank Heimes (fheimes)]

I think the main reason why hirsute is missing here, is because the ticket was opened when groovy was still in development, but since the package was now updated and released for Impish first (as current development release), hirsute became needed on top (groovy EOLed already).
I agree that this will ensure that nobody runs into regressions, even if I strongly assume that no one will seriously use MAAS on a non-LTS version in production, especially on s390x. And MAAS v3.0 (released June 22nd) is the first official version that supports LPARs and requires the zhmcclient for this. However, one may also use the zhmcclient w/o MAAS - so I'll add hirsute as affected.

The initial zhmcclient version was added to the archive in 2019 due to prototype work at that time (maybe it should have been added to a PPA rather than directly to the archive - anyway).
After this early coding was done, the work couldn't be continued for several reasons, but was then picked up again earlier this year - meanwhile the zhmcclient became a bit outdated (and was never used by MAAS). But with new hardware (especially its HMCs in DPM mode) the zhmcclient evolved over time upstream.

Reviewing the changes/commits starting from 0.28 to 0.31 (68 + 37 + 27 + 15 commits) let me think that they are largely unintrusive, just a few new features were added (like DPM Capacity Groups), the docs, examples and lots of meta updated, error and exception handling refined <important>, and last but not least several important bugs fixed (for installation, .find for objects <again important>, certificates, missing argument in method) - some of them were identified during the development of DPM/LPAR support in MAAS earlier this year (for example LP#1916475).

Therefore, and because the zhmcclient is not wide-spread yet (since MAAS v3.0 can make use of it just since June the 22nd), it's for accessing and managing Z systems only via the HMC API, what not many organizations allow (in general) for security reasons and because v0.31 was the version that was already used for development _and_ final product testing (these days as a git clone), makes me rating the regression risk as low.

On top of the testing that was already done, the zhmcclient comes with several examples (https://github.com/zhmcclient/python-zhmcclient/tree/master/examples), which would partly also work with non-DPM HMCs, like our z13 - hence could be (as far as possible) ran (by me), too.
(I'll update the template in the bug description.)

[Lukas Märdian (slyon)]

Thank you Robie for your thorough review and detailed comments & references.

Thank you Frank for improving upon the original test plan, your volunteering to help with the verification on real hardware and for your review and comments regarding the changes.

I also went through the changelog (https://github.com/zhmcclient/python-zhmcclient/blob/9270ac306586b235751d432cf4a7bd4cfac84c0d/docs/changes.rst) since 0.22 (i.e. Focal+) that lists a few "incompatible changes" that we need to be aware of and watch out for:

0.24.0
* Operations that resulted in HTTP status 403, reason 1 ("The user under which the API request was authenticated does not have the required authority to perform the requested action.") so far raised ServerAuthError. However, that exception does not represent that situation properly, because the login user is actually properly authenticated. The handling of this case was changed to now raise HTTPError instead of ServerAuthError. This change is only incompatible to users of the zhmcclient API who have code handling this exception specifically.
=> Tiny API change in the error handling.

0.28.0
* Removed the installed scripts cpcdata and cpcinfo and added them as cpcdata.py and cpcinfo.py to the examples folder.
=> Moving of (example) scripts to a different folder, but they are still available.

0.31.0
* Method 'NotificationReceiver.notifications()' now raises JMS errors returned by the HMC as a new exception 'NotificationJMSError'. JSON parse errors are now raised as a new exception 'NotificationParseError'. Both new exceptions are based on a new base exception 'NotificationError'. (issue #770)
=> Extension of error handling API.

* By default, the zhmcclient now verifies the HMC certificate using the CA certificates in the Python 'certifi' package. This can be controlled with a new 'verify_cert' init parameter to the 'zhmcclient.Session' class. (issue #779)
=> Checking of HMC certificates by default, that can easily be disabled using the 'verify_cert' parameter.

* The 'properties' attribute of the resource classes (e.g. 'Partition') now is an immutable 'DictView' object in order to enforce the stated rule that that callers must not modify the properties dictionary. If your code used to make such modifications nevertheless, it will now get a 'TypeError' or 'AttributeError' exception, dependent on the nature of the modification.
=> This change was reverted in debian/patches/0001-Revert-Changed-properties-attribute-to-be-immutable.patch for compatibility reasons (i.e. missing python immutable-views dependency).

Summary:
- 2 tiny changes in the error handling API
- 1 change of example script installation location
- 1 changed default, that can be controlled via 'verify_cert' parameter
- 1 reverted change that does not affect our users

As Frank stated, the primary user of this package is MAAS v3.0, that was developed and tested with this newer version in mind already. Apart from that the use case of zhmc-client is very small with few (if any at all) external users. Therefore we do not expect to break anybody by introducing those changes.

I've uploaded the version bump to Hirsute as well, albeit we expect very little (if any) usage...

[Łukasz Zemczak (sil2100)]

Thank you Robie, Lukas and Frank for working on this SRU. I think with proper analysis performed in #12 and additional context in #11, I am happy to get this accepted for both hirsute and focal. Please be sure to perform proper verification.

[Łukasz Zemczak (sil2100)]

Hello Frank, or anyone else affected,

Accepted zhmcclient into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zhmcclient/0.31.0-0ubuntu3~21.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Frank, or anyone else affected,

Accepted zhmcclient into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zhmcclient/0.31.0-0ubuntu3~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Frank Heimes (fheimes)]

Just fyi:
I started to have a look at this validation but bumped into some certificate validation issues and needed to put things aside due to the upcoming 21.10 FF.
I'll come back to this after FF - means at the end of this week ...

[Frank Heimes (fheimes)]

Sorry for the delay, but I was focusing on the 21.10 FF and the initial 20.04.3 RC testing in the last couple of days - now had the time to come back to this.

I now ran a large subset of the Examples and test o that are shipped with the zhmclient.
(the results are attached, but I needed to anonymize them a bit - credentials, addresses, names, etc.)
I ran them on focal and hirsute (with latest updates each) and the results were the same.

I had to install some additional python packages and libraries, but NOT due to missing dependencies, but because the examples required them - not the zhmcclient API itself.
And I had to disable the certificate checking (due to the certificate of our HMC, there is a corresponding troubleshooting section in the zhmcclient doc).

The storage group examples are expected to fail, since storage groups are only available on DPM system, and ours is a non-DPM (so these are false positives).

With that I'm updating the tags to verification done.

[Brian Murray (brian-murray)]

The test plan indicates that confirmation should be received from the MAAS team that things are working as expected. Has this happened?

[Frank Heimes (fheimes)]

@Brian, no, and this is unfortunately currently not possible, because:
- we (incl. the MAAS team) do not have access to the remote z14 development system anymore (that we got from IBM)
- and we (incl. MAAS team) are not able to do more testing on our own system, zhmcclient in combination with MAAS, since MAAS (for LPARs) requires a Z system where the HMC runs in DPM mode, and our HMC is running in traditional mode (so not running in DPM mode) - and switching the mode is destructive.
(Well, we may get a new system later this year that will be in DPM mode - but tbh. I imploringly hope that we don't need to wait for this.)
From my point-of-view we did the most testing that is possible in the current situation and with the resources available.

[Frank Heimes (fheimes)]

Well, changing status back to Fix Committed to get the process going again (hoping that my previous comment was sufficient).

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for zhmcclient has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package zhmcclient - 0.31.0-0ubuntu3~20.04.1

---------------
zhmcclient (0.31.0-0ubuntu3~20.04.1) focal; urgency=medium

  * Backporting new upstream release. (LP: #1919451)

 -- Lukas Märdian <email address hidden> Thu, 22 Jul 2021 15:11:31 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package zhmcclient - 0.31.0-0ubuntu3~21.04.1

---------------
zhmcclient (0.31.0-0ubuntu3~21.04.1) hirsute; urgency=medium

  * Backporting new upstream release. (LP: #1919451)

 -- Lukas Märdian <email address hidden> Thu, 22 Jul 2021 15:11:31 +0200

[Frank Heimes (fheimes)]

Thx Brian!