/usr/lib/ubuntu-advantage/timer.py:uaclient.exceptions.InvalidFileFormatError:read:loads:decode:raw_decode:/usr/lib/ubuntu-advantage/timer.py@176:run_jobs:read

[errors.ubuntu.com bug bridge]

[Original Description]

The Ubuntu Error Tracker has been receiving reports about a problem regarding ubuntu-advantage-tools. This problem was most recently seen with package version 27.13.3~22.10.1, the problem page at https://errors.ubuntu.com/problem/9271af8b26844a0fd6f6608db504837125211b97 contains more details, including versions of packages affected, stacktrace or traceback, and individual crash reports.
If you do not have access to the Ubuntu Error Tracker and are a software developer, you can request it at http://forms.canonical.com/reports/.

[Impact]

If the jobs-status.json file gets corrupted for any reason, users will see a degraded state on systemd, since we will not be able to run the timer job anymore.

[ Test Case ]

1) Launch a LXD container with an affected Ubuntu release
2) Install the Pro client version with the fix
3) Manually run the timer job, python3 /usr/lib/ubuntu-advantage/timer.py
4) Corrupt the /var/lib/ubuntu-advantage/jobs-status.json file (i.e. remove the last bracket on the file)
5) Run the job again and guarantee that it doesn't fail
6) Check /var/lib/ubuntu-advantage/jobs-status.json and verify that it is no longer corrupted

[ Regression potential ]

We deleting the corrupted file and running the job to recreate a correct one. Since we are just acting upon a corrupted file, we don't believe this change pose a significant regression potential

[ Discussion]

We are now checking if the jobs-status.json file can be read. If it can't, we delete the file and run the job normally to recreate it. We are also checking for error situations where we can't delete the file. In that situation we will log the error and not continue running the job

[Christian Ehrhardt  (paelzer)]

PR is up, will be fixed soon.

[Andreas Hasenack (ahasenack)]

Hello errors.ubuntu.com, or anyone else affected,

Accepted ubuntu-advantage-tools into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.4~22.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello errors.ubuntu.com, or anyone else affected,

Accepted ubuntu-advantage-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.4~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello errors.ubuntu.com, or anyone else affected,

Accepted ubuntu-advantage-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.4~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello errors.ubuntu.com, or anyone else affected,

Accepted ubuntu-advantage-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.4~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello errors.ubuntu.com, or anyone else affected,

Accepted ubuntu-advantage-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.4~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ubuntu-advantage-tools (27.13.4~18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-advantage-tools/unknown (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#ubuntu-advantage-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Andreas Hasenack (ahasenack)]

Hello errors.ubuntu.com, or anyone else affected,

Accepted ubuntu-advantage-tools into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.5~22.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello errors.ubuntu.com, or anyone else affected,

Accepted ubuntu-advantage-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.5~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello errors.ubuntu.com, or anyone else affected,

Accepted ubuntu-advantage-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.5~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello errors.ubuntu.com, or anyone else affected,

Accepted ubuntu-advantage-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.5~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello errors.ubuntu.com, or anyone else affected,

Accepted ubuntu-advantage-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.5~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ubuntu-advantage-tools (27.13.5~18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-advantage-tools/unknown (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#ubuntu-advantage-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Renan Rodrigo (renanrodrigo)]

Attached is a shell script that runs the steps described above, as well as the results of running this test with the proposed version for all supported releases.

I am marking this bug verification-done.

[Andreas Hasenack (ahasenack)]

I verified the attached logs and am satisfied that they show the executed planned test case, and that the results are correct.

Unfortunately the test case doesn't confirm the crash with the old version, but I accepted it as that and won't change the rules now. I checked the timestamps of the new run after the json file was corrupted, and am satisfied with the result.

The package built correctly in all architectures and Ubuntu releases it was meant for.

There are no DEP8 regressions. The ones that were there before, were fixed with a new run.

There is no SRU freeze ongoing at the moment.

There is a halted phasing on the previous update, which is being fixed by one or more of the other bugs addressed in this upload.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.13.5~22.10.1

---------------
ubuntu-advantage-tools (27.13.5~22.10.1) kinetic; urgency=medium

  * Backport new upstream release

ubuntu-advantage-tools (27.13.5~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.postinst:
    - fix version for cleaning the esm-apps stale unauthenticated files
      (LP: #2006765)

ubuntu-advantage-tools (27.13.4~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.postinst:
    - remove stale esm-apps unauthenticated caches (LP: #2004193)
  * apt-hook:
    - Change esm-apps advertisement message on apt upgrade to
      make it clearer that the service is providing more upgrades
      and not restricting user to only get updates if esm-apps
      is enabled (LP: #2006510)
  * contract:
    - make code aware that the effective date is not a required field
      in the machine-token.json file (LP: #2006351)
  * esm_cache
    - do not fail if we cannot extract information from
      /etc/os-release file (LP: #2006508)
  * security-status:
    - consider packages without a candidate as 'unknown' (LP: #2006049)
  * status:
    - treat null effective contract dates as unknown/expired
      (LP: #2004650)
  * timer:
    - recycle invalid jobs-status.json file if we detect it is
      corrupted (LP: #2006261)

 -- Renan Rodrigo <email address hidden> Thu, 09 Feb 2023 14:36:17 -0300

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for ubuntu-advantage-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.13.5~22.04.1

---------------
ubuntu-advantage-tools (27.13.5~22.04.1) jammy; urgency=medium

  * Backport new upstream release

ubuntu-advantage-tools (27.13.5~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.postinst:
    - fix version for cleaning the esm-apps stale unauthenticated files
      (LP: #2006765)

ubuntu-advantage-tools (27.13.4~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.postinst:
    - remove stale esm-apps unauthenticated caches (LP: #2004193)
  * apt-hook:
    - Change esm-apps advertisement message on apt upgrade to
      make it clearer that the service is providing more upgrades
      and not restricting user to only get updates if esm-apps
      is enabled (LP: #2006510)
  * contract:
    - make code aware that the effective date is not a required field
      in the machine-token.json file (LP: #2006351)
  * esm_cache
    - do not fail if we cannot extract information from
      /etc/os-release file (LP: #2006508)
  * security-status:
    - consider packages without a candidate as 'unknown' (LP: #2006049)
  * status:
    - treat null effective contract dates as unknown/expired
      (LP: #2004650)
  * timer:
    - recycle invalid jobs-status.json file if we detect it is
      corrupted (LP: #2006261)

 -- Renan Rodrigo <email address hidden> Thu, 09 Feb 2023 14:36:10 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.13.5~20.04.1

---------------
ubuntu-advantage-tools (27.13.5~20.04.1) focal; urgency=medium

  * Backport new upstream release

ubuntu-advantage-tools (27.13.5~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.postinst:
    - fix version for cleaning the esm-apps stale unauthenticated files
      (LP: #2006765)

ubuntu-advantage-tools (27.13.4~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.postinst:
    - remove stale esm-apps unauthenticated caches (LP: #2004193)
  * apt-hook:
    - Change esm-apps advertisement message on apt upgrade to
      make it clearer that the service is providing more upgrades
      and not restricting user to only get updates if esm-apps
      is enabled (LP: #2006510)
  * contract:
    - make code aware that the effective date is not a required field
      in the machine-token.json file (LP: #2006351)
  * esm_cache
    - do not fail if we cannot extract information from
      /etc/os-release file (LP: #2006508)
  * security-status:
    - consider packages without a candidate as 'unknown' (LP: #2006049)
  * status:
    - treat null effective contract dates as unknown/expired
      (LP: #2004650)
  * timer:
    - recycle invalid jobs-status.json file if we detect it is
      corrupted (LP: #2006261)

 -- Renan Rodrigo <email address hidden> Thu, 09 Feb 2023 14:36:05 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.13.5~18.04.1

---------------
ubuntu-advantage-tools (27.13.5~18.04.1) bionic; urgency=medium

  * Backport new upstream release

ubuntu-advantage-tools (27.13.5~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.postinst:
    - fix version for cleaning the esm-apps stale unauthenticated files
      (LP: #2006765)

ubuntu-advantage-tools (27.13.4~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.postinst:
    - remove stale esm-apps unauthenticated caches (LP: #2004193)
  * apt-hook:
    - Change esm-apps advertisement message on apt upgrade to
      make it clearer that the service is providing more upgrades
      and not restricting user to only get updates if esm-apps
      is enabled (LP: #2006510)
  * contract:
    - make code aware that the effective date is not a required field
      in the machine-token.json file (LP: #2006351)
  * esm_cache
    - do not fail if we cannot extract information from
      /etc/os-release file (LP: #2006508)
  * security-status:
    - consider packages without a candidate as 'unknown' (LP: #2006049)
  * status:
    - treat null effective contract dates as unknown/expired
      (LP: #2004650)
  * timer:
    - recycle invalid jobs-status.json file if we detect it is
      corrupted (LP: #2006261)

 -- Renan Rodrigo <email address hidden> Thu, 09 Feb 2023 14:35:59 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.13.5~16.04.1

---------------
ubuntu-advantage-tools (27.13.5~16.04.1) xenial; urgency=medium

  * Backport new upstream release

ubuntu-advantage-tools (27.13.5~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.postinst:
    - fix version for cleaning the esm-apps stale unauthenticated files
      (LP: #2006765)

ubuntu-advantage-tools (27.13.4~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.postinst:
    - remove stale esm-apps unauthenticated caches (LP: #2004193)
  * apt-hook:
    - Change esm-apps advertisement message on apt upgrade to
      make it clearer that the service is providing more upgrades
      and not restricting user to only get updates if esm-apps
      is enabled (LP: #2006510)
  * contract:
    - make code aware that the effective date is not a required field
      in the machine-token.json file (LP: #2006351)
  * esm_cache
    - do not fail if we cannot extract information from
      /etc/os-release file (LP: #2006508)
  * security-status:
    - consider packages without a candidate as 'unknown' (LP: #2006049)
  * status:
    - treat null effective contract dates as unknown/expired
      (LP: #2004650)
  * timer:
    - recycle invalid jobs-status.json file if we detect it is
      corrupted (LP: #2006261)

 -- Renan Rodrigo <email address hidden> Thu, 09 Feb 2023 14:35:33 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.13.5~23.04.1

---------------
ubuntu-advantage-tools (27.13.5~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.postinst:
    - fix version for cleaning the esm-apps stale unauthenticated files
      (LP: #2006765)

 -- Renan Rodrigo <email address hidden> Thu, 09 Feb 2023 14:20:47 -0300