Unrecoverable resyncs if DB is restored from backup

Bug #1616116 reported by Andreas Hasenack on 2016-08-23
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Landscape Client
Medium
Simon Poirier
landscape-client (Ubuntu)
Medium
Simon Poirier
Trusty
Medium
Simon Poirier
Xenial
Medium
Simon Poirier
Bionic
Medium
Simon Poirier
Cosmic
Medium
Simon Poirier

Bug Description

[Impact]

 * In some cases generally involving backups/restore, client would get
   inconsistent package data and keep that data upon resync, thus getting
   stuck in a resync loop. This usually gets noticed through the stress
   it adds on the server and though logs which grow abnormally.

[Test Case]

  * deploy landscape-server quickstart from ppa:landscape/18.03
  * register client against server. wait for package info
  * pg_dumpall
  * add a repo and wait for new package to show on in landscape.
  * restore the postgres backup.
  * run ./scripts/hash_id_databases.sh from the server to complete
    the restore.
  * trigger a package install from the new repo to create some package
    info to update
  * client should resync once then will re-fetch hash on the next run.

[Regression Potential]

 * Modified code is used only during resync operations and removes
   cached data when the client state is deemed inconsistent.

 * In the unlikely event the code is called outside of the expected
   cases, the end result would be limited to the package-monitor
   having to re-download the hash-id databases, which shouldn't
   cause issues as that is the behaviour at client registration.

[Other Info]

 * Other cases than server restores have been noticed to generate the
   bug but they are far less common.

[Original description]

Landscape with live clients cannot handle a DB restore to a point in the past.

The scenario is Landscape running as usual, with live clients, restoring to a DB backup taken in the past. After the service ir brought up again with this data, clients will start resyncing and becoming wedged with all sorts of tracebacks on the message server.

I left such a scenario running overnight, hoping that eventually the resyncs would settle down and everything recover, but that didn't happen. The resyncs continued, in the packages scope.

An interesting one in particular was this:
Aug 22 21:46:26 message-server-2 ERR Error handling message 'operation-result' for computer 104: {'status': 6, 'timestamp': 1471901963, 'result-text': u'Mon Aug 22 21:39:23 UTC 2016\n', 'api': '3.3', 'operation-id': 533, 'type': 'operation-result'}#012Traceback (most recent call last):#012 File "/opt/canonical/landscape/canonical/landscape/message/apis.py", line 358, in _process_messages#012 self.handle(message["type"], message)#012 File "/opt/canonical/landscape/canonical/message/api.py", line 66, in handle#012 return handler(type, body)#012 File "/opt/canonical/landscape/canonical/message/handler.py", line 30, in __call__#012 return function(self.message_api, type, body)#012 File "/opt/canonical/landscape/canonical/lib/arguments.py", line 79, in replacement#012 return original(*new_args, **new_kwargs)#012 File "/opt/canonical/landscape/canonical/landscape/message/handlers/activity.py", line 32, in handle_activity_result#012 activity.succeed(code=result_code, text=result_text)#012AttributeError: 'NoneType' object has no attribute 'succeed'

That was about an activity that had been delivered already, but did not exist in the restored DB.

Related branches

description: updated
Simon Poirier (simpoir) wrote :

Hit a similar issue. We have 2 known workarounds:

- purge landscape-client and re-register
- service landscape-client stop && rm -fr /var/lib/landscape/client/package/hash-id/* /var/lib/landscape/client/package/database /var/lib/landscape/client/messages/* && service landscape-client restart; then wait up to 6h for package data to update

Simon Poirier (simpoir) on 2018-09-14
Changed in landscape:
status: New → In Progress
tags: added: lds-squad
Simon Poirier (simpoir) on 2018-09-17
Changed in landscape:
assignee: nobody → Simon Poirier (simpoir)
Simon Poirier (simpoir) on 2018-10-03
Changed in landscape:
status: In Progress → Fix Committed
Simon Poirier (simpoir) on 2018-11-14
description: updated
information type: Proprietary → Public
affects: landscape → landscape-client
Simon Poirier (simpoir) on 2018-11-14
Changed in landscape-client:
importance: Undecided → Medium
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package landscape-client - 18.01-0ubuntu6

---------------
landscape-client (18.01-0ubuntu6) disco; urgency=medium

  * debian/patches/nutanix-kvm.patch: Update vm_info.py to include Nutanix
    hypervisor.
  * Fixes for release-upgrade (LP: #1699179).
    - debian/patches/release-upgrade-success.patch: Enable landscape-client to
      survive trusty upgrade. (LP: #1670291)
    - debian/patches/post-upgrade-reboot.patch: Force reboot operation in
      case systemd fails. (LP: #1670291)
  * debian/patches/unicode-tags-script.patch: Permit environments
    containing unicode chars for script execution. (LP: #1765518)
  * debian/patches/1616116-resync-loop.patch:
    Clear hash id database on package resync. (LP: #1616116)

 -- Simon Poirier <email address hidden> Tue, 27 Nov 2018 09:24:22 -0500

Changed in landscape-client (Ubuntu):
status: New → Fix Released
Changed in landscape-client (Ubuntu Bionic):
status: New → In Progress
Changed in landscape-client (Ubuntu Cosmic):
status: New → In Progress
Changed in landscape-client (Ubuntu Bionic):
importance: Undecided → Medium
Changed in landscape-client (Ubuntu):
importance: Undecided → Medium
Changed in landscape-client (Ubuntu Cosmic):
importance: Undecided → Medium
Changed in landscape-client (Ubuntu):
assignee: nobody → Simon Poirier (simpoir)
Changed in landscape-client (Ubuntu Bionic):
assignee: nobody → Simon Poirier (simpoir)
Changed in landscape-client (Ubuntu Cosmic):
assignee: nobody → Simon Poirier (simpoir)
Changed in landscape-client (Ubuntu Xenial):
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Simon Poirier (simpoir)
Changed in landscape-client (Ubuntu Trusty):
status: New → In Progress
assignee: nobody → Simon Poirier (simpoir)
importance: Undecided → Medium

An upload of landscape-client to cosmic-proposed has been rejected from the upload queue for the following reason: "Includes a change without an SRU bug attached to it".

Timo Aaltonen (tjaalton) wrote :

An upload of landscape-client to bionic-proposed has been rejected from the upload queue for the following reason: "Includes a change without an SRU bug attached to it".

Timo Aaltonen (tjaalton) wrote :

An upload of landscape-client to xenial-proposed has been rejected from the upload queue for the following reason: "Includes a change without an SRU bug attached to it".

Timo Aaltonen (tjaalton) wrote :

An upload of landscape-client to trusty-proposed has been rejected from the upload queue for the following reason: "Includes a change without an SRU bug attached to it".

Hello Andreas, or anyone else affected,

Accepted landscape-client into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/18.01-0ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in landscape-client (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Changed in landscape-client (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Timo Aaltonen (tjaalton) wrote :

Hello Andreas, or anyone else affected,

Accepted landscape-client into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/18.01-0ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in landscape-client (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Timo Aaltonen (tjaalton) wrote :

Hello Andreas, or anyone else affected,

Accepted landscape-client into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/16.03-0ubuntu2.16.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Timo Aaltonen (tjaalton) wrote :

Hello Andreas, or anyone else affected,

Accepted landscape-client into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/14.12-0ubuntu6.14.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in landscape-client (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed-trusty
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers