[1.9] If rack and region have different versions, the error is uninformative and confusing

Bug #1567249 reported by Haw Loeung on 2016-04-07
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MAAS
Undecided
Unassigned
1.9
Undecided
Unassigned

Bug Description

Hi,

We've just recently upgraded our MAAS cluster (one region controller and six cluster controllers) from 1.7 to 1.9. After upgrading, we ran into the following traceback both in the logs and also when trying to view a node:

| File "/usr/lib/python2.7/dist-packages/maasserver/websockets/base.py", line 205, in full_dehydrate
| return self.dehydrate(obj, data, for_list=for_list)
| File "/usr/lib/python2.7/dist-packages/maasserver/websockets/handlers/cluster.py", line 59, in dehydrate
| data["power_types"] = self.dehydrate_power_types(obj)
| File "/usr/lib/python2.7/dist-packages/maasserver/websockets/handlers/cluster.py", line 65, in dehydrate_power_types
| return get_all_power_types_from_clusters(nodegroups=[obj])
| File "/usr/lib/python2.7/dist-packages/maasserver/clusterrpc/power_parameters.py", line 202, in get_all_power_types_from_clusters
| missing_packages = power_type['missing_packages']
| exceptions.KeyError: u'missing_packages'

I worked around this by changing the code in maasserver/clusterrpc/power_parameters.py to:

| description = power_type['description']
| if 'missing_packages' in power_type:
| missing_packages = power_type['missing_packages']
| else:
| missing_packages = None

Is that the correct fix for this?

Related branches

Blake Rouse (blake-rouse) wrote :

Are you sure that all cluster controllers have the same installed version of MAAS as the region controller? MAAS requires that all of them be the same version.

Changed in maas:
status: New → Incomplete
Haw Loeung (hloeung) wrote :

@blake-rouse, ah that would appear it. I had later realised I missed one cluster controller and upgraded that.

Anyways, I reverted the change (and bounced regiond) and it appears to be working fine now - no tracebacks.

Still, I think an error about possible mismatch in versions or something else would be nicer as it took a bit of time for me figuring out how to get our MAAS cluster back up and running.

Launchpad Janitor (janitor) wrote :

[Expired for MAAS because there has been no activity for 60 days.]

Changed in maas:
status: Incomplete → Expired
LaMont Jones (lamont) on 2016-06-16
summary: - 'missing_packages' missing and causes traceback
+ 'missing_packages' missing and causes traceback on version mismatches
summary: - 'missing_packages' missing and causes traceback on version mismatches
+ If rack and region have different versions, the error is uninformative
+ and confusing
Changed in maas:
status: Expired → Confirmed
milestone: none → 1.9.4

Based on the specific error we hit above, we should be able to make MAAS provide a clearer error message for that KeyError, since it indicates that an older version of MAAS is installed on one of the cluster controllers. We should indicate which cluster controller in the error message, if we can determine that.

LaMont Jones (lamont) wrote :

If the region is running 2.0 or 2.1, then the 1.7 cluster simply fails to register, and the region log shows it connecting and disconnecting, over and over and over. While not ideal, the admin will likely go look into why the cluster isn't there, and notice that it's downrev.

LaMont Jones (lamont) on 2016-07-11
summary: - If rack and region have different versions, the error is uninformative
- and confusing
+ [1.9] If rack and region have different versions, the error is
+ uninformative and confusing
LaMont Jones (lamont) on 2016-07-11
no longer affects: maas/2.0
no longer affects: maas/trunk
Changed in maas:
milestone: 2.0.1 → none
status: Confirmed → Incomplete
Andres Rodriguez (andreserl) wrote :

Hi!

**This is an automated message**

We believe this is may no longer be an issue in the latest MAAS release. Due to the report date of this, we are currently marking it as Invalid. If you believe this bug report still valid against the latest release of MAAS, or if you are still interested in this, please re-open this bug report.

Thanks

Changed in maas:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers