Adding new node to MOS env without MU installed updates the new node with latest packages and breaks compatibility.

[Javier Diaz Jr]

Detailed bug description:

Customer reported having to redeploy a compute node in their environment after the disk on the node failed. When redeployment of node finished the node was not being picked up by controllers. After further investigation it was determined that the controller nodes and other compute nodes where running MOS 8.0 without MU1 while the newly deployed compute node had MU1 packages installed, thus breaking the compatibility.

Compute node that works:
nova-compute 2:12.0.0-1~u14.04+mos43

New compute node:
nova-compute 2:12.0.2-1~u14.04+mos21

Steps to reproduce:
 Add new node to MOS 8 environment that does not yet have MU1 installed.

Expected results:
 Newly deployed node should have packages that match the environments current deployed status. That is, if environment is pure MOS 8.0 new nodes should have MOS 8.0 packages, not the latest.

Actual result:
 Newly deployed nodes have latest packages which breaks compatibility

Reproducibility:
 I do not have a lab to test this so I cannot confirm reproducibility.

Workaround:
 Downgrade packages.

Impact:
 New nodes are not useable until packages are downgraded to match current environment packages.

Description of the environment:
- Operation system: Ubuntu 14.04
- Versions of components: MOS 8.0

Additional details:

Fuel Master has not yet been updated to MU1 yet.

[Vitaly Sedelnik (vsedelnik)]

Setting to Incomplete as this is more like general concern, not specific compatibility issue.

Installing the most recent package versions from updates repository for newly deployed nodes is expected behavior. If for some reason it's desired to use packages of GA version - updates repo should be removed from repository settings in Fuel, see https://docs.mirantis.com/openstack/fuel/fuel-8.0/operations.html#applying-patches

Javier, is there some specific compatibility issue observed by customer? Please provide more details, thanks.

[Denis Meltsaykin (dmeltsaykin)]

Reproduced the issue, will dig into it.

[Javier Diaz Jr (javierdiazcharles)]

@Vitaly

The customer reported that once the compute node was deployed it wasn't getting picked up by the controllers as a hypervisor. They downgraded the packages to GA version and node was able to register, work, etc. I am deploying a lab as we speak. Will remove mos-update repo and deploy a test env. Then add mos-update repo and add a compute new node. I will let you know what I see, though it seems Denis M. might have already done so.

[Javier Diaz Jr (javierdiazcharles)]

After testing this in my lab I can confirm that a GA environment (3 controllers, 1 compute) with a second compute node added (with MU1) is not getting registered as a hypervisor via nova service-list.

[Denis Meltsaykin (dmeltsaykin)]

Yes, we will be investigating the issue with the nova team today.

[Denis Meltsaykin (dmeltsaykin)]

RCA: The issue is caused by the fix of https://bugs.launchpad.net/nova/+bug/1518200. This happens because not updated controllers have no such a piece of code https://github.com/openstack/nova/blob/stable/liberty/nova/db/sqlalchemy/api.py#L4496-L4497, which is needed by https://github.com/openstack/nova/blob/stable/liberty/nova/compute/manager.py#L817.
As it was a real bug and it was fixed, I believe we shouldn't revert it in our branch making the divergence with upstream. So the workarounds stay the same: one needs either to update the whole environment with MU or to disable updates repository before adding new computes/controllers.

[Josue Palmerin (jpalmerin)]

This is affecting a MOS 9.2 component with Neutron packages as well.

[Denis Meltsaykin (dmeltsaykin)]

Josue, if this is affecting MOS 9.2 you should create another milestone in the bug rather than confirming 8.0-updates.